[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] unsure about potential application
- Date: Sun, 15 Aug 2010 17:11:30 -0400
- From: Matthew Farrellee <matt@xxxxxxxxxx>
- Subject: Re: [Condor-users] unsure about potential application
On 08/15/2010 03:46 PM, andrew.downs@xxxxxxxx wrote:
Greetings, I'm trying to find out if Condor is the right software for
what I'm doing. My application (q3map2.exe) is a type of compiler that
assembles map data for Quake 3 Arena. The kinds of things I've designed
for it have become so complex that I think a small cluster might be in
order. While reading through Condor documentation, I saw this:
"Multi-process jobs are not allowed. This includes system calls such as
fork(), exec(), and system()."
That info is dated at best, and currently only remotely true in the
context of Condor's Standard Universe. Avoiding fork/exec/system and
open sockets is a requirement if you link with Condor's user-space
checkpointing library and expect checkpointing to work. Probably not
something you're looking for atm anyway.
Condor will happily run any program that runs unattended, i.e. in some
sort of batch mode.
q3map2 can be multi-threaded for either SMP or multi-core
configurations. Is that the same as being multi-process?
My long term plan was to have 4 systems, each with a 6-core AMD chip all
chewing through the same data set (but different parts) at the same
time, in other words, my aim is to have q3map2 utilize the computational
resources of each machine as if all of the cores in the separate systems
were all in one system.
My vague understanding of Condor is that if you submit a job (presumably
q3map2.exe) Condor would find all of the available machines and split up
the work between them in some fashion.
That would be pretty incredible. Condor isn't going to give your program
a unified view of the systems, as if they were one, or break up your
problem space automatically. Once you've broken up your problem space
into independent pieces, what Condor will do is all the scheduling of
processing, management of input/output files, process execution
management, failed process re-execution, provide progress and
utilization information, etc.
What leads me to believe this is:
Myth: Condor requires users to recompile their applications.
*Reality: Condor runs ordinary, unmodified applications.*
*I'm not even sure how that's possible, but if it is, it's incredible
and perfect for what i'd like to do.*
Is condor too much for what I'm trying to do? Should I be looking
somewhere else? Reading something else?
Thanks for your time.
"Condor runs ordinary, unmodified applications" - totally true. If you
do recompile you can get additional features. The information in the
manual is from an era when people thought checkpointing and Condor's
Standard Universe (maybe because of the name?) were the only way Condor
would run jobs.
Condor certainly isn't too much for what you're trying to do. It can be
a pretty lightweight system. It contains a lot of features you'll
blissfully ignore at first and maybe get interested in later on.