[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] unsure about potential application

On 08/15/2010 03:46 PM, andrew.downs@xxxxxxxx wrote:

Greetings, I'm trying to find out if Condor is the right software for
what I'm doing. My application (q3map2.exe) is a type of compiler that
assembles map data for Quake 3 Arena. The kinds of things I've designed
for it have become so complex that I think a small cluster might be in
order. While reading through Condor documentation, I saw this:

"Multi-process jobs are not allowed. This includes system calls such as
fork(), exec(), and system()."

That info is dated at best, and currently only remotely true in the context of Condor's Standard Universe. Avoiding fork/exec/system and open sockets is a requirement if you link with Condor's user-space checkpointing library and expect checkpointing to work. Probably not something you're looking for atm anyway.

Condor will happily run any program that runs unattended, i.e. in some sort of batch mode.

q3map2 can be multi-threaded for either SMP or multi-core
configurations. Is that the same as being multi-process?

My long term plan was to have 4 systems, each with a 6-core AMD chip all
chewing through the same data set (but different parts) at the same
time, in other words, my aim is to have q3map2 utilize the computational
resources of each machine as if all of the cores in the separate systems
were all in one system.

My vague understanding of Condor is that if you submit a job (presumably
q3map2.exe) Condor would find all of the available machines and split up
the work between them in some fashion.

That would be pretty incredible. Condor isn't going to give your program a unified view of the systems, as if they were one, or break up your problem space automatically. Once you've broken up your problem space into independent pieces, what Condor will do is all the scheduling of processing, management of input/output files, process execution management, failed process re-execution, provide progress and utilization information, etc.

What leads me to believe this is:

Myth: Condor requires users to recompile their applications.
*Reality: Condor runs ordinary, unmodified applications.*
*I'm not even sure how that's possible, but if it is, it's incredible
and perfect for what i'd like to do.*
Is condor too much for what I'm trying to do? Should I be looking
somewhere else? Reading something else?

Thanks for your time.

"Condor runs ordinary, unmodified applications" - totally true. If you do recompile you can get additional features. The information in the manual is from an era when people thought checkpointing and Condor's Standard Universe (maybe because of the name?) were the only way Condor would run jobs.

Condor certainly isn't too much for what you're trying to do. It can be a pretty lightweight system. It contains a lot of features you'll blissfully ignore at first and maybe get interested in later on.