[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] unsure about potential application

On 08/15/2010 02:46 PM, andrew.downs@xxxxxxxx wrote:
Greetings, I'm trying to find out if Condor is the right software for
what I'm doing. My application (q3map2.exe) is a type of compiler that
assembles map data for Quake 3 Arena. The kinds of things I've designed
for it have become so complex that I think a small cluster might be in
order. While reading through Condor documentation, I saw this:

Possibly... read on.

"Multi-process jobs are not allowed. This includes system calls such as
fork(), exec(), and system()."

Condor has the notion of universes (flavors/characteristics of jobs you'd like to run). You would almost certainly be using the vanilla universe. This comment is about standard universe, so I think you could safely ignore it.

q3map2 can be multi-threaded for either SMP or multi-core
configurations. Is that the same as being multi-process?

No. Multi-process means more than one process ID for you job.

My long term plan was to have 4 systems, each with a 6-core AMD chip all
chewing through the same data set (but different parts) at the same
time, in other words, my aim is to have q3map2 utilize the computational
resources of each machine as if all of the cores in the separate systems
were all in one system.

Condor won't create the interface of "one big computer" that you're running on.

Condor can help you if you can take advantage of process-level parallelism. Meaning, you'd run multiple instances of q3map2.exe, and allow each of them to work on different portions of your map. Condor will help you with the resource allocation (finding a CPU core that is free), job execution/monitoring (transferring input, watching over the executable while it runs, and transfering output when done), and even the work flow (complete part A and part B before beginning part C, because C requires results from A and B).

In summary, Condor is a perfect match for video game studios. It's an open source alternative to things like Xoreax's IncrediGrid. You have to be willing to work with it, though. Take a step back from thinking about high-performance on one computer (like your typical game engine), and think about running tons of executables across a cluster of computers (and having the power to coordinate that execution in a workflow of your choosing).