Re: [Condor-users] unsure about potential application

[Mick Beaver <mick@xxxxxxxxxxx>]

On 08/15/2010 02:46 PM, andrew.downs@xxxxxxxx wrote:
> Greetings, I'm trying to find out if Condor is the right software for
> what I'm doing. My application (q3map2.exe) is a type of compiler that
> assembles map data for Quake 3 Arena. The kinds of things I've designed
> for it have become so complex that I think a small cluster might be in
> order. While reading through Condor documentation, I saw this:

Possibly... read on.

> "Multi-process jobs are not allowed. This includes system calls such as
> fork(), exec(), and system()."

Condor has the notion of universes (flavors/characteristics of jobs 
you'd like to run). You would almost certainly be using the vanilla 
universe. This comment is about standard universe, so I think you could 
safely ignore it.

> q3map2 can be multi-threaded for either SMP or multi-core
> configurations. Is that the same as being multi-process?

No. Multi-process means more than one process ID for you job.

> My long term plan was to have 4 systems, each with a 6-core AMD chip all
> chewing through the same data set (but different parts) at the same
> time, in other words, my aim is to have q3map2 utilize the computational
> resources of each machine as if all of the cores in the separate systems
> were all in one system.

Condor won't create the interface of "one big computer" that you're 
running on.

Condor can help you if you can take advantage of process-level 
parallelism. Meaning, you'd run multiple instances of q3map2.exe, and 
allow each of them to work on different portions of your map. Condor 
will help you with the resource allocation (finding a CPU core that is 
free), job execution/monitoring (transferring input, watching over the 
executable while it runs, and transfering output when done), and even 
the work flow (complete part A and part B before beginning part C, 
because C requires results from A and B).

In summary, Condor is a perfect match for video game studios. It's an 
open source alternative to things like Xoreax's IncrediGrid. You have to 
be willing to work with it, though. Take a step back from thinking about 
high-performance on one computer (like your typical game engine), and 
think about running tons of executables across a cluster of computers 
(and having the power to coordinate that execution in a workflow of your 
choosing).

-Mick