[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] unsure about potential application
- Date: Mon, 16 Aug 2010 11:49:03 -0500
- From: Mick Beaver <mick@xxxxxxxxxxx>
- Subject: Re: [Condor-users] unsure about potential application
On 08/15/2010 02:46 PM, andrew.downs@xxxxxxxx wrote:
Greetings, I'm trying to find out if Condor is the right software for
what I'm doing. My application (q3map2.exe) is a type of compiler that
assembles map data for Quake 3 Arena. The kinds of things I've designed
for it have become so complex that I think a small cluster might be in
order. While reading through Condor documentation, I saw this:
Possibly... read on.
"Multi-process jobs are not allowed. This includes system calls such as
fork(), exec(), and system()."
Condor has the notion of universes (flavors/characteristics of jobs
you'd like to run). You would almost certainly be using the vanilla
universe. This comment is about standard universe, so I think you could
safely ignore it.
q3map2 can be multi-threaded for either SMP or multi-core
configurations. Is that the same as being multi-process?
No. Multi-process means more than one process ID for you job.
My long term plan was to have 4 systems, each with a 6-core AMD chip all
chewing through the same data set (but different parts) at the same
time, in other words, my aim is to have q3map2 utilize the computational
resources of each machine as if all of the cores in the separate systems
were all in one system.
Condor won't create the interface of "one big computer" that you're
Condor can help you if you can take advantage of process-level
parallelism. Meaning, you'd run multiple instances of q3map2.exe, and
allow each of them to work on different portions of your map. Condor
will help you with the resource allocation (finding a CPU core that is
free), job execution/monitoring (transferring input, watching over the
executable while it runs, and transfering output when done), and even
the work flow (complete part A and part B before beginning part C,
because C requires results from A and B).
In summary, Condor is a perfect match for video game studios. It's an
open source alternative to things like Xoreax's IncrediGrid. You have to
be willing to work with it, though. Take a step back from thinking about
high-performance on one computer (like your typical game engine), and
think about running tons of executables across a cluster of computers
(and having the power to coordinate that execution in a workflow of your