Re: [Condor-users] Condor performance problem

[Jeff Linderoth <jtl3@xxxxxxxxxx>]

Hello Hao,

> However there are still some comments abouts such performance tests: 
> "Dr. David Jones at UCL conducted some tests of the Condor system.The
> system had 80 free processors but Jones didn't manage to get more than
> 10-15 running at any one time. The management simply made it
> impossible to use the resources effectively. The reason 
> here was quite simple. Jones used a series of relatively short (5-10
> minutes) jobs. The transport and
> assignment service was too slow in comparison with these small tasks.
> Even if previous submitted
> tasks are completed, new tasks might be still in the queue waiting for
> scheduling or hold up in 
> transportation. Consequently he could never make full use of 80
> processors although most of them
> are free. This performance problem is fatal for those applications
> requiring real time execution as
> we discuss here. Because of inefficient resource allocation, Condor
> could not satisfy the 
> requirement for high response performance."
> 
> Does you guys have any thoughts about those tests and comments ?

Do you have a specific citation for this quote that you could provide?

For running many relatively short jobs through Condor, I would recommend
using MW.
http://www.cs.wisc.edu/condor/mw/

Using MW, we have successfully performed large computations, where jobs
were only a few seconds, and scaled Condor configurations to hundreds
(in some cases thousands) of processors.

I hope this helps.  Let me know if you would like help or more specific
information about MW.

Cheers,
-Jeff

-- 
------------------------------------------------------------
Jeff Linderoth                               O: 610-758-4879
Asst. Professor                              
Industrial and Systems Engineering           jtl3@xxxxxxxxxx
Lehigh University                       www.lehigh.edu/~jtl3