[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Betr.: Running short-lived jobs on Condor



There was a presentation at the Condor Week on the so-called "Master-Worker" technology which is supposed to deal with problems
like this, for many processes each taking a couple of minutes or less.
I have not tried it myself so can't say if it would help.

Steve


On Thu, 28 Sep 2006, Zeeuw, L.V. de wrote:

LS,

We are facing more or less the same challenge. We have a large pool (>1500 XP execution nodes) and one central machine from which we submit jobs. When we submit small jobs, which should run for about 30 seconds, then if we submit say 1000 of such jobs it would take 45 minutes for the results to come back to the submitting host from the hundreds of available execution hosts.

So, also for us, any pointers are appreciated to optimize for small jobs.

Cheers.
Luc de Zeeuw
Rotterdam University

Greg.Hitchen@xxxxxxxx 28-09-06 5:10 >>>

Hi All

I realize that Condor is not optimized for running many small jobs
but was wondering what parameters can be adjusted to make it perform
"better" for this situation.

We have a user that has a process whereby they want to submit around
30 jobs that take about 10 mins each to run. Then based on the results
of these generate another 30 jobs to run, and so on, in an optimization
type of problem.

There are a number of whatever_INTERVAL parameters with 5 min defaults
that could be reduced to say 30 secs. Perhaps policies could be put in
place to run "short" jobs regardless of machine usage, but at a nice
priority. Maybe divide machines into virtual cpus with one dedicated
for always running short jobs, etc., etc.

I guess my question is has anyone been down this path before?
And what have you discovered work best? Doesn't make a difference?
Causes problems? Works well?, etc.

Thanks for any info.

Cheers

Greg

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at either
https://lists.cs.wisc.edu/archive/condor-users/
http://www.opencondor.org/spaces/viewmailarchive.action?key=CONDOR


_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at either
https://lists.cs.wisc.edu/archive/condor-users/
http://www.opencondor.org/spaces/viewmailarchive.action?key=CONDOR


--
------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525  timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Div/Core Support Services Dept./Scientific Computing Section
Assistant Group Leader, Farms and Clustered Systems Group
Lead of Computing Farms Team