[Condor-users] Running short-lived jobs on Condor

[<Greg.Hitchen@xxxxxxxx>]

Hi All

I realize that Condor is not optimized for running many small jobs
but was wondering what parameters can be adjusted to make it perform
"better" for this situation.

We have a user that has a process whereby they want to submit around
30 jobs that take about 10 mins each to run. Then based on the results
of these generate another 30 jobs to run, and so on, in an optimization
type of problem.

There are a number of whatever_INTERVAL parameters with 5 min defaults
that could be reduced to say 30 secs. Perhaps policies could be put in
place to run "short" jobs regardless of machine usage, but at a nice
priority. Maybe divide machines into virtual cpus with one dedicated
for always running short jobs, etc., etc.

I guess my question is has anyone been down this path before?
And what have you discovered work best? Doesn't make a difference?
Causes problems? Works well?, etc.

Thanks for any info.

Cheers

Greg