[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] submitting LOTS of jobs

Most of the schedd submit-side limits deal with how many jobs
are running at any given time, not how many are in the queue at all.
DAGman or one of the newer workflow managers should be used to control
the workflow and the job submissions. You might want to look at Falkon/Swift from the U of Chicago, they have been doing a lot of work
like this recently although things seem to be working best for them
when they run on a BlueGene.


On Wed, 4 Nov 2009, Jonathan D. Proulx wrote:

Hi All,

I have a user looking to submit 15million jobs this is about two
orders of magnitude above what weve done previously.  I don't yet know
too much about the calculation other than it's genome based and
running against the human genome (a later goal) would result in
1.8 billion jobs.

leaving aside for a monent the question of how long this will take on
my resources, any advise on facilitating this number of jobs?  Are
there submit side limits I'm going to run into?

Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Steven C. Timm, Ph.D  (630) 840-8525
timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.