Re: [Condor-users] submitting LOTS of jobs

[Steven Timm <timm@xxxxxxxx>]

Most of the schedd submit-side limits deal with how many jobs
are running at any given time, not how many are in the queue at all.
DAGman or one of the newer workflow managers should be used to control
the workflow and the job submissions.  You might want to look at 
Falkon/Swift from the U of Chicago, they have been doing a lot of work
like this recently although things seem to be working best for them
when they run on a BlueGene.

Steve


On Wed, 4 Nov 2009, Jonathan D. Proulx wrote:

> Hi All,
>
> I have a user looking to submit 15million jobs this is about two
> orders of magnitude above what weve done previously.  I don't yet know
> too much about the calculation other than it's genome based and
> running against the human genome (a later goal) would result in
> 1.8 billion jobs.
>
> leaving aside for a monent the question of how long this will take on
> my resources, any advise on facilitating this number of jobs?  Are
> there submit side limits I'm going to run into?
>
> Thanks
> -Jon
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/

--
------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.