[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] submitting LOTS of jobs
- Date: Thu, 12 Nov 2009 07:27:42 -0500
- From: Mag Gam <magawake@xxxxxxxxx>
- Subject: Re: [Condor-users] submitting LOTS of jobs
Do you have an example of a DAGMan job for a situation like this?
Lets say he would only like to have 10 jobs running and 10 jobs idle
while still trying to resolve 1000000000. Is that possible to do ?
On Wed, Nov 4, 2009 at 1:38 PM, Steven Timm <timm@xxxxxxxx> wrote:
> Most of the schedd submit-side limits deal with how many jobs
> are running at any given time, not how many are in the queue at all.
> DAGman or one of the newer workflow managers should be used to control
> the workflow and the job submissions. You might want to look at
> Falkon/Swift from the U of Chicago, they have been doing a lot of work
> like this recently although things seem to be working best for them
> when they run on a BlueGene.
> On Wed, 4 Nov 2009, Jonathan D. Proulx wrote:
>> Hi All,
>> I have a user looking to submit 15million jobs this is about two
>> orders of magnitude above what weve done previously. I don't yet know
>> too much about the calculation other than it's genome based and
>> running against the human genome (a later goal) would result in
>> 1.8 billion jobs.
>> leaving aside for a monent the question of how long this will take on
>> my resources, any advise on facilitating this number of jobs? Are
>> there submit side limits I'm going to run into?
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> The archives can be found at:
> Steven C. Timm, Ph.D (630) 840-8525
> timm@xxxxxxxx http://home.fnal.gov/~timm/
> Fermilab Computing Division, Scientific Computing Facilities,
> Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> The archives can be found at: