Re: [Condor-users] restricting the number of jobs

[Bill Taylor <bt@xxxxxxxxxxx>]

DAGs do lots of good things including allowing RETRY if a job fails(say 
from hitting a bad node).
Within the dag file besides command line things you mentioned there is 
-maxpre. Frequently the
prescript sets up the job node and as such is particularly IO intensive.

But within the DAG you can set up Categories for nodes and set MAXJOBS 
for those categories
thus letting you throttle just about any part of a DAG.

Lets not forget that you know when all the jobs are done when the DAG 
completes which
is always a handy thing to know.

One last addition. As long as you have a DAG and all the jobs are done, 
you can add a node
to do cleanup or wrapup.

Bill
Condor Team

Dillman, Kimberley A wrote:
> I have a similar issue (no I/O intensive but a reasonably large number of jobs).
>
> I set up a dagman job to control it since you can provide a command line option to condor_submit_dag that allows you to specific "maxidle" and "maxjobs". I think (from what I read), "maxjobs" only applies to each separate "job" inside the dagman and not the jobs inside the individual submit script (such as you have here) but I believe that the documentation says that "maxidle" does apply to each individual job submitted even though it is within a single submit script (such as you have).
>
> I haven't tried it though. I just created a dagman job that would have all of the jobs separated out in the dagman submit script because I wanted to vary the input/output file names and initial directory per job as well as control how many jobs were submitted at any one time.
>
> Maybe you can try that.
>
> Kim
>
> ------------------------------------------------------------------------------
> Kim Dillman
> Research Programmer – Rosen Center for Advanced Computing
> Purdue TeraGrid Campus Champion
> YONG 956
> Phone: 765-494-5446
> Email: kadillma@xxxxxxxxxx
>
> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Mag Gam
> Sent: Saturday, October 10, 2009 2:06 PM
> To: Condor-Users Mail List
> Subject: Re: [Condor-users] restricting the number of jobs
>
> how can I let the user control it?
>
> WE have some users whose jobs are very I/O intensive. They want to run
> only 10 at time.
>
>
>
> On Sat, Oct 10, 2009 at 2:55 PM, dawnsong <dawnsong.tsinghua@xxxxxxxxx> wrote:
>   
> > set
> > MAX_JOBS_RUNNING = 10
> > in Condor global configuration file.
> >
> > 2009/10/10 Mag Gam <magawake@xxxxxxxxx>
> >     
> > > Is it possible to restrict the number of jobs to run?
> > >
> > > For example?
> > >
> > > I have something like this:
> > >
> > > Universe       = vanilla
> > > Executable     = hello_world.sh
> > >
> > > input   = /dev/null
> > > output  = hello.out
> > > error   = hello.error
> > >
> > > Queue 5000
> > >
> > >
> > > What is I want only 10 to run at a time?
> > >
> > > Is that possible to do?
> > > _______________________________________________
> > > Condor-users mailing list
> > > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> > > subject: Unsubscribe
> > > You can also unsubscribe by visiting
> > > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > >
> > > The archives can be found at:
> > > https://lists.cs.wisc.edu/archive/condor-users/
> > >       
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/condor-users/
> >
> >
> >     
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/condor-users/
>