Re: [Condor-users] restricting the number of jobs

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

Yes, DAGMan is a pretty easy way to do this, if you can split out your 
submit file into individual submits (one submit per Condor job).

Just make a DAG with no parent/child relationships, like this:

JOB node0000 foo.submit
JOB node0001 foo.submit
JOB node0001 foo.submit
...

Then run

  condor_submit_dag -maxjobs 10 foo.dag

Or you can use a per-DAG configuration file so that you don't need
to remember to use the -maxjobs flag on the command line.  To do this,
put this line in your DAG file:

  CONFIG foo.config

Then in foo.config, have this:

  DAGMAN_MAX_JOBS_SUBMITTED = 10

(Note that this limits you to 10 jobs *in the queue* at a time -- jobs 
that are idle still count against the DAGMan maxjobs limit.)

Kent Wenger
Condor Team

> I have a similar issue (no I/O intensive but a reasonably large number 
> of jobs).
>
> I set up a dagman job to control it since you can provide a command 
> line option to condor_submit_dag that allows you to specific "maxidle" 
> and "maxjobs". I think (from what I read), "maxjobs" only applies to 
> each separate "job" inside the dagman and not the jobs inside the 
> individual submit script (such as you have here) but I believe that the 
> documentation says that "maxidle" does apply to each individual job 
> submitted even though it is within a single submit script (such as you 
> have).
>
> I haven't tried it though. I just created a dagman job that would have 
> all of the jobs separated out in the dagman submit script because I 
> wanted to vary the input/output file names and initial directory per job 
> as well as control how many jobs were submitted at any one time.
>
> Maybe you can try that.
>
> Kim
>
> ------------------------------------------------------------------------------
> Kim Dillman
> Research Programmer – Rosen Center for Advanced Computing
> Purdue TeraGrid Campus Champion
> YONG 956
> Phone: 765-494-5446
> Email: kadillma@xxxxxxxxxx
>
> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Mag Gam
> Sent: Saturday, October 10, 2009 2:06 PM
> To: Condor-Users Mail List
> Subject: Re: [Condor-users] restricting the number of jobs
>
> how can I let the user control it?
>
> WE have some users whose jobs are very I/O intensive. They want to run
> only 10 at time.
>
>
>
> On Sat, Oct 10, 2009 at 2:55 PM, dawnsong <dawnsong.tsinghua@xxxxxxxxx> wrote:
> > set
> > MAX_JOBS_RUNNING = 10
> > in Condor global configuration file.
> >
> > 2009/10/10 Mag Gam <magawake@xxxxxxxxx>
> > > Is it possible to restrict the number of jobs to run?
> > >
> > > For example?
> > >
> > > I have something like this:
> > >
> > > Universe       = vanilla
> > > Executable     = hello_world.sh
> > >
> > > input   = /dev/null
> > > output  = hello.out
> > > error   = hello.error
> > >
> > > Queue 5000
> > >
> > >
> > > What is I want only 10 to run at a time?
> > >
> > > Is that possible to do?
> > > _______________________________________________
> > > Condor-users mailing list
> > > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> > > subject: Unsubscribe
> > > You can also unsubscribe by visiting
> > > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > >
> > > The archives can be found at:
> > > https://lists.cs.wisc.edu/archive/condor-users/
> >
> >
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at: 
https://lists.cs.wisc.edu/archive/condor-users/