Re: [Condor-users] submitting LOTS of jobs

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Thu, 12 Nov 2009, Mag Gam wrote:

> Do you have an example of a DAGMan job for a situation like this?

> Lets say he would only like to have 10 jobs running and 10 jobs idle
> while still trying to resolve 1000000000. Is that possible to do ?

Well 1000000000 separate jobs would be too big for a DAG.  But you could
do, say, 1000000.  You just make a big DAG that has all of the jobs (there 
don't have to be any parent/child relationships in a DAG):

JOB Node0000000 job0000000.submit
JOB Node0000001 job0000001.submit
...

(You might be able to use VARS to re-use the same submit file if your
jobs are similar enough.)

Then you just say:

  condor_submit_dag -maxjobs 10 -maxidle 10 <dag file>

Or if you don't want to have to remember to specify the maxjobs and 
maxidle on the command line, you can put them in a per-DAG configuration 
file:

large.config:

  DAGMAN_MAX_JOBS_SUBMITTED = 10
  DAGMAN_MAX_JOBS_IDLE = 10

large.dag:

  CONFIG large.config
  JOB ...
  ...

Then just do:

  condor_submit_dag large.dag


If you really need to handle a workflow of 1G jobs, I guess you could
use nested DAGs -- have each sub-DAG handle 1M jobs, and have the 
top-level DAG run 1k sub-DAGs, with maxjobs for the top-level DAG set to 
1, and appropriate throttles on the sub-DAGs.

Kent Wenger
Condor Team