[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] submitting LOTS of jobs

On Thu, 12 Nov 2009, Mag Gam wrote:

Do you have an example of a DAGMan job for a situation like this?

Lets say he would only like to have 10 jobs running and 10 jobs idle
while still trying to resolve 1000000000. Is that possible to do ?

Well 1000000000 separate jobs would be too big for a DAG.  But you could
do, say, 1000000. You just make a big DAG that has all of the jobs (there don't have to be any parent/child relationships in a DAG):

JOB Node0000000 job0000000.submit
JOB Node0000001 job0000001.submit

(You might be able to use VARS to re-use the same submit file if your
jobs are similar enough.)

Then you just say:

  condor_submit_dag -maxjobs 10 -maxidle 10 <dag file>

Or if you don't want to have to remember to specify the maxjobs and maxidle on the command line, you can put them in a per-DAG configuration file:




  CONFIG large.config
  JOB ...

Then just do:

  condor_submit_dag large.dag

If you really need to handle a workflow of 1G jobs, I guess you could
use nested DAGs -- have each sub-DAG handle 1M jobs, and have the top-level DAG run 1k sub-DAGs, with maxjobs for the top-level DAG set to 1, and appropriate throttles on the sub-DAGs.

Kent Wenger
Condor Team