[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[condor-users] easily submitting many identical jobs with DAGMan (was Re: Dagman job submission problem)

Antti Eskola wrote:
> I just found out that dagman does not support submitting many jobs
> in one submit file... :(

True -- but this is by design, and shouldn't be a problem, as I'll
explain below.

> but if anybody has any idea pls let me know.
> Next thing is test making 0...n submit files and a really big dagman
> submit file :) fearing that it wont work when manymany jobs...

Although DAGMan (for important reasons) does not allow you to use the
Condor "queue N" feature to submit N instances of the same job from
one submit file, you *can* still use a single, common submit file for
multiple nodes in a DAG.  E.g.:

  Job A submit.cmd
  Job B submit.cmd
  Job C submit.cmd

As long as the submit.cmd file is written like it would be for use
with the "queue N" feature, in that each instance of the job has
unique input/output/error files (e.g., by using the $(Cluster) macro),
then it is not necessary to make N copies of the same submit file in
order to submit N identical jobs in a DAG.

This should be just as convenient as using "queue N" -- you still only
need one submit file -- but it results in a well-defined DAG in which
each Condor job is explicitly declared as a node in the DAG, and there
is only one Condor job per DAG node.

This one-to-one relationship is important for the DAG semantics.  If
"queue N" were allowed and thus more than one Condor job could exist
for a single DAG node, then a node would no longer be atomic, the DAG
semantics would be violated, and recovery, etc. would be complicated

I hope this helps.  Let me know if you have other questions.


Peter Couvares                        http://www.cs.wisc.edu/~pfc
Condor Project Research               pfc@xxxxxxxxxxx
University of Wisconsin-Madison       (608) 265-8936

Condor Support Information:
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>