[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[condor-users] easily submitting many identical jobs with DAGMan (was Re: Dagman job submission problem)



Antti Eskola wrote:
> I just found out that dagman does not support submitting many jobs
> in one submit file... :(

True -- but this is by design, and shouldn't be a problem, as I'll
explain below.

> but if anybody has any idea pls let me know.
>
> Next thing is test making 0...n submit files and a really big dagman
> submit file :) fearing that it wont work when manymany jobs...

Although DAGMan (for important reasons) does not allow you to use the
Condor "queue N" feature to submit N instances of the same job from
one submit file, you *can* still use a single, common submit file for
multiple nodes in a DAG.  E.g.:

  Job A submit.cmd
  Job B submit.cmd
  Job C submit.cmd
  ...

As long as the submit.cmd file is written like it would be for use
with the "queue N" feature, in that each instance of the job has
unique input/output/error files (e.g., by using the $(Cluster) macro),
then it is not necessary to make N copies of the same submit file in
order to submit N identical jobs in a DAG.

This should be just as convenient as using "queue N" -- you still only
need one submit file -- but it results in a well-defined DAG in which
each Condor job is explicitly declared as a node in the DAG, and there
is only one Condor job per DAG node.

This one-to-one relationship is important for the DAG semantics.  If
"queue N" were allowed and thus more than one Condor job could exist
for a single DAG node, then a node would no longer be atomic, the DAG
semantics would be violated, and recovery, etc. would be complicated
unnecessarily.

I hope this helps.  Let me know if you have other questions.

-Peter

-- 
Peter Couvares                        http://www.cs.wisc.edu/~pfc
Condor Project Research               pfc@xxxxxxxxxxx
University of Wisconsin-Madison       (608) 265-8936

Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>