[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] any way to dynamically submit dependent jobs?

There's one important thing to note when using POST scripts to generate later nodes' submit files in a DAG -- since POST scripts may be executed more than once in case of failure, be careful that they are idempotent and do the right thing in that case (i.e., don't just append lines to an existing submit file, etc).


On May 14, 2005, at 9:21 AM, Mark Silberstein wrote:

Just as Peter wrote, you should have a static skeleton of the DAG ready,
but DAGman doesn't check the submission file until the job is actually
submitted ( besides "log" parameter in .sub files, which by the way
should be surrounded with spaces in order for DAGman to be able to read
it correctly. It has some more subtleties, so read the manual about it)
That means that you can simply regenerate the submission files of the
DAG nodes which have not yet been submitted, during the DAG execution.

I would use the following DAG:

JOB A a.sub
JOB GenerateB generate.sub
JOB B B.sub
SCRIPT PRE B regenerateB

If you say that generation of jobs takes significant amount of time, you
may want to do it in separate job: GenerateB
Once A and GenerateB complete, PRE Script of B will regenerate B.sub
according to the results of the two previous jobs. And then B will be
submitted with the new submission file.

Peter Couvares                        University of Wisconsin-Madison
Condor Project Research               Department of Computer Sciences
pfc@xxxxxxxxxxx                       1210 W. Dayton St. Rm #4241
(608) 265-8936                        Madison, WI 53706-1685