Re: [Condor-users] any way to dynamically submit dependent jobs?

[Mark Silberstein <marks@xxxxxxxxxxxxxxxxxxxxxxx>]

Just as Peter wrote, you should have a static skeleton of the DAG ready,
but DAGman doesn't check the submission file until the job is actually
submitted ( besides "log" parameter in .sub files, which by the way
should be surrounded with spaces in order for DAGman to be able to read
it correctly. It has some more subtleties, so read the manual about it)
That means that you can simply regenerate the submission files of the
DAG nodes which have not yet been submitted, during the DAG execution.

I would use the following DAG:

JOB A a.sub
JOB GenerateB generate.sub
JOB B B.sub
SCRIPT PRE B regenerateB
PARENT A GenerateB CHILD B

If you say that generation of jobs takes significant amount of time, you
may want to do it in separate job: GenerateB
Once A and GenerateB complete, PRE Script of B will regenerate B.sub
according to the results of the two previous jobs. And then B will be
submitted with the new submission file.
Mark


On Thu, 2005-05-12 at 15:00 -0700, White, John K wrote:
> hi condor users,
> 
> greetings to the list. i have a use case that i'm struggling to fit into
> condor. i'm a condor novice and i hope this is not an rtfm-type
> question. i have a sequence of jobs. they must be executed in a certain
> order. so far so good, but the complication arises in that the job
> sequence is not fully-formed when the first job is submitted. this is
> because the process of generating each job takes a significant amount of
> time, and the details of subsequent jobs such as their Requirements are
> unknown until they are generated. to maximize efficiency i would like to
> submit a job A so that it can run while i'm generating a job B. the idea
> is that when i'm done generating B, i would submit it under the
> condition that it can't start until A finishes (if A is still running).
> i am aware of dagman, but as far as i can tell it does not satisfy the
> use case as the complete graph must be defined prior to A's submit time.
> am i crazy or missing the obvious? any advice is greatly appreciated.
> 
> -jkw
> 
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users