[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Prefetch all-to-one inter-dependencies in DAGMan

On 12/17/2012 03:15 PM, R. Kent Wenger wrote:
On Mon, 17 Dec 2012, Walid Saad wrote:

Hi all,
I want to submit to condor scheduler the following DAG (attached).

JOB  A  A.condor
JOB  B  B.condor
JOB  C  C.condor
JOB  D  D.condor


The output file of jobs A, B, C are the input file for the  job D.
According to the job DAGMan the job D will be submitted  if all jobs A, B
and  C are completed.
My question is as follows:
Is there a way in Condor to submit the job D as soon as one of jobs A,
B or
C is finished?

Right now DAGMan doesn't support a way to do this.  (There's a ticket
requesting this kind of feature; for some reason, I can't access the
gittrac tickets right now, so I can't give you a link.)

The only way I can think of to implement this would be a lot of work:
you'd have to have POST scripts for jobs A, B, and C that would
condor_rm the other jobs in that "set", and also mark the node as
successful if the job was removed.

Kent Wenger


Is the use case: (0) completion of A, B or C is sufficient for D to complete, i.e. D only needs the output from one of A, B or C; or, (1) completion of A, B or C is sufficient for D to start, i.e. D itself will wait for completion of A, B and C, but can start processing output as soon as any of them completes.

There's a variation on use case (1), where D can only process A then B then C instead of some random ordering, e.g. B C A or C A B.

For (0), you can use Kent's trick with POST. For (1), if you have shared storage, you can make A B C and D into siblings.