Re: [Condor-users] DAG destructor job

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Thu, 16 Oct 2008, Jan Ploski wrote:

> Does DAGMan maintain the DAG's state in memory, or does it re-read the
> .dag file after each subjob's execution?

It maintains state in memory -- the DAG file is only parsed at startup.

> The scenario I would like to implement is as follows:
> in normal case subjobs A, B, C, D are executed
> however, if either of A or B fails, the DAG skips ahead to D (which is a
> cleanup job - should be run regardless of the DAG's success or failure).
>
> One possibility would be to update the .dag file and mark the jobs as done
> in the POST script of A. Do you think that would work? If not, do you have
> any hints on how to best implement a "DAG exception handler (or
> destructor) job"?

That won't work.  There is way to do it that's a little more elaborate,
though -- it makes use of the "nested DAGs" feature.

Here's what you do:

* Put the "work" nodes A, B, C into a DAG, and use the ABORT-DAG-ON
feature to bail out of that DAG if one of the nodes fails.

* Make a top-level DAG that has two nodes -- one is the .condor.sub file
for the lower-level DAG; the second is the cleanup node (D).  For this
to work, the node calling the lower-level DAG has to succeed in order
for the cleanup node to be run.  You can accomplish that by either having
ABORT-DAG-ON declare the lower-level DAG successful (add RETURN 0 to
the ABORT-DAG-ON declarations), or else have a POST script on that node
in the upper-level DAG that always returns 0.


Another way to do it would be to have a POST script on A that turns
B and C into noop jobs if A fails (you'd have to do something like
copy a new submit file over the existing one).  If you do this, it's
critical that the new submit file specify the same log file as the 
original one -- otherwise DAGMan won't "see" the events for that job.

Hopefully this is at least somewhat clear -- let me know if it's not.

Kent Wenger
Condor Team