Re: [HTCondor-users] A couple DAGman questions

[Kent WENGER <wenger@xxxxxxxxxxx>]

I have a three questions I’ve had trouble answering regarding DAGman.

Q 1) It seems in the past it has been impossible to set a macro for use by all jobs in a DAG. Has this changed? Looks like in 2011 it was not possible (https://lists.cs.wisc.edu/archive/htcondor-users/2011-April/msg00130.shtml)

For example, I would like to do something like this:

ubername=“foo”

JOB A foo.sub

VARS A=$(ubername)

JOB B bar.sub

VARS B=$(ubername)

 ….

A1:

There's no way now to have that kind of macro substitution.  But we're working on having the option to have a VARS value apply to all nodes -- probably something like this:

  VARS all_nodes arg="foo"

Q 2) What is the bet way to return a value from one job and use it as input for another?

Like this?

JOB A foo.sub

SCRIPT POST A bar.sh $RETURN

JOB B bar.sub

VARS B=$(RETURN)

In other words, I want to run foo.sub, then get some information from bar.sh based on the results of foo.sub, and pass that information on to bar.sub.

A2:

Right now there's no DAGMan-supported way to do that, either.  (I'll make a ticket for that, though, because I'm pretty sure you're not the first person to request it.)

The only way to do it right now would be something like having the POST script of node A save the return value to some file, and then have a PRE script for B read that file and stuff that into node B's submit file (or else have node B directly read the file).

Of course, this is mostly only relevant if node A's POST script changes a non-zero return value to a zero value, because if B is a descendant of A, and node A as a whole fails, node B will never get run.  And if B is not a descendant of A, B might get run before A finishes, in which case we won't know the return value yet.

Q 3) Is there a better way to provide a list of variables to a JOB in a DAG?

I’m doing it like this:

JOB A foo.sub

VARS A junkus=“hello”

VARS A punkus=“world”

…

That seems to work, but wondering if there is another better syntax like a dictionary in Python style or such.

A3:

You can put multiple name/value pairs on one line, like this:

VARS A junkus=“hello” punkus=“world”

That's the closest to what you want.

Kent Wenger

CHTC Team