Re: [HTCondor-users] A couple DAGman questions

[Michael Fienen <mike@xxxxxxxxxxx>]

Hi Kent

Thanks for the quick answers! All makes sense. One remaining question on Q2, though.

My issue is, the results of Job A will influence what queue I want for JOB B. Your idea of using POST from A to write a file the PRE for B will generally work but can I then rewrite the sub file for B with the PRE SCRIPT such that I update the queue for B.

Specifically, in this case, Job A is splitting a large file into chunks to be run in parallel by job B. Butâthe code in Job A breaks into a difficult to predict number of such chunks.

Thanks, and I hope that makes sense.

Cheers,

mike

On Jul 26, 2016, at 3:51 PM, Kent WENGER <wenger@xxxxxxxxxxx> wrote:

I have a three questions Iâve had trouble answering regarding DAGman.

Q 1) It seems in the past it has been impossible to set a macro for use by all jobs in a DAG. Has this changed? Looks like in 2011 it was not possible (https://lists.cs.wisc.edu/archive/htcondor-users/2011-April/msg00130.shtml)

For example, I would like to do something like this:

ubername=âfooâ

JOB A foo.sub

VARS A=$(ubername)

JOB B bar.sub

VARS B=$(ubername)

 â.

A1:

There's no way now to have that kind of macro substitution.  But we're working on having the option to have a VARS value apply to all nodes -- probably something like this:

  VARS all_nodes arg="foo"

Q 2) What is the bet way to return a value from one job and use it as input for another?

Like this?

JOB A foo.sub

SCRIPT POST A bar.sh $RETURN

JOB B bar.sub

VARS B=$(RETURN)

In other words, I want to run foo.sub, then get some information from bar.sh based on the results of foo.sub, and pass that information on to bar.sub.

A2:

Right now there's no DAGMan-supported way to do that, either.  (I'll make a ticket for that, though, because I'm pretty sure you're not the first person to request it.)

The only way to do it right now would be something like having the POST script of node A save the return value to some file, and then have a PRE script for B read that file and stuff that into node B's submit file (or else have node B directly read the file).

Of course, this is mostly only relevant if node A's POST script changes a non-zero return value to a zero value, because if B is a descendant of A, and node A as a whole fails, node B will never get run.  And if B is not a descendant of A, B might get run before A finishes, in which case we won't know the return value yet.

Q 3) Is there a better way to provide a list of variables to a JOB in a DAG?

Iâm doing it like this:

JOB A foo.sub

VARS A junkus=âhelloâ

VARS A punkus=âworldâ

â

That seems to work, but wondering if there is another better syntax like a dictionary in Python style or such.

A3:

You can put multiple name/value pairs on one line, like this:

VARS A junkus=âhelloâ punkus=âworldâ

That's the closest to what you want.

Kent Wenger

CHTC Team

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/