[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] A couple DAGman questions

From: Michael Fienen <mike@xxxxxxxxxxx>
Date: 07/26/2016 05:05 PM

> My issue is, the results of Job A will influence what queue I want for JOB
> B. Your idea of using POST from A to write a file the PRE for B will
> generally work but can I then rewrite the sub file for B with the PRE
> SCRIPT such that I update the queue for B.

> Specifically, in this case, Job A is splitting a large file into chunks to
> be run in parallel by job B. Butâthe code in Job A breaks into a difficult
> to predict number of such chunks.

> Thanks, and I hope that makes sense.

> Cheers,

> mike

Sounds like you should check out the utterly sweet new "queue" options,
"queue from" "queue in" and "queue matching" introduced in the 8.4

In this case, you'd want the "queue matching" statement:

matching - Each item in the <list of items with file globbing> will be
matched against the names of files and directories relative to the
current directory; the set of matching names is the resulting list of

So your Job B would just do a "queue matching" against whatever Job A
produced, and would automatically queue the right number of jobs based
entirely on the list of files Job A produced.

I love this new syntax, it makes so many things so much simpler.

        -Michael Pelletier.