Re: [HTCondor-users] Determine when all jobs in a cluster have finished?

[Nathan Panike <nwp@xxxxxxxxxxx>]

On Wed, Jan 30, 2013 at 02:42:07PM +0000, Brian Candler wrote:
> On Wed, Jan 30, 2013 at 09:21:46AM -0500, Brian Pipa wrote:
> > I'd really like the whole thing to be self-contained in one DAG like:
> > ###
> > Job QueryDB querydb.job
> > Job Workers workers.job
> > Job PostProcess postprocess.job
> > PARENT QueryDB CHILD Workers
> > PARENT Workers CHILD PostProcess
> > ###
> > 
> > since that seems much simpler and self-contained but I don't think
> > that's doable since the results of the QueryDB job determines the data
> > and number of worker jobs I'll need.

Two ways of doing this

So you can have QueryDB produce the workers.job, or

add something like

script post QueryDB produceworkers.sh

that will run when the QueryDB cluster finishes, and use data from the
QueryDB to produce workers.job

> 
> I thought I read something along those lines a while back. In principle I
> think it is doable, because workers.job is not processed until the time that
> job is actually submitted.
> 
> I don't know the cleanest way of making this dynamic, but you could try
> simply overwriting the workers.job file from the querydb job.
> 
> Maybe what I am thinking of is this:
> http://research.cs.wisc.edu/htcondor/manual/v7.8/2_10DAGMan_Applications.html#SECTION003107800000000000000
> It says explicitly that a DAG may modify a sub-DAG before it is submitted.

A DAG job (meaning PRE script, POST script, or another DAG node) may
modify another (target) DAG node configuration before the target is
read by DAGMan and submitted to HTCondor.
> 
> Regards,
> 
> Brian.