Re: [HTCondor-users] Determine when all jobs in a cluster have finished?

[Mike Fienen <mike@xxxxxxxxxxx>]

Hi Brian

I ran into a similar issue recently.  One option is to use DAGMan with a single node representing your job.  DAGMan will monitor the job for you and report completion.

Mike

Sent from my iPad

On Jan 29, 2013, at 3:18 PM, Brian Pipa <brianpipa@xxxxxxxxx> wrote:

> Short: I'm trying to figure out when all jobs from a job cluster have
> finished so that I can do some post-processing. I can think of lots of
> ways for me to code this up, but it seems like there would be some
> easy way in Condor to do this - does anyone know how?
> 
> Long: I have a single Java master task (that is also a Condor job,
> though that's not relevant) that does a large DB query then splits the
> results into chunks and submits each chunk to Condor as a job via one
> ClassAd so they all have the same Cluster id. These jobs are all Java
> worker jobs that call various tools to process the data. I have all of
> the output for each worker cluster going to a single directory so it's
> easy to keep them together and know what output is from which run. As
> I said above, I can think of a bunch of ways I could code up a
> solution but it seems like Condor might have a way to tell if a
> Cluster of jobs has finished or not.  Does anyone know if Condor does
> have a way to do this?
> 
> UPDATE: while typing this email up I found:
> condor_q <cluster>
> which might work. When I submit the one big worker job, I capture the
> output from condor_submit and I can parse out the id from that "X
> job(s) submitted to cluster Y".  Then, after I submit the job, I can
> call
> condor_q Y
> periodically until it tells me no more jobs are in the q.
> or I could call
> condor_q Y |grep Y
> until I get nothing back.
> 
> Does this sounds right/make sense? is there an easier way to do this?
> My way seems kind of hacky though I think it should work.
> 
> Thanks!
> Brian
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/