[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Determine when all jobs in a cluster have finished?



Won't  NOTIFICATION=complete in your job submission do it?
Should email when the cluster is complete - though it may email you when each job completes which you probably don't want ...

--Russell Smithies

-----Original Message-----
From: htcondor-users-bounces@xxxxxxxxxxx [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Brian Pipa
Sent: Wednesday, 30 January 2013 11:18 a.m.
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Determine when all jobs in a cluster have finished?

Short: I'm trying to figure out when all jobs from a job cluster have finished so that I can do some post-processing. I can think of lots of ways for me to code this up, but it seems like there would be some easy way in Condor to do this - does anyone know how?

Long: I have a single Java master task (that is also a Condor job, though that's not relevant) that does a large DB query then splits the results into chunks and submits each chunk to Condor as a job via one ClassAd so they all have the same Cluster id. These jobs are all Java worker jobs that call various tools to process the data. I have all of the output for each worker cluster going to a single directory so it's easy to keep them together and know what output is from which run. As I said above, I can think of a bunch of ways I could code up a solution but it seems like Condor might have a way to tell if a Cluster of jobs has finished or not.  Does anyone know if Condor does have a way to do this?

UPDATE: while typing this email up I found:
condor_q <cluster>
which might work. When I submit the one big worker job, I capture the output from condor_submit and I can parse out the id from that "X
job(s) submitted to cluster Y".  Then, after I submit the job, I can call condor_q Y periodically until it tells me no more jobs are in the q.
or I could call
condor_q Y |grep Y
until I get nothing back.

Does this sounds right/make sense? is there an easier way to do this?
My way seems kind of hacky though I think it should work.

Thanks!
Brian
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

=======================================================================
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
=======================================================================