Re: [HTCondor-users] run a script when all jobs complete

[John M Knoeller <johnkn@xxxxxxxxxxx>]

I should point out that this is the purpose of the condor_wait tool.

condor_wait <log_file> will not exit until it sees a completion event for every submit event in that file, so this tool does essentially the same thing as the python script if all of the jobs share a log file.

-tj

-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Justin Killebrew
Sent: Thursday, March 8, 2018 2:37 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] run a script when all jobs complete

Thanks Sandeep.  I went with DAGMan since it's part of the HTCondor world and pretty straigtforward.  However I'm stashing your python code just in case :-)

JK


> On Mar 7, 2018, at 5:46 PM, Sandeep Gupta <sandeep@xxxxxxxxxxxx> wrote:
> 
> Hey,
> 
> I wasn't aware of DAGMan, so I wrote a log processor of the condor log in python:
> 
>   class LogProcessor(object):                                                                                                                                                
>       """Processor for condor logs.                                                                                                                                          
>                                                                                                                                                                              
>       This is a wrapper over HTCondor's EventIterator and keeps track of                                                                                                     
>       state of all the submitted jobs. It can be used as a context manager                                                                                                   
>       in the following way:                                                                                                                                                  
>                                                                                                                                                                              
>       with LogProcessor('job.log', num_jobs) as processor:                                                                                                                   
>           while not processor.done():                                                                                                                                        
>               processor.process()                                                                                                                                            
>                                                                                                                                                                              
>       At the exit, it will log all the details about the submitted job.                                                                                                      
>       """      
> 
> Internally, it essentially calls `htcondor.read_events` from the python api on the condor log. Might be helpful. 
> 
> 
> Sandeep.
> 
> On Wed, Mar 7, 2018 at 3:21 PM, Zach Miller <zmiller@xxxxxxxxxxx> wrote:
> One fairly straightforward way would be to use DAGMan:
>         http://research.cs.wisc.edu/htcondor/manual/v8.7/2_10DAGMan_Applications.html
> 
> Your jobs don't have any dependencies, but DAGMan allows you to specify a "FINAL" node that gets run after everything else.
> 
> There's some other advantages offered by dagman, such as retrying failed jobs, recovery of a partially-completed run, throttling job submission (if you need it), running scripts before and/or after each job, etc.
> 
> 
> Cheers,
> -zach
> 
> 
> > -----Original Message-----
> > From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
> > Of Justin Killebrew
> > Sent: Wednesday, March 07, 2018 2:10 PM
> > To: htcondor-users@xxxxxxxxxxx
> > Subject: [HTCondor-users] run a script when all jobs complete
> >
> > Hello.  I need to run a python (or bash) script when all jobs of a
> > particular submission are complete.  Should I use hooks and
> > HOOK_JOB_CLEANUP?  If so, could someone point me to a simple example since
> > I have not been able to configure it properly.  Is there
> > another/better/easier way?
> >
> > Running HTCondor 8.6.8 on Rocks 7.0
> >
> > Thanks,
> > JK
> >
> > _____________________________
> > Justin Killebrew
> > Senior Programmer/Analyst
> > Zanvyl Krieger Mind/Brain Institute
> > Johns Hopkins University
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited. _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/