[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Extracting dag and job run-time information

Hi Duncan,

We do gather some runtime statistics that can probably help you here.

First off, starting in version 8.7.4, we display a bunch of DAGMan runtime statistics in the .dagman.out file, related to how fast DAGMan is actually submitting jobs. We've found that when dealing with lots of short running jobs (on the order of seconds or minutes), DAGMan spends most of its time sitting idle when it could be sending new jobs. If you notice that SubmitCycleTimeSum is dramatically lower than SleepCycleTimeSum, then that's the case here (if so, let me know and I'll send some suggestions).

The script you've used to scrape job running times works very well, but you're right, extending it to work correctly for held/evicted jobs would be a lot of work. I think you're better off using condor_history.

With condor_history, you can get information for all of a DAG's jobs using a constraint. The following looks up all children of $DagJobID$ and displays their individual IDs + running time (RemoteWallClockTime). You can customize the list of attributes as you see fit:

condor_history -constraint "DAGManJobId == $DagJobID$" -af:lh ClusterId DAGManJobId RemoteWallClockTime

Another advantage to this approach is there's probably another attribute describing the job types. I don't know how you tell the different types apart, but if you specify it in the submit files, you should see it somewhere in condor_history.

Hope this helps! Let me know if you have any other questions,


On Wed, Mar 7, 2018 at 12:24 PM, Duncan Meacher <duncan.meacher@xxxxxxx> wrote:
Hi all,

Sorry if this has been asked before, I've tried searching around but haven't seen anything. I'm trying to do some profiling on our dags to get a better idea of where bottle necks are occurring. For this I'm just looking at the run time of all jobs within a dag and histogram them by job type. The current dags I'm looking at contains ~42,000 individual jobs and ~30 unique job types. My current method for this is to search through the dag.dagman.out file for the submit time and completion times to calculate the duration:

# Collect two dictionaries of all job start and stop times
jobstartdict = {}
jobstopdict = {}
for line in open(options.input_file):
    # 09/20/16 22:11:04 Event: ULOG_SUBMIT for HTCondor Node gstlal_inspiral_0001 (472899.0.0) {09/20/16 22:09:49}
    if "ULOG_SUBMIT" in line:
        date, time, _, _, _, _, _, jobname, _, _, _ = line.split()
        jobstartdict[jobname] = datetime.strptime("%s %s" % (date, time), "%x %X")
    # 09/20/16 22:12:45 Node gstlal_inspiral_0001 job proc (472899.0.0) completed successfully.Â
    if "completed successfully" in line:
        date, time, _, jobname, _, _, _, _, _ = line.split()
        jobstopdict[jobname] = datetime.strptime("%s %s" % (date, time), "%x %X")

# Collect jobs that have both start and stop times
correctKeys = set(jobstartdict.keys()) & set(jobstopdict.keys())

# Calculate each job duration
duration = {}
for k in correctKeys:
    duration[k] = (jobstopdict[k] - jobstartdict[k]).total_seconds()

This method works 99% of the time, when the dag/jobs behave normally, however this doesn't work in the case that jobs are either placed on hold or go back into idle after being submitted and running. I'm looking for a better way to get the Run Time of each job.

I'm looking at condor_history, but I can't output exactly what I need. I can get information for all jobs submitted by a user, but need all jobs just from a single dag. I know I can also use -userlog on the nodes.log file, and use the condor ID numbers, but this also has issues if the dag is brought and and then re-submitted, as a new nodes.log file is then created.Â

Any help on this would be greatly appreciated.

Thanks, Duncan


Duncan Meacher, PhD
Postdoctoral Researcher
Institute for Gravitation and the Cosmos
Department of Physics
Pennsylvania State University
104 Davey Lab #040
University Park, PA 16802
Tel: +1 814 865 3243

HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Mark Coatsworth
Systems Programmer
Center for High Throughput Computing
Department of Computer Sciences
University of Wisconsin-Madison
+1 608 206 4703