Re: [HTCondor-users] Avoiding CPU wastage

[Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

Vikrant,

Hereâs the approach I took a couple of years ago for RemoteCpuUtilizationPercent. (Should it be "Pct" instead? Hmm...)

In particular, it takes suspension time into account in its calculations. If I'm remembering correctly the RemoteWallClockTime attribute is not defined until the job is completed, hence the multiple clauses. Because the Remote*Cpu attributes are cumulative, the time factor in calculating overall CPU utilization needs to cover all previous runs (the first clause in TotalExecutingTime) as well as the current run (the JobStatus == 2 clause) to get an accurate percentage.

Due to the nature of some of our workloads, which do some single-threaded work at the beginning of the run before shifting into the simulation phase leveraging all the CPUs, your five-minute window wouldn't work well here (I'm thinking of approaches to build a DAG for them), but when you've got jobs that are supposed to go full throttle right out of the gate, you've come up with a good way to manage their CPU allocations there.

======
TotalExecutingTime = \
  ( ifThenElse(! isUndefined(RemoteWallClockTime), \
        RemoteWallClockTime, 0) - \
    ifThenElse(! isUndefined(CumulativeSuspensionTime), \
        CumulativeSuspensionTime, 0) \
  ) + \
  ( ifThenElse(JobStatus == 2, \
        CurrentTime - JobCurrentStartDate, 0) \
  ) + \
  ( ifThenElse(JobStatus == 7, \
        LastSuspensionTime - JobCurrentStartDate, 0) \
  )

RemoteCpuUtilizationPercent = \
  ifThenElse(! isUndefined(TotalExecutingTime) && TotalExecutingTime > 0, \
    ((RemoteSysCpu + RemoteUserCpu) / RequestCpus) / TotalExecutingTime * 100, \
     UNDEFINED)

RemoteUserCpuUtilizationPercent = \
  ifThenElse(! isUndefined(TotalExecutingTime) && TotalExecutingTime > 0, \
    (RemoteUserCpu / RequestCpus) / TotalExecutingTime * 100, \
     UNDEFINED)

RemoteSysCpuUtilizationPercent = \
  RemoteCpuUtilizationPercent - RemoteUserCpuUtilizationPercent

SUBMIT_EXPRS = $(SUBMIT_EXPRS) TotalExecutingTime RemoteCpuUtilizationPercent \
  RemoteUserCpuUtilizationPercent RemoteSysCpuUtilizationPercent
=====

Michael V. Pelletier
Information Technology
Digital Transformation & Innovation
Integrated Defense Systems
Raytheon Company

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Vikrant Aggarwal
Sent: Thursday, May 16, 2019 7:35 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [External] Re: [HTCondor-users] Avoiding CPU wastage

Hello Collin,

Many thanks for your response. I was not aware of this difference.Â

I noticed and read that RemoteWallClockTime is an accumulative time across all states of job. My requirement was to check the runtime on the particular node and if it's higher then 300s and the CPU utilization is higher than certain threshold put the job on hold status. same for low cpu utilization time hence I replaced the expression like below:

RemoteCpuUtilizationPercent = (((RemoteSysCpu + RemoteUserCpu) / RequestCpus) / (time() - EnteredCurrentStatus) * 100)
periodic_hold = ifthenelse(JobStatus == 2 && (time() - EnteredCurrentStatus) > 300, $(RemoteCpuUtilizationPercent) > 40, False)

### With remotewallclocktime:

a) first hold time wasÂ302.0Â Job released because of periodic_release expression
b) second hold time wasÂ554.0 Job again released because of periodic_release expression

For the second time hold I believe it has taken the value 554 for division, if not then for sure it will take this value for 3rd hold.Â

001 (2600.000.000) 05/16 06:23:07 Job executing on host: <xx.xx.xx.xx:9618?addrs=xx.xx.xx.xx-9618&noUDP&sock=8826_abfe_15>
012 (2600.000.000) 05/16 06:28:07 Job was held.
013 (2600.000.000) 05/16 06:28:34 Job was released.
001 (2600.000.000) 05/16 06:29:02 Job executing on host: <xx.xx.xx.xx:9618?addrs=xx.xx.xx.xx-9618&noUDP&sock=14357_494b_184>
012 (2600.000.000) 05/16 06:33:12 Job was held.
013 (2600.000.000) 05/16 06:33:36 Job was released.
001 (2600.000.000) 05/16 06:33:56 Job executing on host: <xx.xx.xx.xx:9618?addrs=xx.xx.xx.xx-9618&noUDP&sock=28124_7ea5_111>


### With time() - EnteredCurrentStatus

a) First hold timeÂ

$ condor_q 2602.0 -af enteredcurrentstatus remotewallclocktime remoteusercpu remotesyscpu jobstatus                           Â
1558003843 302.0 298.0 1.0 2

b) second hold timeÂ

$ condor_q 2602.0 -af enteredcurrentstatus remotewallclocktime remoteusercpu remotesyscpu jobstatus                           Â
1558004203 603.0 8.0 0.0 2

$ grep '2602' dir0/stress.log | egrep -i 'Job executing|job was held|job was released'
001 (2602.000.000) 05/16 06:45:22 Job executing on host: <xx.xx.xx0.100:9618?addrs=xx.xx.xx0.100-9618&noUDP&sock=28124_7ea5_111>
012 (2602.000.000) 05/16 06:50:22 Job was held.
013 (2602.000.000) 05/16 06:50:31 Job was released.
001 (2602.000.000) 05/16 06:50:44 Job executing on host: <xx.xx.xx0.100:9618?addrs=xx.xx.xx0.100-9618&noUDP&sock=28124_7ea5_111>
012 (2602.000.000) 05/16 06:55:44 Job was held.
013 (2602.000.000) 05/16 06:56:09 Job was released.
001 (2602.000.000) 05/16 06:56:45 Job executing on host: <xx.xx.xx1.58:9618?addrs=xx.xx.xx1.58-9618&noUDP&sock=13681_9b46_184>
012 (2602.000.000) 05/16 07:01:46 Job was held.
013 (2602.000.000) 05/16 07:01:49 Job was released.
001 (2602.000.000) 05/16 07:02:27 Job executing on host: <xx.xx.xx1.150:9618?addrs=xx.xx.xx1.150-9618&noUDP&sock=12768_5bc2_105>

Doing some more testing.Â

Once again thanks for your email.Â


Thanks & Regards,
Vikrant Aggarwal