Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Avoiding CPU wastage

After running the tests I realized that my example is creating a time bomb. Job which was supposed to be completed in 360s, it was continuously going into hold/release status and finally remained in hold status because of FALSE periodic release _expression_. And same is happening with sleep.job. It is not happening with yours. Am I missing something here?Â

$ egrep -v "^(#|$)" stress_wall_1.subÂ
executable       = stress.sh
log          Â= stress.log
output         = outfile$(Process).txt
error         Â= errors$(Process).txt
RemoteCpuUtilizationPercent = (((RemoteSysCpu + RemoteUserCpu) / RequestCpus) / (time() - EnteredCurrentStatus) * 100)
periodic_hold = ifthenelse(JobStatus == 2 && (time() - EnteredCurrentStatus) > 300, $(RemoteCpuUtilizationPercent) < 40, False)
periodic_hold_reason = "Using cpu less than threshold"
periodic_hold_subcode = 30
PeriodicRelease = (JobRunCount < 5 && HoldReasonCode == 3 && $(periodic_hold_subcode) == 30)
notification = Error
when_to_transfer_output = ON_EXIT
Initialdir = dir$(Process)
queue


Thanks & Regards,
Vikrant Aggarwal


On Thu, May 16, 2019 at 5:05 PM Vikrant Aggarwal <ervikrant06@xxxxxxxxx> wrote:
Hello Collin,

Many thanks for your response. I was not aware of this difference.Â

I noticed and read that RemoteWallClockTime is an accumulative time across all states of job. My requirement was to check the runtime on the particular node and if it's higher then 300s and the CPU utilization is higher than certain threshold put the job on hold status. same for low cpu utilization time hence I replaced the _expression_ like below:

RemoteCpuUtilizationPercent = (((RemoteSysCpu + RemoteUserCpu) / RequestCpus) / (time() - EnteredCurrentStatus) * 100)
periodic_hold = ifthenelse(JobStatus == 2 && (time() - EnteredCurrentStatus) > 300, $(RemoteCpuUtilizationPercent) > 40, False)

### With remotewallclocktime:

a) first hold time wasÂ302.0Â Job released because of periodic_release _expression_
b) second hold time wasÂ554.0 Job again released because of periodic_release _expression_

For the second time hold I believe it has taken the value 554 for division, if not then for sure it will take this value for 3rd hold.Â

001 (2600.000.000) 05/16 06:23:07 Job executing on host: <xx.xx.xx.xx:9618?addrs=xx.xx.xx.xx-9618&noUDP&sock=8826_abfe_15>
012 (2600.000.000) 05/16 06:28:07 Job was held.
013 (2600.000.000) 05/16 06:28:34 Job was released.
001 (2600.000.000) 05/16 06:29:02 Job executing on host: <xx.xx.xx.xx:9618?addrs=xx.xx.xx.xx-9618&noUDP&sock=14357_494b_184>
012 (2600.000.000) 05/16 06:33:12 Job was held.
013 (2600.000.000) 05/16 06:33:36 Job was released.
001 (2600.000.000) 05/16 06:33:56 Job executing on host: <xx.xx.xx.xx:9618?addrs=xx.xx.xx.xx-9618&noUDP&sock=28124_7ea5_111>


### With time() - EnteredCurrentStatus

a) First hold timeÂ

$ condor_q 2602.0 -af enteredcurrentstatus remotewallclocktime remoteusercpu remotesyscpu jobstatus                           Â
1558003843 302.0 298.0 1.0 2

b) second hold timeÂ

$ condor_q 2602.0 -af enteredcurrentstatus remotewallclocktime remoteusercpu remotesyscpu jobstatus                           Â
1558004203 603.0 8.0 0.0 2

$ grep '2602' dir0/stress.log | egrep -i 'Job executing|job was held|job was released'
001 (2602.000.000) 05/16 06:45:22 Job executing on host: <xx.xx.xx0.100:9618?addrs=xx.xx.xx0.100-9618&noUDP&sock=28124_7ea5_111>
012 (2602.000.000) 05/16 06:50:22 Job was held.
013 (2602.000.000) 05/16 06:50:31 Job was released.
001 (2602.000.000) 05/16 06:50:44 Job executing on host: <xx.xx.xx0.100:9618?addrs=xx.xx.xx0.100-9618&noUDP&sock=28124_7ea5_111>
012 (2602.000.000) 05/16 06:55:44 Job was held.
013 (2602.000.000) 05/16 06:56:09 Job was released.
001 (2602.000.000) 05/16 06:56:45 Job executing on host: <xx.xx.xx1.58:9618?addrs=xx.xx.xx1.58-9618&noUDP&sock=13681_9b46_184>
012 (2602.000.000) 05/16 07:01:46 Job was held.
013 (2602.000.000) 05/16 07:01:49 Job was released.
001 (2602.000.000) 05/16 07:02:27 Job executing on host: <xx.xx.xx1.150:9618?addrs=xx.xx.xx1.150-9618&noUDP&sock=12768_5bc2_105>

Doing some more testing.Â

Once again thanks for your email.Â

Thanks & Regards,
Vikrant Aggarwal


On Wed, May 15, 2019 at 11:53 PM Collin Mehring <collin.mehring@xxxxxxxxxxxxxx> wrote:
Hi Vikrant,

I think your periodic_hold is evaluating to ERROR when runtime < 300 because of division by zero.

Using the values from condor_q:
((((0 + 8) / 1) / ifthenelse(true && (11) > 300, (11), 0 ) * 100) > 40)
((8 / 0 * 100) > 40)

I'd guess it only worked for the sleep job, and in 8.6, because the _expression_ wasn't evaluated until after the job had been running for 300 seconds.

I would try:
RemoteCpuUtilizationPercent = (((RemoteSysCpu + RemoteUserCpu) / RequestCpus) /ÂRemoteWallClockTime * 100)
periodic_hold = ifthenelse(JobStatus == 2 && RemoteWallClockTime > 300, $(RemoteCpuUtilizationPercent) > 40, False)

Best,
Collin

On Wed, May 15, 2019 at 12:22 AM Vikrant Aggarwal <ervikrant06@xxxxxxxxx> wrote:
Team,

Any inputs to make this work on 8.5.8?

Thanks & Regards,
Vikrant Aggarwal


On Mon, May 13, 2019 at 7:32 PM Vikrant Aggarwal <ervikrant06@xxxxxxxxx> wrote:
This seems to be only not behaving properly with condor version 8.5.8 but in version 8.6.13Â working as expected.

Thanks & Regards,
Vikrant Aggarwal


On Mon, May 13, 2019 at 3:30 PM Vikrant Aggarwal <ervikrant06@xxxxxxxxx> wrote:
Hello Team,

Referring [1], [2] old email threads I am testing in lab to take action on the jobs which are running for more than 180s with both higher and lower CPU utilization.Â

- In following submit file I am generating load using stress and if the CPU utilization goes about .4 then putting the job on hold and releasing it using periodic_release so that it can get schedule on another node. Strangely job is going into hold status within 11s of running as per remotewallclocktime, parameters used for evaluating the _expression_ should return value greater than .4 not sure why hold reason is showing condition is UNDEFINED.

~~~
$ cat stress.shÂ
#!/bin/bash
stress --cpu 1 -t 360

$ cat stress.sub
executable       = sleep.sh
log          Â= stress.log
output         = outfile$(Process).txt
error         Â= errors$(Process).txt
runtime = (time() - JobCurrentStartDate)
TotalExecutingTime = ifthenelse(JobStatus == 2 && $(runtime) > 300, $(runtime), 0)
RemoteCpuUtilizationPercent = (((RemoteSysCpu + RemoteUserCpu) / RequestCpus) / $(TotalExecutingTime) * 100)
periodic_hold = ($(RemoteCpuUtilizationPercent) > 40)
periodic_hold_reason = "Using cpu more than threshold"
periodic_hold_subcode = 30
PeriodicRelease = (JobRunCount < 5 && HoldReasonCode == 3 && $(periodic_hold_subcode) == 30)
should_transfer_files Â= Yes
when_to_transfer_output = ON_EXIT
Initialdir = dir$(Process)
queue


$ condor_q 2564.0


-- Schedd: testmachine : <IPaddress:9618?... @ 05/13/19 05:32:39
OWNERÂ Â ÂBATCH_NAMEÂ Â Â Â SUBMITTEDÂ ÂDONEÂ ÂRUNÂ Â IDLEÂ ÂHOLDÂ TOTAL JOB_IDS
vaggarwal CMD: stress.sh Â5/13 05:30   _   _   _   1   1 2564.0

1 jobs; 0 completed, 0 removed, 0 idle, 0 running, 1 held, 0 suspended

$ condor_q 2566.0 -af holdreason holdreasoncode
The job attribute PeriodicHold _expression_ '( ( ( ( RemoteSysCpu + RemoteUserCpu ) / RequestCpus ) / ifthenelse(JobStatus == 2 && ( time() - JobCurrentStartDate ) > 300,( time() - JobCurrentStartDate ),0) * 100 ) > 40 )' evaluated to UNDEFINED 5

$ condor_q 2564.0 -af RemoteSysCpu RemoteUserCpu RequestCpus remotewallclocktime
0.0 8.0 1 11.0
~~~

- If I am running the same scenario for sleep job, it's working as expected. Job went into hold status 3 times and during the last time because of false PeriodicRelease condition, it remains in hold status.Â

~~~
cat sleep.sh
#!/bin/bash
# file name: sleep.sh

TIMETOWAIT="1020"
echo "sleeping for $TIMETOWAIT seconds"
/bin/sleep $TIMETOWAITÂ

$ cat sleep.subÂ
executable       = sleep.sh
log          Â= sleep.log
output         = outfile$(Process).txt
error         Â= errors$(Process).txt
runtime = (time() - JobCurrentStartDate)
TotalExecutingTime = ifthenelse(JobStatus == 2 && $(runtime) > 300, $(runtime), 0)
RemoteCpuUtilizationPercent = (((RemoteSysCpu + RemoteUserCpu) / RequestCpus) / $(TotalExecutingTime) * 100)
periodic_hold = ($(RemoteCpuUtilizationPercent) < 20)
periodic_hold_reason = "Using cpu less than threshold"
periodic_hold_subcode = 30
PeriodicRelease = (JobRunCount < 5 && HoldReasonCode == 3 && $(periodic_hold_subcode) == 30)
should_transfer_files Â= Yes
when_to_transfer_output = ON_EXIT
Initialdir = dir$(Process)
queue


$ condor_q 2555.0 -af holdreason holdreasoncode
Using cpu less than threshold 3

$ condor_q 2555.0 -af RemoteSysCpu RemoteUserCpu RequestCpus remotewallclocktime
0.0 0.0 1 301.0
~~~


My objective is to hold the job if it's not doing any activity which seems to be working fine but I want to confirm the other way around as well to ensure that behavior is as expected.Â

Some queries related to RemoteSysCpu RemoteUserCpu

- During testing I observed that RemoteSysCpu RemoteUserCpu are getting change only when job status is changed otherwise they remain 0.
- Also are these parameters are accumulative like remotewallclocktime?



Thanks & Regards,
Vikrant Aggarwal
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


--
Collin Mehring | PE-JoSE - Software Engineer

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/