[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Taking action on job breaching scratch space limit



Did you also set PREEMPT? 

>From the manual
Configuration Macros — HTCondor Manual 8.9.13 documentation

WANT_HOLD : ... When True and the value of PREEMPT becomes True ...

Think of WANT_HOLD as a modifier on PREEMPT.  It says what you want to do with the job after it is preempted.

-tj


From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of ervikrant06@xxxxxxxxx <ervikrant06@xxxxxxxxx>
Sent: Tuesday, April 6, 2021 9:36 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Taking action on job breaching scratch space limit
 
Hello Experts,

I am trying to take action on the job using more scratch space than allocated.  

CondorDiskSpace attribute is coming from the dynamic script which is called through condor cron. Script is included before putting this configuration. 

CondorDiskSpace = $(CondorDiskSpace:1024)
DiskPerCore =  $(CondorDiskSpace) / $(NUM_CPUS)
chunksdisk = ifThenElse( RequestDisk <= ($(DiskPerCore) * RequestCpus), quantize(RequestCpus, {1}), quantize(RequestDisk, {$(DiskPerCore)}) / $(DiskPerCore))
MODIFY_REQUEST_EXPR_REQUESTDISK = $(chunksdisk) * $(DiskPerCore)

Commented the last two lines in above configuration and added following to keep the test case simple. 

MODIFY_REQUEST_EXPR_REQUESTDISK = $(DiskPerCore) / 160
 STARTD_JOB_ATTRS = $(STARTD_JOB_ATTRS) TowerTeam DiskUsage  

Confirmed from the output of condor_who -l that Disk and DiskUsage appears in output. However, still using scratch space more than allocated, the job keeps on running.  

DiskUsageBreach = (DiskUsage > Disk)
WANT_HOLD = ($(PREEMPT) || $(DiskUsageBreach))

How can we put the job on hold in this scenario once the job uses more disk space than allocated from scratch directory? 

Thanks & Regards,
Vikrant Aggarwal