[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] request_memory
- Date: Fri, 27 Dec 2013 16:11:59 -0600
- From: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] request_memory
On 12/26/2013 3:57 PM, Rita wrote:
I have a job with request_memory = 256 Megabytes but if I go over it will
get held according to my SYSTEM_PERIODIC_HOLD policy. I would like to
automatically triple the request_memory and then release the job. Is that
possible to do?
Warning - this is off the top of my head, but I think the following
would work or at least point you in the right direction. Also I am
assuming you are using HTCondor 8.0 or above.
I would expect your condor_config file has entries like the following:
# Tell the schedd to hold jobs that have been restarted more than
# 10 times, or if it uses more memory than it requested
SYSTEM_PERIODIC_HOLD = JobRunCount > 10 || MemoryUsage > RequestMemory
# Set a specific HoldReasonSubCode so if the job uses
# more memory than request, so we can identify this particular
# hold reason in automatic release expressions. With the below
# expression, the job's HoldReasonSubCode will be 1 if
# the job is on hold due to memory issues, and 0 otherwise.
SYSTEM_PERIODIC_HOLD_SUBCODE = MemoryUsage > RequestMemory
Then in your job submit file you could try something like:
executable = foo
# HoldReasonSubCode of 26 means SYSTEM_PERIODIC_HOLD became true,
# and HoldReasonSubCode of 1 is configured in condor_config to mean
# job was put on hold due to memory usage.
periodic_release = HoldReasonCode =?= 26 && HoldReasonSubCode =?= 1
# Request 256 MB ram unless MemoryUsage (from a previous run) is
# already defined, in which case triple it for the next try.
request_memory = ifthenelse(isUndefined(MemoryUsage),256,3*MemoryUsage)
Hope the above makes sense - the manual has documentation on all the
above knobs and job classad attributes.
Note that if you have control of the configuration of the execute
machines, consider having the execute machine itself put the job on hold
if memory usage exceeds requested memory. This way the job will be put
on hold much sooner after memory usage is exceeded (perhaps even
instantly). For some HOWTO recipes, see