[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Using cgroups to limit job memory



On 4/1/2015 9:20 AM, Roderick Johnstone wrote:
Hi

I'm using HTCondor 8.2.7 on Redhat 6.6 and have set up cgroups as per
the manual so that jobs with many processes cannot take too much memory.
I have CGROUP_MEMORY_LIMIT_POLICY = hard

When I specify eg request_memory=100M in the job submit file the job is
indeed limited to 100M of resident memory.

While this behaviour is good for the machine owner, its less than ideal
for the job owner since the job may continue but only very slowly since
its paging a lot. This condition might not be obvious to the job owner.

Although this seems to be the behaviour documented in the manual, I'm
sure I have seen a description of a configuration in which the job can
be placed on hold with a suitable message if it tries to allocate more
memory than it requests, although I can't find that now.


The HOWTO recipes are your friend. From the HTCondor.org homepage look for "HOWTO recipes"; the direct link is
  https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToAdminRecipes

Specifically I think you'll find this one of interest
  https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToLimitMemoryUsage
as it gives examples on how to preempt and/or place jobs on hold that use too much memory.

Also likely of interest is
  https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToLimitCpuUsage

Hope the above helps. Also interested in any thoughts you may have to improve the above HOWTOs.

regards,
Todd