[HTCondor-users] How to set a hard limit to condor job slots

["wuwj@xxxxxxxxx" <wuwj@xxxxxxxxx>]

Dear All, 

Recently it happened at our site, some  user submitted a batch of jobs which has request_memory=4096,  request_cpus=1, but the job actually uses 10-16GB memory, but condor did not kill these jobs, 

so eventually  those jobs  "killed" the work nodes (the memory and swap are 100% used). 

I check the cgroup setup [1] by condor for the job slot(single core job slot). It seems it only set a soft limit (4GB, request_memory by user job), and the hard limit is 105G . 

And according to [2], it seems very difficult to trigger the kill with only the soft limit when the hard limit is so high, and oom_kill_disable =1 (disabled oom_kill), and under_oom=0 (no out of memory control)

I wonder Is there anyway we can use to protect our work nodes from being killed by jobs exceeding memory usages?

[1]

[root@c-2-38 condor_var_lib_condor_execute_slot1_6@xxxxxxxxxxxxxxxx]# cat memory.limit_in_bytes 

105078718464

[root@c-2-38 condor_var_lib_condor_execute_slot1_6@xxxxxxxxxxxxxxxx]# cat memory.soft_limit_in_bytes 

4294967296

[root@c-2-38 condor_var_lib_condor_execute_slot1_6@xxxxxxxxxxxxxxxx]# more memory.oom_control 

oom_kill_disable 1

under_oom 0

[2]

https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

7. Soft limits
Soft limits allow for greater sharing of memory. The idea behind soft limits
is to allow control groups to use as much of the memory as needed, provided
a. There is no memory contention
b. They do not exceed their hard limit
When the system detects memory contention or low memory, control groups
are pushed back to their soft limits. If the soft limit of each control
group is very high, they are pushed back as much as possible to make
sure that one control group does not starve the others of memory.
Please note that soft limits is a best-effort feature; it comes with
no guarantees, but it does its best to make sure that when memory is
heavily contended for, memory is allocated based on the soft limit
hints/setup. Currently soft limit based reclaim is set up such that
it gets invoked from balance_pgdat (kswapd).

---

wuwj@xxxxxxxxx