I think there are a couple points to discuss here:
1. Should HTCondor have a CGROUP_MEMORY_LIMIT policy called âsoftâ?
2. Should HTCondor set cgroups soft memory limits?
Question #1 is poorly posed and Iâll address question #2 below. The cgroups (v1) memory controller poses 2 question to the user/htcondor:
A. At what point do you want me to start reclaiming your unnecessary memory usage?
B. At what point do you want me to kill your job itself
(A) is the soft limit and will cause the kernel to remove cached files and that sort of thing from memory. (B) is the hard limit which behaves as you/weâd expect it to.
The fundamental mistake is that HTCondor is only answering 1 of these questions when the kernel is asking it to answer both. You should have neither a âhardâ nor âsoftâ policy but a set of knobs that lets you answer these questions separately.
I also think you are being doctrinal with respect to hard limits. The following statements are true:
* The memory footprint of some jobs are literally unknowable. They may follow a statistical distribution with no a priori way of asserting the footprint of any particular job. Multi-threaded applications may be scheduled in such a way that even deterministic memory footprints vary.
*Users are unlikely to know the memory footprint even if it is a predictable quantity
You will find that the cgroups v2 philosophy is much less oriented around a hard limit. In fact, it changes to a set of 3 levels:
(Low) if your footprint is below this value, only subject to reclaiming if literally every other cgroup has already been subjected to reclaiming
(High) if you go above this limit, subject to heavy reclaim pressure (roughly the current soft limit)
(Max) Never, ever go above this limit. OOM kill.
(Swap Max) Separate accounting for swap with only a single hard limit enforced
Youâre better buying into what the kernel is actually doing rather than what you think it ought to be doing. In the future, it will be doing even less of what you think it ought to be doing.
On October 20, 2017 at 11:29:19 AM, Todd Tannenbaum (tannenba@xxxxxxxxxxx) wrote: