Re: [HTCondor-users] htcondor cgroups and memory limits on CentOS7

[Thomas Patrick Downes <downes@xxxxxxx>]

Also worth pointing out: there are many more âintermediate performanceâ storage technologies coming out that are going to blur the line between RAM and disk.

e.g.

https://arstechnica.com/information-technology/2017/03/intels-first-optane-ssd-375gb-that-you-can-also-use-as-ram/

Some (like the above) will appear like RAM and others like a fast swap disk. It is worth it for the HTCondor team to keep abreast of these developments because they may be necessary for extremely large memory footprint applications.

Tom

On October 23, 2017 at 11:12:42 AM, Thomas Patrick Downes (downes@xxxxxxx) wrote:

On October 23, 2017 at 11:03:33 AM, Alessandra Forti (alessandra.forti@xxxxxxx) wrote:

Most of the problem is the kernel itself: cgroups v1 controllers donât regulate swap separately from RAM. It regulates the combined footprint. So if you set a hard limit on RAM of 4G and a hard limit on RAM+swap of 4G, you could have 0G in RAM and 4G in swap. Itâs just how it is.

true but that would affect only the job itself and not the other jobs. i.e. at least I can confine the job in its 4GB rather than letting it run wild.

Well, I typically find that swap I/O impacts many aspects of machine performance, not just the job itself. But maybe you have dedicated swap or all-flash storage.

But, right now, the limit for RAM+SWAP is set to something like all the RAM+SWAP in the system.

I think they heard us when we complained and that solutions are forthcoming but you should understand that the bulk of the problem is in the kernel itself. The only good swap is dead swap.

fair enough, perhaps that's where the future lies, but at the moment the swap still has a role to play in the linux kernel memory management and it is effectively cheaper than doubling the memory it sounds a bit absurd that there is no way to limit it.

Much of early 20th century literature was about documenting lifeâs absurdity. The 21st century seems to be holding its own in this respect.

FWIW: I agree fully, Condor should set swap settings better than it does and I think they will soon. But everyone should understand clearly: the 1 and only 1 way to eliminate swap usage is to eliminate swap. As I understand it, the original cgroups implementation chose to regulate RAM and swap simultaneously because it was easier to get the kernel patches accepted.

V2 of the memory controller does regulate swap separately and will let you eliminate swap usage on a cgroup-by-cgroup basis. Thatâs probably a year or two away from being a part of RHEL, but itâs built-in to Fedora and Debian now.

Tom