Re: [HTCondor-users] htcondor cgroups and memory limits on CentOS7

[Alessandra Forti <Alessandra.Forti@xxxxxxx>]

On 23/10/2017 17:12, Thomas Patrick Downes wrote:

On October 23, 2017 at 11:03:33 AM, Alessandra Forti (alessandra.forti@xxxxxxx) wrote:

Most of the problem is the kernel itself: cgroups v1 controllers donât regulate swap separately from RAM. It regulates the combined footprint. So if you set a hard limit on RAM of 4G and a hard limit on RAM+swap of 4G, you could have 0G in RAM and 4G in swap. Itâs just how it is.

true but that would affect only the job itself and not the other jobs. i.e. at least I can confine the job in its 4GB rather than letting it run wild.

Well, I typically find that swap I/O impacts many aspects of machine performance, not just the job itself. But maybe you have dedicated swap or all-flash storage.

it does affect the machine performance but not as much as if the job could occupy all the memory and swap as I could do with the earlier example (and as the users have done unwittingly last week). Most of the times jobs that start swapping wildly do so because either they are leaking or shouldn't be running at all and beyond 4GB*cores or a loose multiple of that they should be killed anyway.
Â

But, right now, the limit for RAM+SWAP is set to something like all the RAM+SWAP in the system.

I think they heard us when we complained and that solutions are forthcoming but you should understand that the bulk of the problem is in the kernel itself. The only good swap is dead swap.

fair enough, perhaps that's where the future lies, but at the moment the swap still has a role to play in the linux kernel memory management and it is effectively cheaper than doubling the memory it sounds a bit absurd that there is no way to limit it.

Much of early 20th century literature was about documenting lifeâs absurdity. The 21st century seems to be holding its own in this respect.

FWIW: I agree fully, Condor should set swap settings better than it does and I think they will soon. But everyone should understand clearly: the 1 and only 1 way to eliminate swap usage is to eliminate swap. As I understand it, the original cgroups implementation chose to regulate RAM and swap simultaneously because it was easier to get the kernel patches accepted.

V2 of the memory controller does regulate swap separately and will let you eliminate swap usage on a cgroup-by-cgroup basis. Thatâs probably a year or two away from being a part of RHEL, but itâs built-in to Fedora and Debian now.

Tom

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

-- 
Respect is a rational process. \\//
Fatti non foste a viver come bruti, ma per seguir virtute e canoscenza(Dante)
For Ur-Fascism, disagreement is treason. (U. Eco)
But but but her emails... covfefe!