[HTCondor-users] Limiting and reporting swap usage

[Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx>]

Dear HTCondor experts,

we are running 8.6.13 with cgroups on CentOS 7.6 and use:
CGROUP_MEMORY_LIMIT_POLICY = hard

Limiting resident memory works very well, but as soon as a user exceeds the requested memory,
the job is pushed into swap. That's fine and exactly what we want, since exceeding the limit a bit and spilling to swap is ok,
and also some usecases exist where jobs come with a lot of rarely accessed memory and again swap is fine. We also use zswap
to reduce disk IO and gain performance in those cases. 

However, there are two problems:
- Jobs with severe bugs leaking memory will eat up all swap, and are only killed after that has happened. 
  Machines almost grind to a halt when that happens. 
- Users do not see their actual usage. They can only check whether the 
  ResidentSetSize_RAW has reached RequestMemory and *assume* they have spilled over to swap. 

We don't want to remove swap due to the aforementioned reasons - in general, it helps. Also, it allows us to consider using "suspend" in the future
for the jobs for which we can do that. 

So my two questions are:
- Is there a possibility to report swap usage / total usage to the users, so they can estimate their resources better? 
  I did not check the origin of what HTCondor currently reports, but I guess it is something like
  memory.max_usage_in_bytes
  of the cgroup. In cgroup terms would be interesting to also get:
  memory.memsw.max_usage_in_bytes
- Is there a possibility to put a hard limit on this value? 
  For example, a resource limit on a job to (at maximum) consume twice as much memory.memsw as memory was reserved. 
  Currently, it seems like memory.memsw.limit_in_bytes is set to something like system memory + swap. 
  Is there an underlying reason for that? 

As it is now, our users sometimes lack information to better express their resource limits,
and we sometimes have operational issues if users submit thousands of jobs with a severe bug (and jobs are only removed after all swap is eaten,
mostly freezing the machines in the meantime). 

Suggestions / solutions welcome :-). 

Cheers,
	Oliver

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature