Mailing List ArchivesPublic Access |
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesPublic Access |
|
|
Hi Todd, On 20/10/2017 17:26, Todd Tannenbaum
wrote:
On 10/20/2017 9:44 AM, Alessandra Forti wrote:I'm using 8.6.6 But I have some answers/suggestions below that I hope will help...let me get this straight if I replace ResidentSetSize_RAW with MemoryUsage it should work? Note that MemoryUsage is in the same units as RequestMemory, so only need to multiply by 2 instead of 2000.I read that the submit machine evaluates the _expression_ every 60 seconds since version 7.4 (though admitedly the blog I read is quite old so things might have changed again (https://spinningmatt.wordpress.com/2009/12/05/cap-job-runtime-debugging-periodic-job-policy-in-a-condor-pool/) A runaway job could consume a lot of memory in a few minutes :).Do you mean I should move SYSTEM_PERIODIC_REMOVE to the WN? or is there another recipe? This I wrote recipe I'm using is used by several other sites. at the moment there are no limits set in cgroups, i.e. the limit number is practically infinite, so either policy - soft or hard - might not work without (OOM doesn't kick in). This is why sites are setting the system_periodic_remove. The machines were stressed because the application was using up to 15 times what it requested. For example using stress I just submitted a job that usses 80GB of memory on a machine that has 64GB RAM [root@wn2208290 ~]# for a in $(seq -w 1 50); do egrep '^rss|^swap' /sys/fs/cgroup/memory/system.slice/condor.service/condor_scratch_condor_pool_condor_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/memory.stat|grep -v huge; sleep 5 ;echo;done rss 65429585920 swap 457154560 [..] rss 65468846080 swap 1864413184 it is happily filling the swap. I don't think removing the swap is a good idea but the sum RAM+swap should be indeed limited to either a multiple of what is requested or a default max limit. If I put a soft limit to 4GB in the job it does bring down the memory to 40GB but the limit doesn't affect the swap which starts increasing at a faster pace. [root@wn2208290 ~]# for a in $(seq -w 1 50); do egrep '^rss|^swap' /sys/fs/cgroup/memory/system.slice/condor.service/condor_scratch_condor_pool_condor_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/memory.stat|grep -v huge; sleep 5 ;echo;done rss 64724119552 swap 16926076928 [....] rss 64367165440 swap 21876707328 however the soft limit is the only thing it lets me set with a brutal echo redirection. The memory general limit and the memsw limit give error echo 4G > /sys/fs/cgroup/memory/system.slice/condor.service/condor_scratch_condor_pool_condor_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/memory.limit_in_bytes -bash: echo: write error: Device or resource busy echo 4G > /sys/fs/cgroup/memory/system.slice/condor.service/condor_scratch_condor_pool_condor_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/memory.memsw.limit_in_bytes -bash: echo: write error: Invalid argument so my plan to set this stuff on the fly doesn't seem feasible. I wonder if any of these condor daemons that actually create the condor jobs process groups could do that at creation time? I'm just at the start with cgroups reading the docs I thought things where quite straightforward but now I'm confused on how it works. cheers alessandra HTCondor sets the OOM priority of job process such that the OOM killer should always pick job processes ahead of other processes on the system. Furthermore, HTCondor "captures" the OOM request to kill a job and only allows it to continue if the job is indeed using more memory than requested (i.e. provisioned in the slot). This is probably what you wanted by setting the limit to soft in the first place. -- Respect is a rational process. \\// Fatti non foste a viver come bruti, ma per seguir virtute e canoscenza(Dante) For Ur-Fascism, disagreement is treason. (U. Eco) But but but her emails... covfefe! |