Re: [HTCondor-users] cgroups issue (cgroup invalid operation)

Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

Hi *,

I fear I might be facing some kind of condor oddity.

It seems to me that I have to restart some daemons in a specific order in order to get working cgroups.

I have confirmed this on several malfunctioning condor startd :

For an unknown reason, /cgroup/cpuacct/htcondor does not contain any slot directory, but /cgroup/cpu/ does.

It might be related to the fact I deployed cgroups on running condor instances, I have no idea how this could have happened :

[root@wn041 ~]# ll /cgroup/cpu*/htcondor/

/cgroup/cpuacct/htcondor/:

total 0

--w--w---- 1 root root 0 Mar 17 12:26 cgroup.event_control

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cgroup.procs

-r--r--r-- 1 root root 0 Mar 17 12:26 cpuacct.stat

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cpuacct.usage

-r--r--r-- 1 root root 0 Mar 17 12:26 cpuacct.usage_percpu

-rw-rw-r-- 1 root root 0 Mar 17 12:26 notify_on_release

-rw-rw-r-- 1 root root 0 Mar 17 12:26 tasks

/cgroup/cpu/htcondor/:

total 0

--w--w---- 1 root root 0 Mar 17 12:26 cgroup.event_control

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cgroup.procs

drwxr-xr-x 2 root root 0 Mar 17 12:26 condor_home_condor_slot1_1@wn041

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cpu.cfs_period_us

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cpu.cfs_quota_us

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cpu.rt_period_us

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cpu.rt_runtime_us

-rw-rw-r-- 1 root root 0 Mar 17 12:26 cpu.shares

-r--r--r-- 1 root root 0 Mar 17 12:26 cpu.stat

-rw-rw-r-- 1 root root 0 Mar 17 12:26 notify_on_release

-rw-rw-r-- 1 root root 0 Mar 17 12:26 tasks

In the ProcLog I found these lines (thatâs on another machine I tested, but same logs) :

03/22/16 16:27:37 : Setting cgroup to htcondor/condor_home_condor_slot1_1@wn001 for ProcFamily 1568537.

03/22/16 16:27:37 : Unable to modify cgroup htcondor/condor_home_condor_slot1_1@wn001. Some cgroup functionality may not work: 50016 Invalid argument

03/22/16 16:27:37 : Cannot attach pid 1568537 to cgroup htcondor/condor_home_condor_slot1_1@wn001 for ProcFamily 1568537: 50002 Cgroup does not exist

03/22/16 16:27:37 : Unable to read cgroup htcondor/condor_home_condor_slot1_1@wn001 cpuacct stats (ProcFamily 1568537): Cgroup invalid operation.

03/22/16 16:27:37 : Internal cgroup error when retrieving CPU statistics: Cgroup invalid operation

03/22/16 16:27:41 : PROC_FAMILY_GET_USAGE

03/22/16 16:27:41 : gathering usage data for family with root pid 1568531

03/22/16 16:27:41 : Unable to read cgroup htcondor/condor_home_condor_slot1_1@wn001 cpuacct stats (ProcFamily 1568537): Cgroup invalid operation.

03/22/16 16:27:41 : Internal cgroup error when retrieving CPU statistics: Cgroup invalid operation

03/22/16 16:27:41 : Unable to read cgroup htcondor/condor_home_condor_slot1_1@wn001 memory stats (ProcFamily 1568537): 50016 No such file or directory.

And these errors remain EVEN IF I restart condor (full service restart, not even a condor_restart).

What I found is that even on service restarts, the slot directories remain in /cgroup/cpu/.

And I have the impression that if they are found, their equivalent directories in /cgroup/cpuacct (or any other subsystem) is not recreated on condor restart

So, in order to get rid of cgroup errors, I have to :

- Stop condor

- Restart the cgconfig service (which unmounts/remounts/cleans /cgroup)

- Start condor

e.g :

[root@wn001 ~]# service condor stop ; service cgconfig restart ; service condor start ; ll /cgroup/cpu*/htcondor

Stopping Condor daemons: [ OK ]

Stopping cgconfig service: [ OK ]

Starting cgconfig service: [ OK ]

Starting Condor daemons: [ OK ]

Once this is done and when a job lands, I then finally get everything apparently working, and the /cgroup/cpuacct directory finally is populated :

[root@wn001 ~]# ll /cgroup/cpu*/htcondor

/cgroup/cpuacct/htcondor:

total 0

--w--w---- 1 root root 0 Mar 22 16:28 cgroup.event_control

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cgroup.procs

drwxr-xr-x 2 root root 0 Mar 22 16:29 condor_home_condor_slot1_1@wn001

-r--r--r-- 1 root root 0 Mar 22 16:28 cpuacct.stat

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cpuacct.usage

-r--r--r-- 1 root root 0 Mar 22 16:28 cpuacct.usage_percpu

-rw-rw-r-- 1 root root 0 Mar 22 16:28 notify_on_release

-rw-rw-r-- 1 root root 0 Mar 22 16:28 tasks

/cgroup/cpu/htcondor:

total 0

--w--w---- 1 root root 0 Mar 22 16:28 cgroup.event_control

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cgroup.procs

drwxr-xr-x 2 root root 0 Mar 22 16:29 condor_home_condor_slot1_1@wn001

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cpu.cfs_period_us

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cpu.cfs_quota_us

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cpu.rt_period_us

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cpu.rt_runtime_us

-rw-rw-r-- 1 root root 0 Mar 22 16:28 cpu.shares

-r--r--r-- 1 root root 0 Mar 22 16:28 cpu.stat

-rw-rw-r-- 1 root root 0 Mar 22 16:28 notify_on_release

-rw-rw-r-- 1 root root 0 Mar 22 16:28 tasks

And no more cgroup error in the logsâ

I now have to find a way to run these restarts without loosing all running jobs (i.e : running a condor_restart command with a gracefull shutdown, but with something between the stop and the start â Iâll check the docs if thereâs a knob for that ;) )

P.S : I also found these :

open("/cgroup/cpu/htcondor/condor_home_condor_slot1_1@wn020 /cpu.stat", O_RDWR|O_CLOEXEC) = 9

fstat(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbb1841d000

write(9, "nr_periods 0\nnr_throttled 0\nthrottled_time 0", 44) = -1 EINVAL (Invalid argument)

close(9) = 0

munmap(0x7fbb1841d000, 4096) = 0

stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2945, ...}) = 0

write(3, "03/22/16 15:53:02 : Unable to modify cgroup htcondor/condor_home_condor_slot1_1@wn020. Some cgroup functionality may not work: 50016 Invalid argument\n", 167) = 167

Ã Writing to cpu.stat seems impossible even as root, I therefore donât know why this is attempted but this explains at least some logs I faced.

Frederic

De : HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] De la part de SCHAER Frederic
EnvoyÃ : vendredi 18 mars 2016 21:40
Ã : HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Objet : [PROVENANCE INTERNET] Re: [HTCondor-users] cgroups issue (cgroup invalid operation)

Ah, I donât know if it can help, but :

- I see memory page allocation failures in the syslogs since I activated the cgroups this week, for instance

attr: page allocation failure. order:4, mode:0xd0

Pid: 3124060, comm: attr Not tainted 2.6.32-xxxx #1

Call Trace:

[<ffffffff811344ba>] ? __alloc_pages_nodemask+0x74a/0x8d0

[<ffffffff811734d2>] ? kmem_getpages+0x62/0x170

[<ffffffff811740ea>] ? fallback_alloc+0x1ba/0x270

[<ffffffff81173b3f>] ? cache_grow+0x2cf/0x320

[<ffffffff81173e69>] ? ____cache_alloc_node+0x99/0x160

[<ffffffff811b667d>] ? getxattr+0x14d/0x170

[<ffffffff81174ab9>] ? __kmalloc+0x199/0x230

[<ffffffff811b667d>] ? getxattr+0x14d/0x170

[<ffffffff8119f6eb>] ? path_walk+0x7b/0xe0

[<ffffffff8119be46>] ? final_putname+0x26/0x50

[<ffffffff8119c44b>] ? putname+0x2b/0x40

[<ffffffff811a0a22>] ? user_path_at+0x62/0xa0

[<ffffffff8115594c>] ? do_brk+0x26c/0x350

[<ffffffff8115302e>] ? remove_vma+0x6e/0x90

[<ffffffff811b66f5>] ? sys_lgetxattr+0x55/0x80

[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

(+lots of memory and cpu informations)

- And I see /cgroup/*/â stat files, e.g :

# cat /cgroup/memory/htcondor/condor_home_condor_slot1_1\@wn296*/memory.stat

cache 4570165248

rss 221777920

mapped_file 1368064

pgpgin 132555261

pgpgout 131555005

swap 16384

inactive_anon 120786944

active_anon 101003264

inactive_file 2430099456

active_file 2140016640

unevictable 0

hierarchical_memory_limit 4792057856

hierarchical_memsw_limit 4792061952

total_cache 4570165248

total_rss 221777920

total_mapped_file 1368064

total_pgpgin 132555261

total_pgpgout 131555005

total_swap 16384

total_inactive_anon 120786944

total_active_anon 101003264

total_inactive_file 2430099456

total_active_file 2140016640

total_unevictable 0

funny, mmmmh ? ;)

Fred

De : HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] De la part de SCHAER Frederic
EnvoyÃ : vendredi 18 mars 2016 21:32
Ã : HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Objet : [PROVENANCE INTERNET] Re: [HTCondor-users] cgroups issue (cgroup invalid operation)

Salut Brian :]

Ah, yes, forgot to give that part of the config : itâs like what I could read in various guides :

[root@wn296 htcondor]# condor_config_val -startd -dump|grep -i cgroup

BASE_CGROUP = htcondor

CGROUP_MEMORY_LIMIT_POLICY = soft

Fred

De : HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] De la part de Brian Bockelman
EnvoyÃ : vendredi 18 mars 2016 21:23
Ã : HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Objet : Re: [HTCondor-users] cgroups issue (cgroup invalid operation)

Hi Frederic,

Whatâs the value of âBASE_CGROUPâ configuration variable? I ask because I would have thought the ProcLog would show something like âUnable to read cgroup /htcondor/ââ instead of âUnable to read cgroup htcondor/ââ. Maybe thereâs a base â/â that is getting automatically added in some places but not others?

Brian

On Mar 18, 2016, at 3:19 PM, SCHAER Frederic <frederic.schaer@xxxxxx> wrote:

Hi,

So, I deployed cgroups for htcondorâ. And Iâm having issues.

First : looks like I have to restart condor, not just condor_reconfig it. (right ?)

Without a service restart, I get logs about âcgroups not initializedâ.

But problem is that even after restarting condor, the ProcsLog shows these errors :

03/18/16 20:59:53 : Unable to read cgroup htcondor/condor_home_condor_slot1_2@wn296 cpuacct stats (ProcFamily 3278233): Cgroup invalid operation.

03/18/16 20:59:53 : Internal cgroup error when retrieving CPU statistics: Cgroup invalid operation

03/18/16 20:59:53 : Unable to read cgroup htcondor/condor_home_condor_slot1_2@wn296 memory stats (ProcFamily 3278233): 50016 No such file or directory.

I am deploying cgroups so that memory accounting (RSS) doesnât double-tripple count stuff, which in the end is causing the jobs to be killed after they are reported as consuming more memory than requested.

Iâm wondering how to fix this ?

To answer coming questions :

-          Yes, cgconfig is running, and cgroups are mounted (but not visible with the mount command â sl6x/sl6.7 here) :

[root@wn296 htcondor]# service cgconfig status

Running

[root@wn296 htcondor]# grep cgroup /proc/mounts

cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0

cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0

cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0

cgroup /cgroup/devices cgroup rw,relatime,devices 0 0

cgroup /cgroup/memory cgroup rw,relatime,memory 0 0

cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0

cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0

cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0

-          Yes, I see htcondor subdirectories, and I even see PIDs in the subdirectories tasks files :

[root@wn296 htcondor]# wc -l /cgroup/memory/htcondor/condor_home_condor_slot1_*/tasks|sed -r -e 's%(wn...).*/%\1/%'

0 /cgroup/memory/htcondor/condor_home_condor_slot1_10@wn296/tasks

0 /cgroup/memory/htcondor/condor_home_condor_slot1_11@wn296/tasks

6 /cgroup/memory/htcondor/condor_home_condor_slot1_1@wn296/tasks

22 /cgroup/memory/htcondor/condor_home_condor_slot1_2@wn296/tasks

22 /cgroup/memory/htcondor/condor_home_condor_slot1_3@wn296/tasks

22 /cgroup/memory/htcondor/condor_home_condor_slot1_4@wn296/tasks

0 /cgroup/memory/htcondor/condor_home_condor_slot1_5@wn296/tasks

0 /cgroup/memory/htcondor/condor_home_condor_slot1_6@wn296/tasks

0 /cgroup/memory/htcondor/condor_home_condor_slot1_7@wn296/tasks

0 /cgroup/memory/htcondor/condor_home_condor_slot1_8@wn296/tasks

0 /cgroup/memory/htcondor/condor_home_condor_slot1_9@wn296/tasks

72 total

-          Cgroups config :

[root@wn296 htcondor]# cat /etc/cgconfig.{conf,d/htcondor.conf}

# This file is being maintained by Puppet.

# DO NOT EDIT

mount {

      cpu     = /cgroup/cpu;

      cpuset = /cgroup/cpuset;

      cpuacct = /cgroup/cpuacct;

      devices = /cgroup/devices;

      memory = /cgroup/memory;

      freezer = /cgroup/freezer;

      net_cls = /cgroup/net_cls;

      blkio   = /cgroup/blkio;

}

group htcondor {

      cpu {}

      cpuacct {}

      memory {}

      freezer {}

      blkio {}

}

-          And condor version : 8.4.3-2

The problem I have is that I have jobs submitted (not by me) with Memory requirements and that these jobs are still killed because of the RSS approximative accounting without cgroups â and for now, the killing is still going on :â(

Any ideas about whatâs wrong, and better, how to fix ? I admin Iâm very new to cgroups, so it might be that Iâm mistaken somewhereâ

Thanks

Frederic

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Mailing List Archives

Public Access

Re: [HTCondor-users] cgroups issue (cgroup invalid operation)