Mailing List Archives
Public Access
|
|
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Limiting HTCondor total RAM usage
- Date: Tue, 24 Feb 2015 17:30:21 +0000
- From: Brian Candler <b.candler@xxxxxxxxx>
- Subject: Re: [HTCondor-users] Limiting HTCondor total RAM usage
On 24/02/2015 16:07, Greg Thain wrote:
Unfortunately, I spoke to soon. What happened then is that oom_killer
started killing processes when I don't think it should.
I can cat memory.usage_in_bytes every second and I see it slowly
creeping up to 1.6G or 1.7G, and then suddenly the process dies, and
dmesg shows a splurge of backtrace of oom_killer output.
Can you share the specific dmesg output about the oom killer?
Sure. One is below. Note: these jobs do a lot of writing (including
mainly output to NFS), and there might be a lot of dirty pages in the
cache to be flushed, which is what the backtrace suggests to me. Do
dirty VFS pages count towards a cgroup's memory utilisation?
However what I don't understand is why I get oom killers with this config:
BASE_CGROUP = htcondor
CGROUP_MEMORY_LIMIT_POLICY = none
but get no oom killers when those two lines are removed. The only
difference then is whether condor creates cgroups for its slots, but
they should be non-enforcing cgroups.
Regards,
Brian.
[ 665.666724] decompressqsi invoked oom-killer: gfp_mask=0x200d2,
order=0, oom_adj=0, oom_score_adj=16
[ 665.666728] decompressqsi
cpuset=condor_var_lib_condor_execute_slot1_24@xxxxxxxxxxxxxxxxxxxx
mems_allowed=0
[ 665.666731] Pid: 7146, comm: decompressqsi Not tainted 3.2.0-4-amd64
#1 Debian 3.2.65-1+deb7u1
[ 665.666733] Call Trace:
[ 665.666742] [<ffffffff810b787a>] ? dump_header+0x78/0x1bd
[ 665.666748] [<ffffffff81350e3f>] ? _raw_spin_unlock_irqrestore+0xe/0xf
[ 665.666752] [<ffffffff8109817e>] ? delayacct_end+0x72/0x7d
[ 665.666756] [<ffffffff811654d2>] ?
security_real_capable_noaudit+0x40/0x4f
[ 665.666761] [<ffffffff81087760>] ?
__cpuset_node_allowed_hardwall+0x47/0x50
[ 665.666764] [<ffffffff81350e3f>] ? _raw_spin_unlock_irqrestore+0xe/0xf
[ 665.666767] [<ffffffff810b7c4f>] ? oom_kill_process+0x49/0x271
[ 665.666769] [<ffffffff810b834a>] ? out_of_memory+0x2ea/0x337
[ 665.666773] [<ffffffff810bbfe2>] ? __alloc_pages_nodemask+0x62a/0x7ab
[ 665.666777] [<ffffffff81027cf3>] ? __x2apic_send_IPI_mask+0x116/0x13b
[ 665.666782] [<ffffffff810e5b2b>] ? alloc_pages_current+0xc7/0xe4
[ 665.666786] [<ffffffff81036618>] ? should_resched+0x5/0x23
[ 665.666790] [<ffffffff81102200>] ? pipe_write+0x23e/0x475
[ 665.666794] [<ffffffff810cef9d>] ? set_pmd+0x5/0x8
[ 665.666798] [<ffffffff810facc8>] ? do_sync_write+0xb4/0xec
[ 665.666801] [<ffffffff811653c5>] ? security_file_permission+0x16/0x2d
[ 665.666804] [<ffffffff810fb3b9>] ? vfs_write+0xa2/0xe9
[ 665.666806] [<ffffffff810fb596>] ? sys_write+0x45/0x6b
[ 665.666810] [<ffffffff81355ed2>] ? system_call_fastpath+0x16/0x1b
[ 665.666811] Mem-Info:
[ 665.666812] Node 0 DMA per-cpu:
[ 665.666815] CPU 0: hi: 0, btch: 1 usd: 0
[ 665.666816] CPU 1: hi: 0, btch: 1 usd: 0
[ 665.666817] CPU 2: hi: 0, btch: 1 usd: 0
[ 665.666819] CPU 3: hi: 0, btch: 1 usd: 0
[ 665.666820] CPU 4: hi: 0, btch: 1 usd: 0
[ 665.666822] CPU 5: hi: 0, btch: 1 usd: 0
[ 665.666823] CPU 6: hi: 0, btch: 1 usd: 0
[ 665.666824] CPU 7: hi: 0, btch: 1 usd: 0
[ 665.666825] CPU 8: hi: 0, btch: 1 usd: 0
[ 665.666827] CPU 9: hi: 0, btch: 1 usd: 0
[ 665.666828] CPU 10: hi: 0, btch: 1 usd: 0
[ 665.666829] CPU 11: hi: 0, btch: 1 usd: 0
[ 665.666830] CPU 12: hi: 0, btch: 1 usd: 0
[ 665.666832] CPU 13: hi: 0, btch: 1 usd: 0
[ 665.666833] CPU 14: hi: 0, btch: 1 usd: 0
[ 665.666834] CPU 15: hi: 0, btch: 1 usd: 0
[ 665.666835] CPU 16: hi: 0, btch: 1 usd: 0
[ 665.666837] CPU 17: hi: 0, btch: 1 usd: 0
[ 665.666838] CPU 18: hi: 0, btch: 1 usd: 0
[ 665.666839] CPU 19: hi: 0, btch: 1 usd: 0
[ 665.666841] CPU 20: hi: 0, btch: 1 usd: 0
[ 665.666842] CPU 21: hi: 0, btch: 1 usd: 0
[ 665.666843] CPU 22: hi: 0, btch: 1 usd: 0
[ 665.666845] CPU 23: hi: 0, btch: 1 usd: 0
[ 665.666846] CPU 24: hi: 0, btch: 1 usd: 0
[ 665.666847] CPU 25: hi: 0, btch: 1 usd: 0
[ 665.666848] CPU 26: hi: 0, btch: 1 usd: 0
[ 665.666850] CPU 27: hi: 0, btch: 1 usd: 0
[ 665.666851] CPU 28: hi: 0, btch: 1 usd: 0
[ 665.666852] CPU 29: hi: 0, btch: 1 usd: 0
[ 665.666853] CPU 30: hi: 0, btch: 1 usd: 0
[ 665.666855] CPU 31: hi: 0, btch: 1 usd: 0
[ 665.666856] Node 0 DMA32 per-cpu:
[ 665.666858] CPU 0: hi: 186, btch: 31 usd: 0
[ 665.666859] CPU 1: hi: 186, btch: 31 usd: 0
[ 665.666860] CPU 2: hi: 186, btch: 31 usd: 0
[ 665.666862] CPU 3: hi: 186, btch: 31 usd: 0
[ 665.666863] CPU 4: hi: 186, btch: 31 usd: 0
[ 665.666864] CPU 5: hi: 186, btch: 31 usd: 0
[ 665.666865] CPU 6: hi: 186, btch: 31 usd: 0
[ 665.666867] CPU 7: hi: 186, btch: 31 usd: 0
[ 665.666868] CPU 8: hi: 186, btch: 31 usd: 0
[ 665.666869] CPU 9: hi: 186, btch: 31 usd: 0
[ 665.666871] CPU 10: hi: 186, btch: 31 usd: 0
[ 665.666872] CPU 11: hi: 186, btch: 31 usd: 0
[ 665.666873] CPU 12: hi: 186, btch: 31 usd: 0
[ 665.666874] CPU 13: hi: 186, btch: 31 usd: 0
[ 665.666876] CPU 14: hi: 186, btch: 31 usd: 0
[ 665.666877] CPU 15: hi: 186, btch: 31 usd: 0
[ 665.666878] CPU 16: hi: 186, btch: 31 usd: 0
[ 665.666880] CPU 17: hi: 186, btch: 31 usd: 30
[ 665.666881] CPU 18: hi: 186, btch: 31 usd: 0
[ 665.666882] CPU 19: hi: 186, btch: 31 usd: 0
[ 665.666884] CPU 20: hi: 186, btch: 31 usd: 0
[ 665.666885] CPU 21: hi: 186, btch: 31 usd: 0
[ 665.666886] CPU 22: hi: 186, btch: 31 usd: 0
[ 665.666887] CPU 23: hi: 186, btch: 31 usd: 0
[ 665.666889] CPU 24: hi: 186, btch: 31 usd: 0
[ 665.666890] CPU 25: hi: 186, btch: 31 usd: 0
[ 665.666891] CPU 26: hi: 186, btch: 31 usd: 0
[ 665.666892] CPU 27: hi: 186, btch: 31 usd: 0
[ 665.666894] CPU 28: hi: 186, btch: 31 usd: 0
[ 665.666895] CPU 29: hi: 186, btch: 31 usd: 0
[ 665.666896] CPU 30: hi: 186, btch: 31 usd: 0
[ 665.666898] CPU 31: hi: 186, btch: 31 usd: 0
[ 665.666898] Node 0 Normal per-cpu:
[ 665.666900] CPU 0: hi: 186, btch: 31 usd: 3
[ 665.666901] CPU 1: hi: 186, btch: 31 usd: 0
[ 665.666903] CPU 2: hi: 186, btch: 31 usd: 3
[ 665.666904] CPU 3: hi: 186, btch: 31 usd: 3
[ 665.666905] CPU 4: hi: 186, btch: 31 usd: 1
[ 665.666906] CPU 5: hi: 186, btch: 31 usd: 3
[ 665.666908] CPU 6: hi: 186, btch: 31 usd: 0
[ 665.666909] CPU 7: hi: 186, btch: 31 usd: 2
[ 665.666910] CPU 8: hi: 186, btch: 31 usd: 2
[ 665.666911] CPU 9: hi: 186, btch: 31 usd: 0
[ 665.666912] CPU 10: hi: 186, btch: 31 usd: 0
[ 665.666914] CPU 11: hi: 186, btch: 31 usd: 30
[ 665.666915] CPU 12: hi: 186, btch: 31 usd: 2
[ 665.666916] CPU 13: hi: 186, btch: 31 usd: 4
[ 665.666918] CPU 14: hi: 186, btch: 31 usd: 2
[ 665.666919] CPU 15: hi: 186, btch: 31 usd: 4
[ 665.666920] CPU 16: hi: 186, btch: 31 usd: 0
[ 665.666922] CPU 17: hi: 186, btch: 31 usd: 29
[ 665.666923] CPU 18: hi: 186, btch: 31 usd: 0
[ 665.666924] CPU 19: hi: 186, btch: 31 usd: 0
[ 665.666926] CPU 20: hi: 186, btch: 31 usd: 0
[ 665.666927] CPU 21: hi: 186, btch: 31 usd: 4
[ 665.666928] CPU 22: hi: 186, btch: 31 usd: 2
[ 665.666929] CPU 23: hi: 186, btch: 31 usd: 2
[ 665.666931] CPU 24: hi: 186, btch: 31 usd: 1
[ 665.666932] CPU 25: hi: 186, btch: 31 usd: 1
[ 665.666933] CPU 26: hi: 186, btch: 31 usd: 0
[ 665.666934] CPU 27: hi: 186, btch: 31 usd: 2
[ 665.666936] CPU 28: hi: 186, btch: 31 usd: 1
[ 665.666937] CPU 29: hi: 186, btch: 31 usd: 3
[ 665.666938] CPU 30: hi: 186, btch: 31 usd: 32
[ 665.666940] CPU 31: hi: 186, btch: 31 usd: 1
[ 665.666944] active_anon:8008254 inactive_anon:43 isolated_anon:0
[ 665.666944] active_file:195745 inactive_file:92438 isolated_file:363
[ 665.666945] unevictable:0 dirty:0 writeback:0 unstable:110
[ 665.666946] free:7922054 slab_reclaimable:11805 slab_unreclaimable:84666
[ 665.666946] mapped:3480 shmem:87 pagetables:21183 bounce:0
[ 665.666948] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15668kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? yes
[ 665.666955] lowmem_reserve[]: 0 1949 32249 32249
[ 665.666958] Node 0 DMA32 free:125220kB min:4076kB low:5092kB
high:6112kB active_anon:1833060kB inactive_anon:0kB active_file:64kB
inactive_file:1160kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1995876kB mlocked:0kB dirty:12kB
writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:1324kB
slab_unreclaimable:14180kB kernel_stack:64kB pagetables:3700kB
unstable:12kB bounce:0kB writeback_tmp:0kB pages_scanned:3580
all_unreclaimable? yes
[ 665.666966] lowmem_reserve[]: 0 0 30300 30300
[ 665.666968] Node 0 Normal free:62940kB min:63412kB low:79264kB
high:95116kB active_anon:30152284kB inactive_anon:92kB
active_file:12196kB inactive_file:16544kB unevictable:0kB
isolated(anon):0kB isolated(file):1452kB present:31027200kB mlocked:0kB
dirty:0kB writeback:132kB mapped:1712kB shmem:216kB
slab_reclaimable:13596kB slab_unreclaimable:269052kB kernel_stack:2280kB
pagetables:62312kB unstable:428kB bounce:0kB writeback_tmp:0kB
pages_scanned:59998 all_unreclaimable? yes
[ 665.666976] lowmem_reserve[]: 0 0 0 0
[ 665.666978] Node 0 DMA: 1*4kB 0*8kB 1*16kB 0*32kB 2*64kB 1*128kB
1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15892kB
[ 665.666984] Node 0 DMA32: 80*4kB 205*8kB 141*16kB 106*32kB 248*64kB
73*128kB 11*256kB 1*512kB 5*1024kB 1*2048kB 20*4096kB = 125240kB
[ 665.666990] Node 0 Normal: 2533*4kB 1113*8kB 1000*16kB 308*32kB
215*64kB 43*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 68252kB
[ 665.666996] 287658 total pagecache pages
[ 665.666997] 0 pages in swap cache
[ 665.666998] Swap cache stats: add 0, delete 0, find 0/0
[ 665.666999] Free swap = 0kB
[ 665.667000] Total swap = 0kB
[ 665.848067] 16777200 pages RAM
[ 665.848069] 255489 pages reserved
[ 665.848070] 367434 pages shared
[ 665.848071] 8260659 pages non-shared
[ 665.848072] [ pid ] uid tgid total_vm rss cpu oom_adj
oom_score_adj name
[ 665.848110] [ 861] 0 861 5475 460 21 -17
-1000 udevd
[ 665.848113] [ 1321] 0 1321 5474 346 18 -17
-1000 udevd
[ 665.848115] [ 1324] 0 1324 5474 347 19 -17
-1000 udevd
[ 665.848119] [ 2829] 0 2829 4744 212 18
0 0 rpcbind
[ 665.848122] [ 2840] 102 2840 5837 324 2
0 0 rpc.statd
[ 665.848124] [ 2862] 0 2862 6351 244 21
0 0 rpc.idmapd
[ 665.848127] [ 3390] 0 3390 29743 582 0
0 0 rsyslogd
[ 665.848129] [ 3484] 0 3484 3272 154 25
0 0 mdadm
[ 665.848132] [ 3522] 0 3522 4169 38 0
0 0 atd
[ 665.848134] [ 3570] 0 3570 5103 260 18
0 0 cron
[ 665.848136] [ 3628] 0 3628 1030 151 8
0 0 acpid
[ 665.848138] [ 3648] 106 3648 7452 71 25
0 0 dbus-daemon
[ 665.848140] [ 3843] 104 3843 12144 1270 1
0 0 snmpd
[ 665.848142] [ 3884] 0 3884 12484 282 11 -17
-1000 sshd
[ 665.848144] [ 3916] 101 3916 11704 247 0
0 0 exim4
[ 665.848147] [ 3952] 0 3952 17815 548 1
0 0 sshd
[ 665.848149] [ 3959] 0 3959 5161 887 13
0 0 bash
[ 665.848151] [ 4068] 104 4068 2393 448 1
0 0 snmpd-mdraid-co
[ 665.848153] [ 4127] 107 4127 9768 428 18
0 0 ntpd
[ 665.848155] [ 4155] 0 4155 4065 226 8
0 0 getty
[ 665.848157] [ 4156] 0 4156 4065 225 1
0 0 getty
[ 665.848159] [ 4157] 0 4157 4065 226 9
0 0 getty
[ 665.848161] [ 4158] 0 4158 4065 228 28
0 0 getty
[ 665.848163] [ 4159] 0 4159 4065 226 0
0 0 getty
[ 665.848165] [ 4160] 0 4160 4065 227 11
0 0 getty
[ 665.848167] [ 5485] 0 5485 24169 797 28
0 0 condor_master
[ 665.848169] [ 5486] 0 5486 5958 1481 6
0 0 condor_procd
[ 665.848171] [ 5487] 0 5487 24965 1839 22
0 0 condor_startd
[ 665.848174] [ 6073] 0 6073 17815 885 1
0 0 sshd
[ 665.848176] [ 6077] 0 6077 5180 911 8
0 0 bash
[ 665.848178] [ 6159] 104 6159 2398 474 19
0 0 snmpd-smartctl-
[ 665.848181] [ 6947] 0 6947 23930 1759 4 0
-16 condor_starter
[ 665.848183] [ 6957] 1012 6957 20792 11282 28
0 16 python
[ 665.848185] [ 6977] 0 6977 23930 1762 5 0
-16 condor_starter
[ 665.848187] [ 6986] 0 6986 23930 1762 7 0
-16 condor_starter
[ 665.848189] [ 6995] 1012 6995 20792 11299 4
0 16 python
[ 665.848191] [ 7004] 1012 7004 20792 11299 1
0 16 python
[ 665.848201] [ 7007] 1012 7007 4415 367 19
0 16 bash
[ 665.848204] [ 7008] 1012 7008 213460 205852 9
0 16 filterzqsi
[ 665.848206] [ 7009] 1012 7009 32833 252 15
0 16 decompressqsi
[ 665.848208] [ 7010] 1012 7010 238619 172724 11
0 16 qsitorsiandspli
[ 665.848210] [ 7015] 1012 7015 4415 366 22
0 16 bash
[ 665.848212] [ 7016] 1012 7016 4415 367 6
0 16 bash
[ 665.848214] [ 7017] 1012 7017 213460 205828 21
0 16 filterzqsi
[ 665.848217] [ 7018] 1012 7018 32833 291 2
0 16 decompressqsi
[ 665.848218] [ 7019] 1012 7019 238619 214757 3
0 16 qsitorsiandspli
[ 665.848220] [ 7020] 1012 7020 213460 205813 15
0 16 filterzqsi
[ 665.848223] [ 7021] 1012 7021 32833 289 8
0 16 decompressqsi
[ 665.848225] [ 7022] 1012 7022 238619 215675 11
0 16 qsitorsiandspli
[ 665.848227] [ 7026] 0 7026 23930 1763 13
0 -16 condor_starter
[ 665.848229] [ 7035] 1012 7035 20792 11302 15
0 16 python
[ 665.848231] [ 7045] 0 7045 23930 1765 16
0 -16 condor_starter
[ 665.848233] [ 7058] 1012 7058 20792 11301 3
0 16 python
[ 665.848235] [ 7061] 1012 7061 4415 366 25
0 16 bash
[ 665.848237] [ 7062] 1012 7062 213460 205827 23
0 16 filterzqsi
[ 665.848239] [ 7063] 1012 7063 32833 259 7
0 16 decompressqsi
[ 665.848241] [ 7064] 1012 7064 238619 82527 0
0 16 qsitorsiandspli
[ 665.848243] [ 7066] 1012 7066 4415 366 18
0 16 bash
[ 665.848245] [ 7067] 1012 7067 213460 205865 6
0 16 filterzqsi
[ 665.848247] [ 7068] 1012 7068 32833 257 18
0 16 decompressqsi
[ 665.848249] [ 7069] 1012 7069 238619 161585 18
0 16 qsitorsiandspli
[ 665.848251] [ 7072] 0 7072 23930 1765 6 0
-16 condor_starter
[ 665.848262] [ 7081] 1012 7081 20792 11301 11
0 16 python
[ 665.848264] [ 7100] 1012 7100 4415 366 24
0 16 bash
[ 665.848266] [ 7101] 1012 7101 213460 205798 12
0 16 filterzqsi
[ 665.848268] [ 7102] 1012 7102 32833 261 10
0 16 decompressqsi
[ 665.848270] [ 7103] 1012 7103 238619 182830 20
0 16 qsitorsiandspli
[ 665.848272] [ 7109] 0 7109 23930 1766 16
0 -16 condor_starter
[ 665.848274] [ 7117] 0 7117 23930 1765 0 0
-16 condor_starter
[ 665.848277] [ 7135] 1012 7135 20792 11301 11
0 16 python
[ 665.848279] [ 7136] 1012 7136 20792 11300 15
0 16 python
[ 665.848281] [ 7144] 1012 7144 4415 367 26
0 16 bash
[ 665.848282] [ 7145] 1012 7145 213460 205864 25
0 16 filterzqsi
[ 665.848284] [ 7146] 1012 7146 32833 273 9
0 16 decompressqsi
[ 665.848286] [ 7147] 1012 7147 238619 134726 8
0 16 qsitorsiandspli
[ 665.848288] [ 7152] 1012 7152 4415 366 30
0 16 bash
[ 665.848290] [ 7153] 1012 7153 213460 205813 2
0 16 filterzqsi
[ 665.848292] [ 7154] 1012 7154 32833 265 3
0 16 decompressqsi
[ 665.848294] [ 7155] 1012 7155 238619 135585 5
0 16 qsitorsiandspli
[ 665.848296] [ 7156] 0 7156 23930 1765 16
0 -16 condor_starter
[ 665.848298] [ 7163] 0 7163 23930 1765 18
0 -16 condor_starter
[ 665.848300] [ 7172] 1012 7172 20792 11302 8
0 16 python
[ 665.848302] [ 7175] 0 7175 23930 1766 0 0
-16 condor_starter
[ 665.848304] [ 7176] 1012 7176 20792 11301 14
0 16 python
[ 665.848306] [ 7193] 1012 7193 20792 11303 8
0 16 python
[ 665.848308] [ 7197] 1012 7197 4415 366 29
0 16 bash
[ 665.848310] [ 7198] 1012 7198 213460 205817 4
0 16 filterzqsi
[ 665.848312] [ 7199] 1012 7199 32833 268 22
0 16 decompressqsi
[ 665.848314] [ 7200] 1012 7200 238619 108655 0
0 16 qsitorsiandspli
[ 665.848316] [ 7201] 1012 7201 4415 367 31
0 16 bash
[ 665.848318] [ 7202] 1012 7202 213460 205808 6
0 16 filterzqsi
[ 665.848320] [ 7203] 1012 7203 32833 287 2
0 16 decompressqsi
[ 665.848322] [ 7204] 1012 7204 238619 110709 3
0 16 qsitorsiandspli
[ 665.848324] [ 7207] 1012 7207 4415 366 28
0 16 bash
[ 665.848325] [ 7208] 1012 7208 213460 205832 9
0 16 filterzqsi
[ 665.848327] [ 7209] 1012 7209 32833 295 15
0 16 decompressqsi
[ 665.848329] [ 7210] 1012 7210 238619 79891 10
0 16 qsitorsiandspli
[ 665.848332] [ 7211] 0 7211 23930 1767 0 0
-16 condor_starter
[ 665.848334] [ 7224] 1012 7224 20792 11301 11
0 16 python
[ 665.848335] [ 7230] 0 7230 23930 1766 0 0
-16 condor_starter
[ 665.848337] [ 7243] 1012 7243 20792 11302 1
0 16 python
[ 665.848339] [ 7249] 1012 7249 4415 367 18
0 16 bash
[ 665.848341] [ 7250] 1012 7250 213460 205829 3
0 16 filterzqsi
[ 665.848343] [ 7251] 1012 7251 32833 283 6
0 16 decompressqsi
[ 665.848345] [ 7252] 1012 7252 238619 70850 29
0 16 qsitorsiandspli
[ 665.848347] [ 7253] 0 7253 23930 1767 0 0
-16 condor_starter
[ 665.848350] [ 7264] 1012 7264 20792 11298 9
0 16 python
[ 665.848352] [ 7267] 1012 7267 4415 367 16
0 16 bash
[ 665.848354] [ 7268] 1012 7268 213460 205844 15
0 16 filterzqsi
[ 665.848356] [ 7269] 1012 7269 32833 301 13
0 16 decompressqsi
[ 665.848358] [ 7270] 1012 7270 238619 83781 29
0 16 qsitorsiandspli
[ 665.848360] [ 7274] 0 7274 23930 1766 6 0
-16 condor_starter
[ 665.848362] [ 7283] 1012 7283 20792 11301 4
0 16 python
[ 665.848364] [ 7299] 1012 7299 4415 366 31
0 16 bash
[ 665.848366] [ 7301] 1012 7301 213460 205828 4
0 16 filterzqsi
[ 665.848368] [ 7302] 1012 7302 32833 283 0
0 16 decompressqsi
[ 665.848370] [ 7303] 1012 7303 238619 84577 2
0 16 qsitorsiandspli
[ 665.848372] [ 7306] 0 7306 23930 1765 1 0
-16 condor_starter
[ 665.848374] [ 7315] 1012 7315 20792 11301 4
0 16 python
[ 665.848376] [ 7325] 1012 7325 4415 366 18
0 16 bash
[ 665.848378] [ 7326] 1012 7326 213460 205819 14
0 16 filterzqsi
[ 665.848380] [ 7327] 1012 7327 32833 296 10
0 16 decompressqsi
[ 665.848382] [ 7328] 1012 7328 238619 84308 9
0 16 qsitorsiandspli
[ 665.848384] [ 7331] 0 7331 23930 1767 5 0
-16 condor_starter
[ 665.848386] [ 7340] 1012 7340 20792 11303 1
0 16 python
[ 665.848388] [ 7343] 1012 7343 4415 367 18
0 16 bash
[ 665.848390] [ 7344] 1012 7344 213460 205819 23
0 16 filterzqsi
[ 665.848392] [ 7345] 1012 7345 32833 268 4
0 16 decompressqsi
[ 665.848394] [ 7346] 1012 7346 238619 84379 0
0 16 qsitorsiandspli
[ 665.848396] [ 7347] 1012 7347 4415 366 22
0 16 bash
[ 665.848398] [ 7348] 1012 7348 213460 205829 16
0 16 filterzqsi
[ 665.848400] [ 7349] 1012 7349 32833 300 1
0 16 decompressqsi
[ 665.848402] [ 7350] 1012 7350 238619 109918 2
0 16 qsitorsiandspli
[ 665.848404] [ 7362] 0 7362 23930 1767 6 0
-16 condor_starter
[ 665.848406] [ 7371] 1012 7371 20792 11303 9
0 16 python
[ 665.848408] [ 7379] 1012 7379 4415 366 29
0 16 bash
[ 665.848410] [ 7380] 1012 7380 213460 205840 30
0 16 filterzqsi
[ 665.848412] [ 7381] 1012 7381 32833 285 30
0 16 decompressqsi
[ 665.848413] [ 7382] 1012 7382 238619 40424 10
0 16 qsitorsiandspli
[ 665.848415] [ 7386] 0 7386 23930 1767 0 0
-16 condor_starter
[ 665.848417] [ 7387] 0 7387 23930 1766 2 0
-16 condor_starter
[ 665.848419] [ 7402] 1012 7402 20792 11300 10
0 16 python
[ 665.848421] [ 7403] 1012 7403 20792 11301 12
0 16 python
[ 665.848423] [ 7418] 1012 7418 4415 365 27
0 16 bash
[ 665.848425] [ 7419] 1012 7419 4415 357 25
0 16 bash
[ 665.848427] [ 7421] 1012 7421 213460 205820 6
0 16 filterzqsi
[ 665.848429] [ 7420] 1012 7420 213460 205846 11
0 16 filterzqsi
[ 665.848431] [ 7422] 1012 7422 32833 291 1
0 16 decompressqsi
[ 665.848433] [ 7423] 1012 7423 32833 248 10
0 16 decompressqsi
[ 665.848435] [ 7424] 1012 7424 238619 101797 26
0 16 qsitorsiandspli
[ 665.848437] [ 7425] 1012 7425 238619 54217 24
0 16 qsitorsiandspli
[ 665.848439] [ 7432] 0 7432 23930 1767 1 0
-16 condor_starter
[ 665.848441] [ 7437] 0 7437 23930 1766 0 0
-16 condor_starter
[ 665.848442] [ 7450] 1012 7450 20792 11302 8
0 16 python
[ 665.848444] [ 7451] 1012 7451 20792 11302 9
0 16 python
[ 665.848446] [ 7471] 0 7471 23930 1766 6 0
-16 condor_starter
[ 665.848448] [ 7480] 1012 7480 4415 367 25
0 16 bash
[ 665.848450] [ 7481] 1012 7481 4415 367 24
0 16 bash
[ 665.848452] [ 7482] 1012 7482 213460 205841 2
0 16 filterzqsi
[ 665.848454] [ 7483] 1012 7483 32833 305 17
0 16 decompressqsi
[ 665.848456] [ 7484] 1012 7484 238619 52699 2
0 16 qsitorsiandspli
[ 665.848458] [ 7485] 1012 7485 167310 159609 4
0 16 filterzqsi
[ 665.848460] [ 7486] 1012 7486 32832 284 11
0 16 decompressqsi
[ 665.848462] [ 7487] 1012 7487 238619 33229 8
0 16 qsitorsiandspli
[ 665.848464] [ 7490] 1012 7490 20792 11302 20
0 16 python
[ 665.848466] [ 7497] 0 7497 23930 1765 0 0
-16 condor_starter
[ 665.848468] [ 7504] 0 7504 23930 1765 0 0
-16 condor_starter
[ 665.848470] [ 7511] 1012 7511 4415 367 26
0 16 bash
[ 665.848472] [ 7512] 1012 7512 149358 141652 8
0 16 filterzqsi
[ 665.848474] [ 7513] 1012 7513 32832 229 2
0 16 decompressqsi
[ 665.848475] [ 7514] 1012 7514 238619 33230 22
0 16 qsitorsiandspli
[ 665.848478] [ 7519] 1012 7519 20792 11302 12
0 16 python
[ 665.848479] [ 7520] 1012 7520 20792 11302 13
0 16 python
[ 665.848481] [ 7535] 1012 7535 4415 367 28
0 16 bash
[ 665.848483] [ 7536] 1012 7536 199287 191556 15
0 16 filterzqsi
[ 665.848485] [ 7537] 1012 7537 32832 280 10
0 16 decompressqsi
[ 665.848487] [ 7538] 1012 7538 238619 33231 1
0 16 qsitorsiandspli
[ 665.848489] [ 7539] 1012 7539 4415 367 25
0 16 bash
[ 665.848491] [ 7540] 1012 7540 123585 115854 21
0 16 filterzqsi
[ 665.848493] [ 7541] 1012 7541 32832 282 29
0 16 decompressqsi
[ 665.848495] [ 7542] 1012 7542 238619 33230 2
0 16 qsitorsiandspli
[ 665.848497] [ 7546] 0 7546 23930 1766 2 0
-16 condor_starter
[ 665.848499] [ 7555] 1012 7555 20791 11298 9
0 16 python
[ 665.848501] [ 7568] 1012 7568 4415 367 18
0 16 bash
[ 665.848503] [ 7569] 1012 7569 213204 205250 0
0 16 filterzqsi
[ 665.848505] [ 7570] 1012 7570 32832 280 3
0 16 decompressqsi
[ 665.848507] [ 7571] 1012 7571 238619 33238 5
0 16 qsitorsiandspli
[ 665.848509] [ 7575] 0 7575 23930 1766 1 0
-16 condor_starter
[ 665.848511] [ 7584] 1012 7584 19645 9655 12
0 16 python
[ 665.848513] [ 7594] 0 7594 23930 1766 1 0
-16 condor_starter
[ 665.848515] [ 7603] 1012 7603 20792 11340 14
0 16 python
[ 665.848517] [ 7607] 0 7607 23930 1766 22
0 -16 condor_starter
[ 665.848519] [ 7614] 1012 7614 4415 366 26
0 16 bash
[ 665.848521] [ 7615] 1012 7615 37752 30006 27
0 16 filterzqsi
[ 665.848522] [ 7616] 1012 7616 32832 245 26
0 16 decompressqsi
[ 665.848524] [ 7617] 1012 7617 39631 963 27
0 16 qsitorsiandspli
[ 665.848526] [ 7618] 1012 7618 796 1 6
0 16 python