[HTCondor-users] Memory leak in collectd in 8.8.5?

[Chris Brew - UKRI STFC <chris.brew@xxxxxxxxxx>]

Hi,

This may well be due to something odd in my config since I would have thought someone else would have seen it but after upgrading to 8.8.5 weâve been getting occasions where the collector stop collecting with many:

10/28/19 08:29:33 Create_Thread: fork() failed: Cannot allocate memory (12)
10/28/19 08:29:33 ERROR: Create_Thread failed trying to fork a QueryWorker!

In its log file.

Iâve doubled the memory (now at 8GB) on the Collector/Negotiator VM twice in case it just needed more space to work in but the ganglia plots show monotonically increasing memory usage by the collector until thereâs just less than half the system memory left and it all just stops.

This is 8.8.5 on SL6 x86_64:

# uname -a
Linux heplnv148.pp.rl.ac.uk 2.6.32-754.23.1.el6.x86_64 #1 SMP Mon Sep 23 04:00:04 CDT 2019 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep condor
condor-procd-8.8.5-1.el6.x86_64
condor-external-libs-8.8.5-1.el6.x86_64
condor-classads-8.8.5-1.el6.x86_64
python2-condor-8.8.5-1.el6.x86_64
condor-8.8.5-1.el6.x86_64

Any ideas what in my config could be causing this or additional diagnostics you want?

Thanks,
Chris.

--
Dr Chris Brew
Scientific Computing Manager
Particle Physics Department
UKRI - STFC - Rutherford Appleton Laboratory
Harwell Oxford,
Didcot
OX11 0QX
+44 1235 446326