Our HTCondor-CE is eating up all the memory of the system, constantly increasing. When the memory used arrives to 65-70% (it's a 32GB RAM machine), we start to see the next errors in the SchedLog:
Create_Thread: fork() failed: Cannot allocate memory (12)
ForkWorker::Fork: Fork failed
And all the submitted jobs remain in Hold state (Hold reason: Spooling input data files).
The issue is solved reloading the condor-ce services but, then, the memory starts to increase again, constantly and slowly.
I would like to know if you are facing similar problems with your HTcondor-CEs and how you solve them.
Nowadays, we are running HTCondor 8.5.6 and HTCondor-CE 2.0.7-1.
Thank you in advance.
-- Carles Acosta i Silva PIC (Port d'InformaciÃ CientÃfica) Campus UAB, Edifici D E-08193 Bellaterra, Barcelona Tel: +34 93 581 33 22 Fax: +34 93 581 41 10 http://www.pic.es AvÃs - Aviso - Legal Notice: http://www.ifae.es/legal.html