[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Condor-users] Debugging condor_preen problems
- Date: Mon, 21 Aug 2006 12:06:19 -0400
- From: "Ian Chesal" <ICHESAL@xxxxxxxxxx>
- Subject: [Condor-users] Debugging condor_preen problems
I have a handful of new Windows machines that were running jobs fine for
a time and then slowly, one by one, the VM's stopped accepting new jobs.
They're just sitting there in the U+I state. Looking in on the box it
appears the disk where executions are occurring is full to the point
where new jobs can't find enough space on the box to execute. These are
big disk drives (80GB) so there's a lot of accumulated left over junk.
Hundreds of dir_<pid> directories are left in my d:\abc\condor\execute
I noticed in my MasterLog on the machine that condor_preen was running,
but only taking about 5 seconds to execute:
8/21 11:51:59 Preen pid is 624
8/21 11:52:04 DaemonCore: Command received via UDP from host
8/21 11:52:04 DaemonCore: received command 60011 (DC_NOP), calling
8/21 11:52:04 Child 624 died, but not a daemon -- Ignored
So I ran it by hand with '-r -m -v' and it certainly found all the left
over execute directories in d:\abc\condor\execute but for every
directory it said OK instead of removing the directory. My preen
PREEN_INTERVAL = 1800
PREEN_ARGS = -r -m -v
PREEN_ADMIN = $(CONDOR_ADMIN)
VALID_SPOOL_FILES = job_queue.log, job_queue.log.tmp, history,
INVALID_LOG_FILES = core
Pretty standard. And preen is working on many other Windows machines.
Just not these.
Is there a way to get more information out of condor_preen so I can
figure out why it thinks these hundreds of left over directories are
okay and is not cleaning up the junk?
Ian R. Chesal <ichesal@xxxxxxxxxx>
Senior Software Engineer
Toronto Technology Center
Tel: (416) 926-8300