[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] jobs stuck; cannot get rid of them.

Hi Stephen,

We ran into this too. In our case the condor_starter process that was handling each of those jobs didn't exit properly and was still running. Connecting to the host and killing the stuck condor_starter process fixed the issue.Â

Alternatively, restarting Condor on the hosts will also get rid of anything still running and update the collector.

Hope that helps,

More Information for the curious:

Here's the end of the StarterLog for one of the affected slots:
08/28/18 18:51:52 ChildAliveMsg: failed to send DC_CHILDALIVE to parent daemon at <IP removed> (try 1 of 3): SECMAN:2003:TCP connection to daemon at <IP removed> failed.
08/28/18 18:53:34 ChildAliveMsg: giving up because deadline expired for sending DC_CHILDALIVE to parent.
08/28/18 18:53:34 Process exited, pid=43481, status=0

The pid listed was for the job running on that slot, which successfully exited and finished elsewhere.

We noticed this happening because it was affecting the group accounting during negotiation. The negotiator would allocate the correct number of slots using the number of jobs from the Schedd, but would then skip negotiation for that group because it used the incorrect number of jobs from the collector when determining current resource usage.

Here's an example where the submitter had only one pending 1-core job and no running jobs, but there were two stuck slots with 32-core jobs from that submitter:
11/26/18 12:40:59 group quotas: group= prod.<group removed>Â quota= 511.931Â requested= 1Â allocated= 1Â unallocated= 0
11/26/18 12:40:59 subtree_usage at prod.<group removed>Âis 64
11/26/18 12:41:01 Group prod.<group removed>Â- skipping, at or over quota (quota=511.931) (usage=64) (allocation=1)

On Thu, Nov 29, 2018 at 5:25 AM Stephen Jones <sjones@xxxxxxxxxxxxxxxx> wrote:
Hi all,

There is a discrepancy between what condor_q thinks is runing, and what
condor_status things is running. I run this set of commands to see the

# condor_status -af JobId -af Cpus | grep -v undef | sort | sed -e
"s/\.0//"> s
# condor_q -af ClusterId -af RequestCpus -constraint "JobStatus=?=2"
| sort > q
# diff s q
< 1079641 8
< 1080031 8
< 1080045 8
< 1080321 1

See; condor_status has 4 jobs that actually don't exist in condor_q !?!

They've been there for days, since I had some Linux problems that needed
a reboot (not very related to htcondor.)

So I'm losing 25 slots, due to this. How can I purge this stale
information from the HTCondor system, good and proper?



Steve Jones              Âsjones@xxxxxxxxxxxxxxxx
Grid System Administrator       Âoffice: 220
High Energy Physics Division      tel (int): 43396
Oliver Lodge Laboratory        Âtel (ext): +44 (0)151 794 3396
University of Liverpool        Âhttp://www.liv.ac.uk/physics/hep/

HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Collin Mehring | PE-JoSE - Software Engineer