[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] CondorCE: job submission to Condor-LRMS fails due to stdout/stderr files missing during staging(?)

Hi Brian,

I have attached a tarball of all job_queue.* on our CE grid-htcondorce0.
One odd thing(?) might be, that on both our prod CEs, there are each one additional rotated(?) job_queue.log.{5,12} - so somewhat with a "random" number. Both have (c)mod timestamps from when I restarted the condor.service unit - no idea, why these ended up as remnants?

Anyway, the current job_queue.log as well as the pre-restart one look good to me - with the job IDs + their ads well formatted (on a quick grep I dod not find any non-ascii char, that might point to some corruption.


root@grid-htcondorce1: [~] ls -all /var/lib/condor/spool/job_queue.log*
-rw------- 1 condor condor 15030781 Mar 8 16:33 /var/lib/condor/spool/job_queue.log -rw------- 1 condor condor 33740967 Mar 8 13:57 /var/lib/condor/spool/job_queue.log.12

root@grid-htcondorce0: [/etc/condor/config.d] ls -hall /var/lib/condor/spool/job_queue.log* -rw------- 1 condor condor 7,6M Mar 8 16:36 /var/lib/condor/spool/job_queue.log -rw------- 1 condor condor 3,3M Mar 8 13:55 /var/lib/condor/spool/job_queue.log.5

On 05/03/2021 15.54, Brian Lin wrote:
Hi Thomas,

That's quite strange and certainly shouldn't happen! There should be a plain-text /var/lib/condor/spool/job_queue.log: does that file look corrupted at all?

As for the local SYSTEM_PERIODIC_REMOVE, even though it may not be the culprit here, you should move them to your config and append them to the CE's SYSTEM_PERIODIC_REMOVE to avoid similar issues. And if you're on a new enough version of HTCondor-CE, you should be able to remove a few of the clauses:

- Since at least HTCondor-CE 3, held CE jobs are removed after 24 hrs
- HTCondor-CE 4.0.0 <https://htcondor.github.io/htcondor-ce/releases/#disabled-job-retries-by-default> disables job retries by default - HTCondor-CE 5.0.0 <https://htcondor.github.io/htcondor-ce/releases/#500> (available as a release candidate [1]) will remove jobs that exceed the configured value of "ROUTED_JOB_MAX_TIME"


[1] https://research.cs.wisc.edu/htcondor/repo/8.9/el7/rc/

On 3/5/21 5:06 AM, Thomas Hartmann wrote:
Hi again,

maybe related(??) - I just noticed, that a restart of the condor unit caused the Schedd to loose all its jobs [1]. Since the restart was more or less instantaneous, I would have expected the Sched to pick up its jobs.


03/05/21 10:55:11 (pid:3997828) WARNING - Cluster 437906 was deleted with proc ads still attached to it. This should only happen during schedd shutdown.

Mar 05 10:55:11 grid-htcondorce0.desy.de systemd[1]: Stopping Condor Distributed High-Throughput-Computing...
-- Subject: Unit condor.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

HTCondor-users mailing list
To unsubscribe, send a message tohtcondor-users-request@xxxxxxxxxxx  with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature