[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] schedd stopped working (died?) with SchedLog filled with 'WriteUserLog checking for event, log rotation, but no lock'



On 10/17/2016 4:44 AM, Thomas Hartmann wrote:
Job submission failed around 2:00 tonight after which the SchedLog [1]
contained only of lines as
 > WriteUserLog checking for event log rotation, but no lock
which occured before as well but not solely.

A bit later at ~2:16 the MasterLog [2] started to log sched daemons to
be reaped/to die(?) exiting with code 44. Restarts of the schedd went on
for ~20m after which the MasterLog went silent until the service got
restarted.

I found so far no information on the schedd error code 44 but only for
the shadow [3].


Hi Thomas,

For any/all HTCondor daemons, exit code 44 means that the HTCondor daemon in question failed to write to write or rotate a log file. I.e., the write() system call failed. Look at the filesystems holding the paths for LOG, LOCK, and/or EVENT_LOG via
  condor_config_val LOG LOCK EVENT_LOG

The most likely cause for this is the filesystem(s) in question were full, or fell offline (i.e. NFS mount failed if they are not local). BTW, I would encourage your to have these directories on local disk if they are not already.

Hope the above helps
Todd