Re: [HTCondor-users] CondorCE: job submission to Condor-LRMS fails due to stdout/stderr files missing during staging(?)

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

OK, thanks - I will dig a bit deeper into it.

tbh I adapted the rules mostly from our user of the LRMS and had not 
checked for already present rules/routes in the CE.

The problems seem to concentrate on one of our CEs, while another CE the 
same puppet config behaved normal. To get a clean slate, I purged the 
CE's and the Condor's spool directories under /var/lib/condor*/... - 
just in case of a broken remnant info or so.
Since then, I have not noticed another new broken job. Let's see, how it 
develops over the weekend.

Cheers and thanks,
  Thomas

On 05/03/2021 15.54, Brian Lin wrote:
> Hi Thomas,
>
> That's quite strange and certainly shouldn't happen! There should be a 
> plain-text /var/lib/condor/spool/job_queue.log: does that file look 
> corrupted at all?
>
> As for the local SYSTEM_PERIODIC_REMOVE, even though it may not be the 
> culprit here, you should move them to your config and append them to the 
> CE's SYSTEM_PERIODIC_REMOVE to avoid similar issues. And if you're on a 
> new enough version of HTCondor-CE, you should be able to remove a few of 
> the clauses:
>
> - Since at least HTCondor-CE 3, held CE jobs are removed after 24 hrs
> - HTCondor-CE 4.0.0 
> <https://htcondor.github.io/htcondor-ce/releases/#disabled-job-retries-by-default> 
> disables job retries by default
> - HTCondor-CE 5.0.0 
> <https://htcondor.github.io/htcondor-ce/releases/#500> (available as a 
> release candidate [1]) will remove jobs that exceed the configured value 
> of "ROUTED_JOB_MAX_TIME"
>
> Brian
>
> [1] https://research.cs.wisc.edu/htcondor/repo/8.9/el7/rc/
>
> On 3/5/21 5:06 AM, Thomas Hartmann wrote:
> > Hi again,
> >
> > maybe related(??) - I just noticed, that a restart of the condor unit 
> > caused the Schedd to loose all its jobs [1]. Since the restart was 
> > more or less instantaneous, I would have expected the Sched to pick up 
> > its jobs.
> >
> > Cheers,
> > Â Thomas
> >
> > [1]
> > 03/05/21 10:55:11 (pid:3997828) WARNING - Cluster 437906 was deleted 
> > with proc ads still attached to it. This should only happen during 
> > schedd shutdown.
> >
> >
> > [2]
> > Mar 05 10:55:11 grid-htcondorce0.desy.de systemd[1]: Stopping Condor 
> > Distributed High-Throughput-Computing...
> > -- Subject: Unit condor.service has begun shutting down
> > -- Defined-By: systemd
> > -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message tohtcondor-users-request@xxxxxxxxxxx  with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature