Mailing List ArchivesPublic Access |
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesPublic Access |
|
_______________________________________________Hi,maybe not very helpful but we saw similar behaviour on scheds that were overly busy reordering there state database due to a high number of jobs combined with some typos in submit files etc ...Hence I agree with Joe, the sched logs should shed some light on this ;)Bestchristoph
--
Christoph Beyer
DESY Hamburg
IT-Department
Notkestr. 85
Building 02b, Room 009
22607 Hamburg
phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxxVon: "JOSEPH RYAN REUSS via HTCondor-users" <htcondor-users@xxxxxxxxxxx>
An: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
CC: "JOSEPH RYAN REUSS" <jrreuss@xxxxxxxx>
Gesendet: Donnerstag, 2. November 2023 16:22:49
Betreff: Re: [HTCondor-users] Scheduling delay in cluster mix of 8.8.5 and 9.0.17 versionVikrant,
It looks like around when the Schedd was failing, it was communicating to the negotiator and continued to send jobs. I would look at the NegotiatorLog at that timestamp to see if there is any helpful information there. Also, is "test.example.com" the real name or did you sanitize the log?
Best,Joe Reuss
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Vikrant Aggarwal <ervikrant06@xxxxxxxxx>
Sent: Monday, October 30, 2023 4:11 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Scheduling delay in cluster mix of 8.8.5 and 9.0.17 versionHello Experts,
Sched running 9.0.17 version.HTcondor masters running 8.8.5 version (Primary and all in flock_to list)
Special setup details: We are dynamically modifying the job requirements to give it an opportunity first to run on private pool (team owned pool) if not then on public pool (which is shared by multiple teams) ensuring we are not creating too many autoclusterIDs.
Despite having available cores in both primary and flock pools, the job stays in queue forever until we do the restart of condor service on scheduler.
Sched doesn't present the jobs for matchmaking.10/30/23 13:38:10 0 seconds so far for this submitter 10/30/23 13:38:10 0 seconds so far for this schedd 10/30/23 13:38:10 Got NO_MORE_JOBS; schedd has no more requests
In sched logs, the following message was reported but still after this message it was keep on sending jobs to negotiator for matchmaking.
10/30/23 12:07:01 (pid:43091) condor_write(): Socket closed when trying to write 354 bytes to negotiator test.example.com, fd is 25 10/30/23 12:07:01 (pid:43091) Buf::write(): condor_write() failed 10/30/23 12:07:01 (pid:43091) SECMAN: failed to end classad message 10/30/23 12:07:01 (pid:43091) Failed to send RESCHEDULE to negotiator test.example.com: SECMAN:2007:Failed to end classad message.
10/30/23 12:07:01 (pid:43091) (cid:1237904) actOnJobs: didn't do any work, abortingImmediately before sched stops advertising jobs to negotiators following message reported but doesn't look problematic.10/30/23 12:34:00 (pid:43091) Shadow pid 3450123 for job 37192219.2 exited with status 115 10/30/23 12:34:00 (pid:43091) Match record (slot1@xxxxxxxxxxxxxxxxxxxx <10.xx.xx.xx:9618?addrs=10.xx.xx.xx-9618&alias=testnode.example.com&noUDP&sock=startd_283196_7aaf> for test.user1, 37192219.2) deletedAny thoughts on what could be an issue here?
Thanks & Regards,Vikrant Aggarwal
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/