[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] How to lower job lease duration on worker failure?



Sorry. I misunderstood what you were asking.  

 

Yes a smaller value of JobLeaseDuration will allow a job to be rescheduled more quickly when an execute node disappears unexpectedly.

 

The SUBMIT_ATTRS config knob is only read by condor_submit, not by the Schedd, so it is never necessary to reconfigure any daemons to have changes to SUBMIT_ATTRS take effect â but changes to this knob will not affect jobs already in the Schedd.  

 

HOWEVER - If the config knob JOB_DEFAULT_LEASE_DURATION is set then itâs value will be used by condor_submit when the user does not specify a value of in their submit file.    The default value for JOB_DEFAULT_LEASE_DURATION is 2400.

 

A value of JobLeaseDuration set by SUBMIT_ATTRS will be overridden by any value that the user specifies in their submit file, or by any value of the JOB_DEFAULT_LEASE_DURATION configuration knob.  Both of these are read by condor_submit, so no reconfiguration is needed for changes to take effect.  

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Jin Mao
Sent: Tuesday, May 14, 2019 9:14 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] How to lower job lease duration on worker failure?

 

The job was submitted after SUBMIT_ATTRS was added and condor daemon reconfigured. I am using 8.6.13. So, the questions are: 1) am I using SUBMIT_ATTRS and JobLeaseDuration correctly; 2) is adjusting job lease duration the right solution to address frequently server reset? 

 

Thanks.

 

Jin.

 

On Mon, May 13, 2019 at 11:35 AM John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

Changing SUBMIT_ATTRS will affect jobs that are submitted after the change.  but it will not affect jobs already in the Schedd.

use condor_qedit to change jobs in the Schedd. 

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Jin Mao
Sent: Monday, May 13, 2019 10:23 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] How to lower job lease duration on worker failure?

 

Condor users,

 

I have a use case that my worker nodes may come and go ungracefully after jobs scheduled on them. I hope to learn the right way to move the jobs to other worker nodes as soon as possible. 

 

In below example,  job_lease_duration was not defined, so it is 2400 seconds default value. When a worker node suddenly left without gracefully shutdown, the job will not be rescheduled on another spare worker until the end of job_lease_duration window. 

 

05/13/19 10:46:58 (2.1) (59635): JobLeaseDuration remaining: 1756

05/13/19 10:46:58 (2.1) (59635): Scheduling another attempt to reconnect in 300 seconds

05/13/19 10:51:58 (2.1) (59635): Attempting to locate disconnected starter

05/13/19 10:52:18 (2.1) (59635): attempt to connect to <10.143.128.35:9618> failed: timed out after 20 seconds.

05/13/19 10:52:18 (2.1) (59635): locateStarter(): Failed to connect to startd <10.143.128.35:9618?addrs=10.143.128.35-9618&noUDP&sock=3439_c2b1_3>    

05/13/19 10:52:18 (2.1) (59635): JobLeaseDuration remaining: 1436

05/13/19 10:52:18 (2.1) (59635): Scheduling another attempt to reconnect in 300 seconds

05/13/19 10:57:18 (2.1) (59635): Attempting to locate disconnected starter

05/13/19 10:57:38 (2.1) (59635): attempt to connect to <10.143.128.35:9618> failed: timed out after 20 seconds.

05/13/19 10:57:38 (2.1) (59635): locateStarter(): Failed to connect to startd <10.143.128.35:9618?addrs=10.143.128.35-9618&noUDP&sock=3439_c2b1_3>

...

05/13/19 11:13:38 (2.1) (59635): JobLeaseDuration remaining: 156

05/13/19 11:13:38 (2.1) (59635): Scheduling another attempt to reconnect in 156 seconds

05/13/19 11:16:14 (2.1) (59635): Attempting to locate disconnected starter

05/13/19 11:16:34 (2.1) (59635): attempt to connect to <10.143.128.35:9618> failed: timed out after 20 seconds.

05/13/19 11:16:34 (2.1) (59635): locateStarter(): Failed to connect to startd <10.143.128.35:9618?addrs=10.143.128.35-9618&noUDP&sock=3439_c2b1_3>    

05/13/19 11:16:34 (2.1) (59635): JobLeaseDuration remaining: EXPIRED!

05/13/19 11:16:34 (2.1) (59635): Reconnect FAILED: Job disconnected too long: JobLeaseDuration (2400 seconds) expired

05/13/19 11:16:34 (2.1) (59635): Exiting with JOB_SHOULD_REQUEUE

05/13/19 11:16:34 (2.1) (59635): **** condor_shadow (condor_SHADOW) pid 59635 EXITING WITH STATUS 107

05/13/19 11:16:34 (pid:3299) Shadow pid 59635 for job 2.1 exited with status 107

05/13/19 11:16:34 (pid:3299) Match record (slot1@xxxxxxxxxxxxxxxxxxx <10.143.128.35:9618?addrs=10.143.128.35-9618&noUDP&sock=3439_c2b1_3> for jin, 2.1) deleted 

05/13/19 11:16:41 (pid:3299) Starting add_shadow_birthdate(2.1)

05/13/19 11:16:41 (pid:3299) Started shadow for job 2.1 on slot1@xxxxxxxxxxxxxxxxxxx <10.143.128.46:9618?addrs=10.143.128.46-9618&noUDP&sock=3377_8c39_3> for jin, (shadow pid = 65855)

05/13/19 11:16:41 Initializing a VANILLA shadow for job 2.1

05/13/19 11:16:41 (2.1) (65855): Request to run on slot1_1@xxxxxxxxxxxxxxxxxxx <10.143.128.46:9618?addrs=10.143.128.46-9618&noUDP&sock=3377_8c39_3> was ACCEPTED

 

 

At this moment, I only see job_lease_duration helped but it requires users to update their submit file. I have tried SUBMIT_ATTRS with JobLeaseDuration like below. After reconfig/restart various condor daemons, job ClassAd still shows JobLeaseDuration=2400. I am wondering if there is a better way to let schedd to relocate jobs before expiration of JobLeaseDuration. 

 

SUBMIT_ATTRS = JobLeaseDuration
JobLeaseDuration = 300
 
Thanks.
 
Jin.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/