[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Preemption question




Something I didn't make very clear in my previous post: although MaxJobRetirementTime limits the preemption of jobs within the time window you specify, it does not interfere with the preemption of resource claims. Therefore, you can go ahead and enable the various forms of preemption that you want, while still providing a guarantee to jobs that they can expect to run uninterrupted for a certain amount of time (barring events beyond the control of Condor). A claim that is preempted will not accept any new jobs; instead, it enters a retirement phase where it waits for the final job to finish or for the retirement time to run out.

--Dan

Dan Bradley wrote:

There are a number of different possible causes of preemption in Condor, and your policy eliminates most but not all of them. The startd RANK expression is treated as an overriding directive by the negotiator, trumping the normal user-priority based calculations (and therefore PREEMPTION_REQUIREMENTS). This means that your policy will cause precisely the kind of preemption that you have observed--members of group "numi" will preempt other users.

One solution to this is to use MaxJobRetirementTime. This allows you to have preemption of resource claims without having killing of jobs. The expression specifies the number of seconds since the job started running that the job will be allowed to run without interruption from kill signals, even if the claim is preempted. This applies to all forms of preemption, including startd RANK preemption.

If you do decide to use this policy mechanism, then you could consider turning back on PREEMPTION_REQUIREMENTS, which allows the normal fair share algorithm to adjust resource claims. If you disable this form of preemption, then the problem is that once a user gets a claim to a machine, the schedd may hang on to it indefinitely if the user keeps enough jobs waiting in the queue. Another way to solve that is to use CLAIM_WORKLIFE to set an upper bound on how long a claim will keep accepting more jobs.

--Dan

Steven Timm wrote:

I have a condor pool where most of the machines are set to
never pre-empt.  I thought that this setting would mean that
pre-emption doesn't happen but it  appears I am wrong.

On 15 of my machines I have the following settings
(and condor_config_val acknowledges they are seen both
by the startd on the machine and by the negotiator/collector).

[root@fnpc182 log]# condor_config_val -startd PREEMPT
FALSE
[root@fnpc182 log]# condor_config_val -startd PREEMPTION_REQUIREMENTS
FALSE
[root@fnpc182 log]# condor_config_val -startd START
TRUE
[root@fnpc182 log]# condor_config_val -startd RANK
(agroup == "group_numi" ) * 1000


What I want to happen is to give this machine priority of
starting jobs from group_numi, (agroup is a group attribute that
I set in the classads of all jobs).  But I don't want it to
pre-empt an existing job of some other group if that job is
not yet finished yet.

What is actually happening is the following:

From StartLog
3/23 13:22:56 DaemonCore: Command received via UDP from host <131.225.167.42:198
21>
3/23 13:22:56 DaemonCore: received command 440 (MATCH_INFO), calling handler (co
mmand_match_info)
3/23 13:22:56 vm1: match_info called
3/23 13:22:56 DaemonCore: Command received via UDP from host <131.225.167.42:198
21>
3/23 13:22:56 DaemonCore: received command 440 (MATCH_INFO), calling handler (co
mmand_match_info)
3/23 13:22:56 vm2: match_info called
3/23 13:22:56 DaemonCore: Command received via TCP from host <131.225.167.42:307
85>
3/23 13:22:56 DaemonCore: received command 442 (REQUEST_CLAIM), calling handler
(command_request_claim)
3/23 13:22:56 vm1: Preempting claim has correct ClaimId.
3/23 13:22:56 vm1: New claim has sufficient rank, preempting current claim.
3/23 13:22:56 vm1: State change: preempting claim based on machine rank
3/23 13:22:56 vm1: State change: retiring due to preempting claim
3/23 13:22:56 vm1: Changing activity: Busy -> Retiring
3/23 13:22:56 vm1: State change: retirement ended/expired
3/23 13:22:56 vm1: Changing state and activity: Claimed/Retiring -> Preempting/V
acating
3/23 13:22:56 DaemonCore: Command received via TCP from host <131.225.167.42:307
86>
3/23 13:22:56 DaemonCore: received command 442 (REQUEST_CLAIM), calling handler
(command_request_claim)
3/23 13:22:56 vm2: Preempting claim has correct ClaimId.
3/23 13:22:56 vm2: New claim has sufficient rank, preempting current claim.
3/23 13:22:56 vm2: State change: preempting claim based on machine rank
3/23 13:22:56 vm2: State change: retiring due to preempting claim
3/23 13:22:56 vm2: Changing activity: Busy -> Retiring
3/23 13:22:56 vm2: State change: retirement ended/expired
3/23 13:22:56 vm2: Changing state and activity: Claimed/Retiring -> Preempting/V
acating
3/23 13:22:56 DaemonCore: Command received via TCP from host <131.225.167.42:307
94>
3/23 13:22:56 DaemonCore: received command 404 (DEACTIVATE_CLAIM_FORCIBLY), call
ing handler (command_handler)
3/23 13:22:56 vm1: Got KILL_FRGN_JOB while in Preempting state, ignoring.
3/23 13:22:56 Starter pid 4093 exited with status 0
3/23 13:22:56 vm1: State change: preempting claim exists - START is true or unde
fined
3/23 13:22:56 vm1: Remote owner is rubin@xxxxxxxx
3/23 13:22:56 vm1: State change: claiming protocol successful
3/23 13:22:56 vm1: Changing state and activity: Preempting/Vacating -> Claimed/I
dle
3/23 13:22:56 DaemonCore: Command received via TCP from host <131.225.167.42:307
96>
3/23 13:22:56 DaemonCore: received command 404 (DEACTIVATE_CLAIM_FORCIBLY), call
ing handler (command_handler)
3/23 13:22:56 vm2: Got KILL_FRGN_JOB while in Preempting state, ignoring.
3/23 13:22:56 DaemonCore: Command received via UDP from host <131.225.167.42:198
49>
3/23 13:22:56 DaemonCore: received command 443 (RELEASE_CLAIM), calling handler
(command_release_claim)
3/23 13:22:56 Warning: can't find resource with ClaimId (<131.225.167.182:22866>
#1142441053#75)
3/23 13:22:57 DaemonCore: Command received via UDP from host <131.225.167.42:198
49>
3/23 13:22:57 DaemonCore: received command 443 (RELEASE_CLAIM), calling handler
(command_release_claim)
3/23 13:22:57 vm2: Got RELEASE_CLAIM while in Preempting state, ignoring.
3/23 13:22:57 DaemonCore: Command received via UDP from host <131.225.167.42:198
49>
3/23 13:22:57 DaemonCore: received command 443 (RELEASE_CLAIM), calling handler
(command_release_claim)
3/23 13:22:57 vm2: Got RELEASE_CLAIM while in Preempting state, ignoring.
3/23 13:23:01 DaemonCore: Command received via TCP from host <131.225.167.42:308
56>
3/23 13:23:01 DaemonCore: received command 444 (ACTIVATE_CLAIM), calling handler
(command_activate_claim)

\
and in NegotiatorLog it indicated that there was indeed
a job from a user in group_numi, with priority 16, who pre-empted
the existing job from a user not in group_numi, at the time had a priority
of 160.

How do we beat this, is there any way to give preference for
starting jobs without having pre-emption go on?

Steve Timm





_______________________________________________
Condor-users mailing list
Condor-users@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/condor-users