Re: [HTCondor-users] Only preempting one job per negotiation cycle per partitionable slot

[duduhandelman@xxxxxxxxxxx]

Sam.

First of all try to avoid preemption.

Anyway you should reduce the time between negotiation cycles. 

Play with 

NEGOTIATOR_MIN_INTERVAL

NEGOTIATOR_CYCLE_DELAY

Thanks 

David 

Get Outlook for Android

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sam Copeland <SCopeland@xxxxxxxxxxxx>
Sent: Monday, November 8, 2021 8:51:15 PM
To: 'htcondor-users@xxxxxxxxxxx' <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Only preempting one job per negotiation cycle per partitionable slot

 

I want to have two accounting groups where each one can accept the surplus from the other when not in use. Each one is given a quota of 0.5. Additionally, I want jobs to be preempted/evicted if one group is currently using more than 50% of resources and the other group submits jobs.

 

For example:

We have group_1 and group_2.

We have 2 machines with16 cores each:

# Slot Configuration

NUM_SLOTS = 1

NUM_SLOTS_TYPE_1 = 1

SLOT_TYPE_1 = auto

SLOT_TYPE_1_PARTITIONABLE = True

 

On the manager config, I’ve set:

PREEMPTION_REQUIREMENTS = debug(((SubmitterGroupResourcesInUse < SubmitterGroupQuota) && (ifThenElse(isUndefined(RemoteGroupResourcesInUse),1000000,RemoteGroupResourcesInUse) > ifThenElse(isUndefined(RemoteGroupQuota),0,RemoteGroupQuota))) || ((SubmitterGroup =!= "nice-user") && (RemoteGroup =?= "nice-user")))

 

Let’s say all machines are unused. Then group_1 submits 100 jobs, each requesting 1 cpu. It uses all 32 cores for 32 jobs. Then group_2 submits 100 jobs while the group_1 jobs are still running. I want 16 of the group_1 jobs to be evicted to allow group_2 to get 0.5 of the total resources, as per the quota. Unfortunately, it only evicts 1 job per negotiation cycle, so it take ~16 minutes to evict all 16 jobs that are required.

 

Is there any way to have this speed up so it can evict multiple jobs per negotiation cycle? Am I missing something in my config?

 

Thanks,

Sam

This communication (both the message and any attachments or links) is confidential and only intended for the use of the person or persons to whom it is addressed unless we have expressly authorized otherwise. It also may contain information that is protected by solicitor-client privilege. If you are reading this communication and are not an addressee or authorized representative of an addressee, we hereby notify you that any distribution, copying or other use of it without our express authorization is strictly prohibited. If you have received this communication in error, please delete both the message and any attachments from your system and notify us immediately by e-mail or phone. In addition, we note that this communication and its transmission of data have not been secured by encryption. Therefore, we are not able to confirm or guarantee that the communication has not been intercepted, amended, or read by an unintended third party.