Re: [Condor-users] Fair Resource Allocation question

Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

As a new user, I would like to understand how Condor will handle my expected allocation scenarios below.

1. There are 4 users, A, B, C, D.

2. Each user owns 50 machines and they all agree to pool them together into a 200 machine condor pool.

3. For simplicity, all machines are the same and all jobs have the same requirements all the time, and all users have the same User priority.

4. Assume also that preemption is disabled.

5. case #1: All 4 users, submit exactly N jobs each (where N is lower or equal to 50). The expected behavior after the first negotiation cycle, is that each user will get exactly N machines (1 per job).

Add another assumption and yes, this is what you'll see. The other assumption you need to add is that all four users have the exact same effective user priority. See: http://research.cs.wisc.edu/condor/manual/v7.6/3_4User_Priorities.html#25902

If they all have the same EUP, they'll all get exactly 1/4 of the system after one negotiation cycle assuming everything about their jobs is equal.

This is easy enough to test.

I queued up 10 sleep jobs from four users in a new pool that has four slots available in it. None of these users had accumulated any use history so all had identical EUPs of 0. Before I queued up the jobs, I shut down the negotiator with:

condor_off -negotiator

You can see the jobs ready to go:

-bash-3.2# condor_status -submitter

Name Machine Running IdleJobs HeldJobs

alice@.internal domU-12-31 0 10 0

bob@.internal domU-12-31 0 10 0

eve@.internal domU-12-31 0 10 0

test.user@.internal domU-12-31 0 10 0

RunningJobs IdleJobs HeldJobs

alice@.internal 0 10 0

bob@.internal 0 10 0

eve@.internal 0 10 0

test.user@.internal 0 10 0

Total 0 40 0

I turned on the negotiator for one negotiation cycle and got one job from each user assigned to each of the four slots in my pool:

-bash-3.2# condor_q -const 'jobstatus == 2'

-- Submitter: Q1@domU-12-31-38-04-9C-A1 : <10.220.159.79:59831> : domU-12-31-38-04-9C-A1.compute-1.internal

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

2.0 test.user 8/22 11:12 0+00:05:44 R 0 0.0 sleeper.py --min=6

3.0 alice 8/22 11:14 0+00:05:47 R 0 0.0 sleeper.py --min=6

4.0 bob 8/22 11:14 0+00:05:47 R 0 0.0 sleeper.py --min=6

5.0 eve 8/22 11:14 0+00:05:45 R 0 0.0 sleeper.py --min=6

08/22/12 11:15:23 ---------- Started Negotiation Cycle ----------

08/22/12 11:15:23 Phase 1: Obtaining ads from collector ...

08/22/12 11:15:23 Getting all public ads ...

08/22/12 11:15:24 Sorting 17 ads ...

08/22/12 11:15:24 Getting startd private ads ...

08/22/12 11:15:24 Got ads: 17 public and 4 private

08/22/12 11:15:24 Public ads include 4 submitter, 4 startd

08/22/12 11:15:24 Phase 2: Performing accounting ...

08/22/12 11:15:24 Phase 3: Sorting submitter ads by priority ...

08/22/12 11:15:24 Phase 4.1: Negotiating with schedds ...

08/22/12 11:15:24 Negotiating with alice@.internal at <10.220.159.79:59831>

08/22/12 11:15:24 0 seconds so far

08/22/12 11:15:24 Request 00003.00000:

08/22/12 11:15:24 Matched 3.0 alice@.internal <10.220.159.79:59831> preempting none <10.123.7.99:57106> ip-10-123-7-99.ec2.internal

08/22/12 11:15:24 Successfully matched with ip-10-123-7-99.ec2.internal

08/22/12 11:15:24 Request 00003.00001:

08/22/12 11:15:24 Rejected 3.1 alice@.internal <10.220.159.79:59831>: fair share exceeded

08/22/12 11:15:24 Got NO_MORE_JOBS; done negotiating

08/22/12 11:15:24 Negotiating with bob@.internal at <10.220.159.79:59831>

08/22/12 11:15:24 0 seconds so far

08/22/12 11:15:24 Request 00004.00000:

08/22/12 11:15:24 Matched 4.0 bob@.internal <10.220.159.79:59831> preempting none <10.93.21.85:53716> ip-10-93-21-85.ec2.internal

08/22/12 11:15:24 Successfully matched with ip-10-93-21-85.ec2.internal

08/22/12 11:15:24 Request 00004.00001:

08/22/12 11:15:24 Rejected 4.1 bob@.internal <10.220.159.79:59831>: fair share exceeded

08/22/12 11:15:25 Got NO_MORE_JOBS; done negotiating

08/22/12 11:15:25 Negotiating with eve@.internal at <10.220.159.79:59831>

08/22/12 11:15:25 0 seconds so far

08/22/12 11:15:25 Request 00005.00000:

08/22/12 11:15:25 Matched 5.0 eve@.internal <10.220.159.79:59831> preempting none <10.127.163.251:50135> ip-10-127-163-251.ec2.internal

08/22/12 11:15:25 Successfully matched with ip-10-127-163-251.ec2.internal

08/22/12 11:15:25 Request 00005.00001:

08/22/12 11:15:25 Rejected 5.1 eve@.internal <10.220.159.79:59831>: fair share exceeded

08/22/12 11:15:25 Got NO_MORE_JOBS; done negotiating

08/22/12 11:15:25 Negotiating with test.user@.internal at <10.220.159.79:59831>

08/22/12 11:15:25 0 seconds so far

08/22/12 11:15:25 Request 00002.00000:

08/22/12 11:15:25 Matched 2.0 test.user@.internal <10.220.159.79:59831> preempting none <10.220.109.195:45947> domU-12-31-38-04-6E-39.compute-1.internal

08/22/12 11:15:25 Successfully matched with domU-12-31-38-04-6E-39.compute-1.internal

08/22/12 11:15:25 Reached submitter resource limit: 1.000000 ... stopping

08/22/12 11:15:25 negotiateWithGroup resources used scheddAds length 4

08/22/12 11:15:25 ---------- Finished Negotiation Cycle ----------

Condor determines the fairshare allotments at the outset of the negotiation cycle so it stopped after each user got one machine -- their fair share.

6. case #2: All 4 users, submit exactly M jobs each (where M > 50). The expected behavior after the first negotiation cycle, is that each user will get 50 machines (1 per job), and will still have (M-50) jobs pending in the queue.

Yes. This is what will happen. Again, assuming their EUPs are all equal.

If I understood correctly, from section 3.4.5 (Negotiation) of the Condor version 7.8.1 manual, then the negotiation algorithm will attempt to fulfill the first submitter's full job list, prior to skipping to the next submitter and getting its job list. So in my case#2 above, user A (if it's the first submitter) will get M machines allocated, leaving either B,C or D with less than 50 machines, as I expect.

No, that's not what happens. The negotiator determines up front, using the EUP of each submitter, what each submitter's fair share of the machines should be for this negotiation cycle. And based on that it moves through each submitter's list of idle jobs and tries to match them to slots.

If the EUPs of your users aren't all identical then the allocations will not be equal. Some users will get more because they've used less in the recent past. Some users will get less because they've used more in the recent past.

So to summarize my questions are:

1. Will my expected behavior for both case #1 and case#2 above, indeed occur, under my assumptions ?

Only if you also add the assumption that all EUPs are identical for the users.

2. How does setting Hierarchical Group Quotas (as in section 3.4.8), affect the negotiation flow ? does it change the Pie Slice value ?

Accounting groups help ensure that, regardless of EUP, people get some minimum (and possibly maximum) number of slots in your pool when they have jobs in a queue.

If you wanted each user to always get 50 machines, but user >50 machines if other users aren't using their machines, you'd setup soft quotas for 4 different groups and put each user in a unique group. Now, Condor will attempt to fulfill their quotas first and, once all the quotas have been satisfied, it'll let excess free resources be used, fair share, but anyone who has a soft quota limit.

---

Ian Chesal

Cycle Computing, LLC

Leader in Open Compute Solutions for Clouds, Servers, and Desktops

Enterprise Condor Support and Management Tools

http://www.cyclecomputing.com

http://www.cyclecloud.com

http://twitter.com/cyclecomputing

Mailing List Archives

Public Access

Re: [Condor-users] Fair Resource Allocation question