[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Priorities



From: Marc Jackson - NOAA Affiliate <marc.jackson@xxxxxxxx>
Date: 07/20/2016 05:46 PM

> Here is my condor_userprio output:
>
> Group                                  Config     Use    Effective   Priority   Res   Total Usage  Time Since Requested
>   User Name                             Quota   Surplus   Priority    Factor   In Use (wghted-hrs) Last Usage Resources
> ------------------------------------- --------- ------- ------------ --------- ------ ------------ ---------- ----------
> group_arr_prep_validation                  0.14 Regroup                1000.00      0       111.13    0+01:00          0

>   User1                                                       500.00   1000.00      0        86.11    0+01:00
> group_integrator_request                   0.04 Regroup                1000.00      0        43.53    0+00:07          0
>   User2                                                       772.60   1000.00      0        42.22    0+00:07
> group_pseudo_operational_processing        0.25 Regroup                1000.00      0       262.43    0+00:05          0
>   User3                                                         1.00      2.00      0        19.45    0+00:05
>   User4                                                         1.00      2.00      0        55.09    0+00:59
> <none>                                     0.00    yes                 1000.00      0       214.77    0+00:02          6
>   User5                                                         5.00     10.00      0        81.44    0+00:59
>   User6                                                       500.00   1000.00      0        15.80    0+00:02
> ------------------------------------- --------- ------- ------------ --------- ------ ------------ ---------- ----------
> Number of users: 6                              Regroup                             0       300.12    0+23:59
>
> It looks like this is what we want from our priorities. However when we
> submit jobs they are not running in order according to these settings. For example:

> when jobs are waiting in the queue, User1 jobs are starting before User3
> jobs even though User3 is set to a higher priority than User1.


I see you're at the SMCD - I assume you've got some Raytheon VIIRS
data flowing through there... :-) Do you know any AWIPS folks?

The important thing to remember is that HTCondor priority has more to do
with your share, rather than your position in line. The EUP is
inversely proportional to the number of matching slots you can be
expected to get.

With an EUP of 500 vs an EUP of 1, User1 will, roughly speaking, get
one slot for every 500 slots User3 gets.

But, if User3's jobs don't match resources which User1's jobs
do, you will see User1 jobs starting based on the leftovers.

For example...

If User3 is submitting jobs with 6GB of requested memory while User1
is requesting 1GB, on a 64GB server User3 will be able to start 10 jobs
for a total of 60GB, while User1 will be able to start four jobs in the
four gigabytes that's now left over and which is too small for
another User3 job.

So you'll end up with 10 User3 jobs and four User1 jobs running,
regardless of the priority values, because that's what maximizes
resource utilization. (Assuming Partitionable slots are configured,
of course.)

There's a few other factors involved as well such as round robin rate,
pie spin/slice, slot lease durations, etc. which I'm sure someone else
can go into, or you can delve into section 3.4 in the manual.

        -Michael Pelletier.
_