Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] dynamic hierarchical group quotas clarification / python APÏ ?

>> “Note that a job specifying a value for the AccountingGroup ClassAd attribute that lacks at least one period in the specification will cause the job to not be considered part of a group when negotiating, even if the group name (highest within the hierarchy) has a quota. Furthermore, there will be no warnings that the group quota is not in effect for the job, as this syntax defines group accounting.

Note that the above phrase is not in the current HTCondor manual, because it describes a short term, broken design that we have since reverted. 

when the +AccountingGroup attribute was first defined for use in submit files it was not used to describe an accounting group in modern terms at all.  Instead it was use to define what would be more reasonably be called a user alias.    

So lets say that I run a submit a job like this:

    Universe=Vanilla
    Executable=sleep
    Arguments=600
   
condor_userprio will show

                                 Effective   Priority   Res   Total Usage  Time Since
User Name                         Priority    Factor   In Use (wghted-hrs) Last Usage
------------------------------- ------------ --------- ------ ------------ ----------
johnkn@machine                        500.00   1000.00      0       421.03    1+21:53

and the relevant attributes of condor_userprio -long are

AccountingGroup26 = "<none>"
Name26 = "johnkn@machine"
IsAccountingGroup26 = false


Since my username is johnkn.  But if I run this job, I'm telling the account to treat "Science" as my NAME for accounting purposes

    Universe=Vanilla
    Executable=sleep
    Arguments=600
    +AccountingGroup="Science"

And so condor_userprio shows this.

                                 Effective   Priority   Res   Total Usage  Time Since
User Name                         Priority    Factor   In Use (wghted-hrs) Last Usage
------------------------------- ------------ --------- ------ ------------ ----------
Science@machine                       500.00   1000.00      0       421.03    1+21:53
  
and the relevant attributes of condor_userprio -long are

AccountingGroup17 = "<none>"
Name17 = "Science@machine"
IsAccountingGroup17 = false

Thus the use of +AccountingGroup="Science" is NOT setting a group in the sense that the group accounting uses the term

So far this agrees with the above sentence.   But for a short period of time HTCondor would treat a . in the AccountingGroup
attribute as triggering new behavior.  so +AccountingGroup="Science.johnkn" would give be treated as a group specification
rather than a user alias.  

But this didn't work very well, because It's fairly common for usernames to have dots in them in some places
(for instance "John.Knoeller" was my username at a place where I worked previously.)

So the dots-are-special behavior of +AccountingGroup was reverted, and no the dot has no meaning - except, that it is used
as a sort of fields separator, so if there is a dot and everything before the dot matches the name of a defined accounting group,
then you get different behavior.

    Universe=Vanilla
    Executable=sleep
    Arguments=600
    +AccountingGroup="Science.johnkn"

Gives one of two different behaviors depending on the group configuration.   if Science is NOT in the GROUP_NAMES attribute
you get this.

                                 Effective   Priority   Res   Total Usage  Time Since
User Name                         Priority    Factor   In Use (wghted-hrs) Last Usage
------------------------------- ------------ --------- ------ ------------ ----------
Science.johnkn@machine                500.00   1000.00      0       421.03    1+21:53

and

AccountingGroup29 = "<none>"
Name29 = "Science.johnkn@machine"
IsAccountingGroup29 = false

but if Science IS in the GROUP_NAMES attribute, you get this.

Group                                   Config     Use    Effective   Priority   Res   Total Usage  Time Since Requested
  User Name                              Quota   Surplus   Priority    Factor   In Use (wghted-hrs) Last Usage Resources
-------------------------------------- --------- ------- ------------ --------- ------ ------------ ---------- ----------
group_Science                               0.33 no                     1000.00      0         0.02    0+00:07          0
  johnkn@machine                                               500.00   1000.00      0         0.02    0+00:07

and the relevant attributes of condor_userprio -long are

AccountingGroup6 = "<none>"
Name6 = "group_Science"
IsAccountingGroup6 = true

AccountingGroup41 = "group_Science"
Name41 = "group_Science.johnkn@xxxxxxxxxxxxxxxxxx"
IsAccountingGroup41 = false

-tj

On 6/25/2015 7:23 AM, SCHAER Frederic wrote:

Hi,

 

the htcondor doc states :

“Note that a job specifying a value for the AccountingGroup ClassAd attribute that lacks at least one period in the specification will cause the job to not be considered part of a group when negotiating, even if the group name (highest within the hierarchy) has a quota. Furthermore, there will be no warnings that the group quota is not in effect for the job, as this syntax defines group accounting.”

 

A bit further in the doc , it is said “the (…)remaining machine assigned to job submitters in the parent group, group_stat”

 

I’d like to ask if it would be possible to give an exemple/clarify this first paragraph ?

I indeed thought the first paragraph might explain the atlas effective quota (0) bellow, but the 2nd paragraph seems to mean that even if the subgroup is wrong, the primary group quota would be used… ?

 

# condor_userprio -all

Last Priority Update:  6/25 14:13

Group                               Effective  Config     Use    Subtree   Effective     Real   Priority   Res   Total Usage       Usage             Last       Time Since Requested

  User Name                           Quota     Quota   Surplus   Quota     Priority   Priority  Factor   In Use (wghted-hrs)    Start Time       Usage Time    Last Usage Resources

----------------------------------- --------- --------- ------- --------- ------------ -------- --------- ------ ------------ ---------------- ---------------- ---------- ----------

group_atlas                              0.00      0.33 Regroup    487.95               1176.82 100000.00   1320   1037210.12  9/10/2014 11:25  6/25/2015 14:13      <now>       1192

  admin@xxxxxxxxxxxxxxx                                                       50000.00     0.50 100000.00      0       908.85  9/10/2014 11:39  6/25/2015 10:15    0+03:58

  pilot@xxxxxxxxxxxxxxx                                                       50000.00     0.50 100000.00      0       880.15  9/10/2014 11:25  6/25/2015 10:22    0+03:51

  production@xxxxxxxxxxxxxxx                                              117682032.00  1176.82 100000.00   1320   1035432.88  9/10/2014 12:07  6/25/2015 14:13      <now>

group_alice                             80.23      0.05 Regroup     80.23                  2.00 100000.00      2    298768.59  2/11/2015 07:47  6/25/2015 14:13      <now>          2

  pilot@xxxxxxxxxxxxxxx                                                       50000.00     0.50 100000.00      0       219.64  2/11/2015 11:25  6/25/2015 10:11    0+04:01

  admin@xxxxxxxxxxxxxxx                                                      200000.00     2.00 100000.00      2    298548.22  2/11/2015 07:47  6/25/2015 14:13      <now>

group_cms                              304.51      0.21 Regroup    304.51                 17.00 100000.00     17    560510.19  9/22/2014 19:10  6/25/2015 14:13      <now>         17

  admin@xxxxxxxxxxxxxxx                                                       50000.00     0.50 100000.00      0       978.47  9/22/2014 19:40  6/25/2015 14:04    0+00:08

  production@xxxxxxxxxxxxxxx                                                  50000.00     0.50 100000.00      0       749.61  9/22/2014 19:20  6/25/2015 09:58    0+04:15

  pilot@xxxxxxxxxxxxxxx                                                     1700119.25    17.00 100000.00     17    558788.19  9/22/2014 19:10  6/25/2015 14:13      <now>

<none>                                1464.00      0.00 yes       1464.00                  0.50 100000.00      0       869.46  7/29/2014 16:27  6/25/2015 14:10    0+00:02          0

  group_ops@xxxxxxxxxxxxxxx                                                   50000.00     0.50 100000.00      0        29.12  7/31/2014 08:26  6/25/2015 10:23    0+03:50

  group_alice@xxxxxxxxxxxxxxx                                                 50000.00     0.50 100000.00      0       180.65  2/11/2015 10:52  6/25/2015 10:07    0+04:05

  group_ops.admin@xxxxxxxxxxxxxxx                                             50000.00     0.50 100000.00      0       652.60  7/30/2014 17:21  6/25/2015 14:10    0+00:02

----------------------------------- --------- --------- ------- --------- ------------ -------- --------- ------ ------------ ---------------- ---------------- ---------- ----------

Number of users: 11                                     Regroup                                             1339   1897368.50                   6/24/2015 14:14    0+23:59

 

 

Now concerning the API : I’d like to graph the user priorities using graphite so that I can tell at once who has the highest priority just looking at one graph – and I’d also like to graph quotas to see how they evolve compared to the priorities to see if anything is not working as expected…

Unfortunately, I don’t see anything in the pyhon (or perl ?) bindings that would enable me to graph quotas (I only see the getPriorities() function…) : any idea ?

 

(condor 8.2 here)

 

Thanks && regards

Frederic



_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/