Re: [HTCondor-users] Negotiator only allocating 1 job per machine per cycle

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 8/31/2021 9:12 AM, John M Knoeller wrote:

Ok, yes,  when a concurrency limit is in place, the Negotiator is responsible for managing the limits and the Schedd won't be able to start more than one job for each match. 

    The above is a limitation we hope to remove at some point in the
    future.  In the meantime, there is a workaround available, which is
    to enable the "consumption policies" mechanism.  The good news is
    with consumption policies enabled, multiple jobs can be started per
    server per cycle even with concurrency limits enabled.  The bad news
    is consumption policies is off by default because 
       a) enabling it reduces scalability, which could be an issue for
    large pools with many thousands of slots, and 
       b) it is a mechanism we are may drop at some point in the future
    because it favors a centralized architecture where the negotiator
    has more control, and moving forward we want to emphasize a more
    decentralized approach where the schedd has more control control.
    
    If the pros outweigh the cons for your situation, and you wish to
    enable consumption policies in your pool, add the following to the
    condor config on all your execute nodes:
       
  # Disable CLAIM_PARTITIONABLE_LEFTOVERS and instead enable
  # Consumption Policies so that concurrency limits behavior
  # can assign multiple jobs per pslot per negotiator cycle.
  CLAIM_PARTITIONABLE_LEFTOVERS = False
  CONSUMPTION_POLICY = True
    After doing the above change, I believe you will need to restart
    HTCondor (condor_restart)...  I don't think the above config change
    can be done on-the-fly with condor_reconfig.
    
    More details on consumption policies are in the Manual at:
       
     
https://htcondor.readthedocs.io/en/latest/admin-manual/policy-configuration.html?condor-negotiator-side-resource-consumption-policies#condor-negotiator-side-resource-consumption-policies
    
    You can also see more discussions about this by searching the
    htcondor-users email archive for CONSUMPTION_POLICIES like so:
    
     
https://www-auth.cs.wisc.edu/lists/htcondor-users/htdig/search.shtml#gsc.tab=0&gsc.q=CONSUMPTION_POLICIES&gsc.sort=
    
    Hope the above helps,
    Todd