[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Improving negotiator match rate

On 1/14/2021 10:19 AM, Zohar Kol wrote:


Setup description:

Condor version 8.8.9

1.5k physical machines

18.9k cores

~200TB RAM

~50k jobs in queues

4 schedulers machines (16 cores 128GB RAM each)

Negotiator / Collector machine runs on 4 core 16GB RAM machine.

Condor configuration description:

Accounting groups quotas

Pslot preemption enabled

All partitionable slots are running within Docker universe

Issue description:

We experiencing a slow job matching rate ~15 per second when the cluster is ~50% idle.

Can anyone share their tips on how to improve this rate?



Hi Zohar,

For most HTC workloads, a matching rate of 15 per second is reasonable because the matching rate is different than the rate at which jobs can start running.  When the schedd is given a matching slot by the negotiator, the schedd will keep reusing that slot for job after job after job without needing any addition matches from the negotiator.  The only time a slot needs to be matched again is when a) the slot is preempted, b) the CLAIM_WORKLIFE time has expired (default of 20 minutes), or c) the schedd unclaims the slot because it no longer has any jobs queued that match the slot.    Besides the Manual, some details can be found in this recorded HTCondor Architecture workshop presentation at https://indico.cern.ch/event/936993/contributions/4022092/

Thus I am wondering why match time is impacting you...  some thoughts:

Are you constantly submitting many short jobs in small bursts?  For example, say you are submitting a pile of 30 second jobs every 60 seconds.  In this case, perhaps the schedd gets a bunch of matches and claims a bunch of slots, but ends up releasing these slots before the next pile of jobs is submitted because the queue is empty.  And then when the next pile of jobs is submitted 60 seconds later, the schedd has to wait for the negotiator to match slots again.  If this is your scenario, you can tell the schedd to "hold on" to slot for X amount of seconds even if there are no jobs remaining in anticipation that more jobs will be submitted soon.  To do this, use the "keep_claim_idle" setting in your job submit file - the man page for condor_submit discusses the "keep_claim_idle" setting here: https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html

What is the setting on your pool for CLAIM_WORKLIFE ?  The default is 1200 seconds (20 minutes).  Did you change it to be something much smaller? 

Is there a lot of slot preemption occurring in your pool?

Hope the above helps