Re: [HTCondor-users] negotiator questions

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

> On Mar 6, 2016, at 8:45 AM, Krieger, Donald N. <kriegerd@xxxxxxxx> wrote:
> 
> Itâs my understanding that at each pass, the negotiator goes through each queued job to decide whether and which job slot to assign it.
> I use the directive, Priority = 0-n to control which jobs run first but I also have 2 groups of jobs with disjoint IP address requirements to divide my jobs between two subsets of the condor pool.
> My question is about the order and âcompletenessâ with which the negotiator âexaminesâ my queued jobs.
>  
> I presume that it goes through every single one of them regardless of how well and how many it has matched to job slots as it proceeds through.
> Yes?

Mostly.  The exceptions are:
1) If there are other users in the system, you might run out of fair share during the negotiation cycle and the negotiator moves onto other usersâ jobs before it gets through your complete list.
2) The schedd the negotiator is talking to times out or has some other error.

>  
> I presume that it goes through them by âPriorityâ first and by fifo within each priority level.
> Yes?

Mostly true.  You can assign other attributes (PreJobPrio* and PostJobPrio*) that affect the sort order for the jobs in the schedd.  Further, things get more complicated if you are submitting from multiple schedds.

>  
> I presume the time for each pass scales linearly as the total number of jobs queued.
> Yes?
>  

Nope.  Identical jobs are grouped together into a single âresource requestâ.  The negotiator builds a list of matches against the resource request, then sends them back to the schedd asynchronously.

Hence, the more âidentical jobsâ (typically, the jobs resulting from a âqueue Nâ statement in the submit file), the less the negotiation time.

> Can the negotiator get clobbered by a memory overrun due a very large number of queued jobs?

The negotiator, no.  It only considers a single resource request at a time - it does not buffer them all in memory (but it does buffer all machine ads in memory).

The schedd, however, is more sensitive to number of queued jobs.  Each takes up somewhere in the neighborhood of 50KB of RAM and a small amount of CPU time (which becomes noticeable once there are tens of thousands of idle jobs).

Brian