[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Trying to figure out how rank works when submitting




On 6/12/20 2:14 AM, Fabrice Bouye via HTCondor-users wrote:

Hi,

We have a flock of over 40 machines (Linux & Windows) spread over 2 countries (A & B). With a couple of HAD-enabled central managers to the flock located in country A. All nodes are partitionable.


Just so I'm clear, this is one single HTCondor pool -- with HAD, only one negotiator is running at any one time?

 

Recently we’ve tried to set a Rank _expression_ in sub files so machines in country B would get preferential matchmaking for jobs. We know from condor_status -server that most machines in country B have a higher mips than those in country A

 

Despite this value, when submitting from country A, most execution nodes for most (but not all) the runs were located in country A.

Similarly, when submitting from country B, most execution nodes for most (but not all) the runs were located in country B.


I couldn't tell you why some jobs end up in the wrong country, note that job RANK is just a preference, but if, for some reason, the job doesn't match any idle machines for the preferred RANK, HTCondor will find machines that are not RANK-optimal.


If you want this preference to run closer to where the job is submitted, I might make this a pool-wide property of the central manager.  First, each machine would need to advertise what country they are in.  Sound like you are already doing this with your REALM setting. 

Second, each job would need to advertise what REALM it has been submitting from.  You could have each schedd automatically insert the country it is in for each job, without forcing users to make any changes.  If you always submit with condor_submit, you can do this with SUBMIT_ATTRS, by setting an attribute maybe called CountryOfOrigin


Then, you can configure the negotiator to try to prefer matches within the same country by setting

NEGOTIATOR_PRE_JOB_RANK  = My.REALM == Target.CountryOfOrigin


-greg