We have a flock of over 40 machines (Linux & Windows) spread over 2 countries (A & B). With a couple of HAD-enabled central managers to the flock located in country A. All nodes are partitionable.
Just so I'm clear, this is one single HTCondor pool -- with HAD,
only one negotiator is running at any one time?
Recently we’ve tried to set a Rank _expression_ in sub files so machines in country B would get preferential matchmaking for jobs. We know from condor_status -server that most machines in country B have a higher mips than those in country A
Despite this value, when submitting from country A, most execution nodes for most (but not all) the runs were located in country A.
Similarly, when submitting from country B, most execution nodes for most (but not all) the runs were located in country B.
I couldn't tell you why some jobs end up in the wrong country, note that job RANK is just a preference, but if, for some reason, the job doesn't match any idle machines for the preferred RANK, HTCondor will find machines that are not RANK-optimal.
If you want this preference to run closer to where the job is
submitted, I might make this a pool-wide property of the central
manager. First, each machine would need to advertise what country
they are in. Sound like you are already doing this with your
Second, each job would need to advertise what REALM it has been
submitting from. You could have each schedd automatically insert
the country it is in for each job, without forcing users to make
any changes. If you always submit with condor_submit, you can do
this with SUBMIT_ATTRS, by setting an attribute maybe called
Then, you can configure the negotiator to try to prefer matches within the same country by setting
NEGOTIATOR_PRE_JOB_RANK = My.REALM == Target.CountryOfOrigin