Re: [HTCondor-users] Negotiation with partitionable slots

[Steffen Grunewald <steffen.grunewald@xxxxxxxxxx>]

On Thu, 2017-04-27 at 09:23:16 +0200, Mathieu Bahin wrote:
> Hi,
> 
> Our cluster, configured with partitionable slots and having very
> heterogeneous nodes, is being more loaded for a few months and we are
> facing a new issue: it's complicated for a user to run a job requesting
> big CPUs and/or memory.
> 
> Ideally, we would have liked to have the smallest machines first filled
> up to ~80% (not to overload them when there is still space anywhere
> else) in order to leave free a few of the biggest machines when possible.
> 
> We designed the NEGOCIATOR_JOB_POST_RANK value to something like that:
> (RemoteOwner =!= UNDEFINED) * ((floor(TotalCpus / 10) * -10000000) +
> KFlops - 1.0e10 * (Offline =?= True))
> Our nodes are then divided into 4 classes (our biggest node has 32
> CPUs), the one having less than 10 CPUs are now prioritary and within
> classes, the most powerful nodes comes first (with Kflops).
> 
> Though, if we run a few dozens of jobs, from what I noted, the priority
> order is satisfying but only one job is allocated to the nodes of the
> more priority classes (I guess because with the partitionable slots,
> nodes only advertise one big free slot for one negotiation cycle) and
> the next ones are allocated, one by one also, to the less priority nodes.
> How can we do for several jobs really filling the highest priority nodes
> before considering the others?

Bonjour Mathieu,

we're using NEGOTIATOR_PRE_JOB_RANK (btw, is that a typo in your config or
only your mail?) to prefer small nodes (including partially assigned ones)
if the job isn't a MPI one:

DEFAULT_NPREJ_RANK       = 2000000000 \
    - 100000000 * Cpus \
    - 10000 * Memory \
    + $(RANDOM_NUMBER)

PARU_NPREJ_RANK         = 0

NEGOTIATOR_PRE_JOB_RANK = ifThenElse( (Target.JobUniverse =?= 11), \
        $(PARU_NPREJ_RANK), \
        $(DEFAULT_NPREJ_RANK) )


Does this help?

- S

-- 
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1
D-14476 Potsdam-Golm
Germany
~~~
Fon: +49-331-567 7274
Fax: +49-331-567 7298
Mail: steffen.grunewald(at)aei.mpg.de
~~~