Re: [HTCondor-users] Detecting available CPUs instead of hardware CPUs

[Michael V Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

From: Marco Mambelli <marcom@xxxxxxxx>
Date: 01/12/2016 06:17 PM
> 
> Hi,
> if I start condor in a condor slot and do not set NUM_CPUS, then condor
> (the starting one) is detecting the hardware CPUs, not the ones that
the 
> slot is providing to it (and not the requested ones in request_cpus).
> E.g. in a node with 8 cpus and 2 equal slots (4 cpus each), condor
> starting in one of the slots thinks to have 8 cpus, even if the job
that 
> is starting the new condor had request_cpus=4.
> 
> I have 2 questions:
> 1. I observed this in 8.4, is it the desired behavior in all versions?
> 2. If I want to manually change the NUM_CPUS in the configuration
of the 
> condor that Iâm starting within the slot, which is the best way to
detect 
> the CPUS available (something that works for static and dynamic slots)?
> - assuming always the request_cpus value
> - RemoteUserCpu
> -?

Why are you starting HTCondor in a slot? That seems
like a rather odd thing to do. I doubt that the CHTC team ever envisioned
recursive HTCondor pools. :-D Some sort of test framework, maybe?

The job isn't inherently aware of how many CPUs it
has been assigned - that information is used by the negotiator to allocate
machine resources, and unless you configure things appropriately, the job
can use as many CPUs as it wants once it starts up on the machine.

When you use cgroups on Linux, the cgroup that's created
for the job in /cgroup/htcondor has its cpu.shares value set to the number
of requested CPUs times 100, that is, request_cpus = 4 sets cpu.shares
for the job to 400. This does not limit the job to that number of CPUs
unless there's competition in the kernel task scheduler for available CPUs
- if there's nothing else running the job can use all CPUs, but if there's
a dozen other jobs, it will at least get four CPUs worth of time based
on that cpu.shares value.

This is all covered in section 3.12 of the v8.2.9
manual.

The job's cgroup follows the format: /cgroup/htcondor/condor_var_lib_condor_execute_slot1@execnode1/cpu.shares

You'd want to derive this directory name using the
$_CONDOR_SCRATCH_DIR path and the RemoteHost attribute from the job's ad,
since it will vary depending on the machine's configuration.

This still won't get you to the point where your interior
HTCondor will detect and advertise the slot's CPUs, though, and there's
no hard limit available on CPU utilization as the documentation indicates,
but it will at least impose the limit when necessary.

In order to get the interior HTCondor to automatically
advertise the right number of CPUs, I think your best bet would be a prepare_job
hook (which runs before the interior HTCondor is started) that looks at
the job's Cpus attribute and then modifies the configuration for the interior
HTCondor to set NUM_CPUS accordingly, so that when the interior HTCondor
is started it will pick up what was provided by the slot from the modified
config file, rather than the whole machine.

     -Michael Pelletier.