[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Detecting available CPUs instead of hardware CPUs

From: Marco Mambelli <marcom@xxxxxxxx>
Date: 01/12/2016 06:17 PM
> Hi,
> if I start condor in a condor slot and do not set NUM_CPUS, then condor
> (the starting one) is detecting the hardware CPUs, not the ones that the
> slot is providing to it (and not the requested ones in request_cpus).
> E.g. in a node with 8 cpus and 2 equal slots (4 cpus each), condor
> starting in one of the slots thinks to have 8 cpus, even if the job that
> is starting the new condor had request_cpus=4.
> I have 2 questions:
> 1. I observed this in 8.4, is it the desired behavior in all versions?
> 2. If I want to manually change the NUM_CPUS in the configuration of the
> condor that Iâm starting within the slot, which is the best way to detect
> the CPUS available (something that works for static and dynamic slots)?
> - assuming always the request_cpus value
> - RemoteUserCpu
> -?

Why are you starting HTCondor in a slot? That seems like a rather odd thing to do. I doubt that the CHTC team ever envisioned recursive HTCondor pools. :-D Some sort of test framework, maybe?

The job isn't inherently aware of how many CPUs it has been assigned - that information is used by the negotiator to allocate machine resources, and unless you configure things appropriately, the job can use as many CPUs as it wants once it starts up on the machine.

When you use cgroups on Linux, the cgroup that's created for the job in /cgroup/htcondor has its cpu.shares value set to the number of requested CPUs times 100, that is, request_cpus = 4 sets cpu.shares for the job to 400. This does not limit the job to that number of CPUs unless there's competition in the kernel task scheduler for available CPUs - if there's nothing else running the job can use all CPUs, but if there's a dozen other jobs, it will at least get four CPUs worth of time based on that cpu.shares value.

This is all covered in section 3.12 of the v8.2.9 manual.

The job's cgroup follows the format: /cgroup/htcondor/condor_var_lib_condor_execute_slot1@execnode1/cpu.shares

You'd want to derive this directory name using the $_CONDOR_SCRATCH_DIR path and the RemoteHost attribute from the job's ad, since it will vary depending on the machine's configuration.

This still won't get you to the point where your interior HTCondor will detect and advertise the slot's CPUs, though, and there's no hard limit available on CPU utilization as the documentation indicates, but it will at least impose the limit when necessary.

In order to get the interior HTCondor to automatically advertise the right number of CPUs, I think your best bet would be a prepare_job hook (which runs before the interior HTCondor is started) that looks at the job's Cpus attribute and then modifies the configuration for the interior HTCondor to set NUM_CPUS accordingly, so that when the interior HTCondor is started it will pick up what was provided by the slot from the modified config file, rather than the whole machine.

     -Michael Pelletier.