[HTCondor-users] Question regarding overlapping GPU slots and nested partitioning of slots

[Voigtländer, Tim Aike (ETP) <tim.voigtlaender@xxxxxxx>]

Hi all,

I'm currently trying to set up a GPU node that accepts both, jobs that request multiple GPUs at once

and jobs that request only a portion of a GPU (by specifying the amount of device memory).

I was able to set up both by using partitionable slots.

For the multi GPU variant, all GPUs are assigned to the same slot and dynamic slots get assigned a number of the available GPUs. (This is the default setup.)

For the split GPU variant, one slot is defined per GPU and by using the `-repeat` option of the `condor_gpu_discovery` multiple jobs can be assigned to the same GPU.

In the second setup, the requested device memory is the limiting factor.

I'd like to use both on the same node, for maximum, flexibility, but this seems to be difficult.

It is possible to have both kinds of setups exist at the same time,

having the GPUs and its copies in partitionable slots, as well as the GPUs in one partitionable slot (see attached config file).

However, there seems to be no way to disallow the multi GPU jobs from being assigned to GPUs that are also already assigned to split GPU jobs and vice versa.

I was wondering if there is a straightforward solution to such an issue of conflicting slots.

Alternatively, is it possible to achieve a sort of nested partitionable slot?

That way, a whole GPU could be dynamically assigned to a partitionable slot that can then be further subdivided.

I've attached an excerpt of the condor_config for clarity.

The version used is 23.0.4 on CentOS7.

Cheers and thanks,
Tim Voigtländer

NUM_MAX_GPU_DIVIDE = 5

# returns `condor_gpu_discovery -properties -packed -repeat $(NUM_MAX_GPU_DIVIDE )`, but the normal output of the discovery is appended to the returned `DetectedGPUs`
# Example: `DetectedGPUs="GPU-edf948a0, GPU-edf948a0, GPU-edf948a0, GPU-78a0883a, GPU-78a0883a, GPU-78a0883a, GPU-9f253d6d, GPU-9f253d6d, GPU-9f253d6d, GPU-90314a2d, GPU-90314a2d, GPU-90314a2d, GPU-af07c1df, GPU-af07c1df, GPU-af07c1df, GPU-975c6ff2, GPU-975c6ff2, GPU-975c6ff2, GPU-bb8bda67, GPU-bb8bda67, GPU-bb8bda67, GPU-51b715a5, GPU-51b715a5, GPU-51b715a5, GPU-edf948a0, GPU-78a0883a, GPU-9f253d6d, GPU-90314a2d, GPU-af07c1df, GPU-975c6ff2, GPU-bb8bda67, GPU-51b715a5"`
MACHINE_RESOURCE_INVENTORY_GPUs = /etc/condor/scripts/gpu_discovery_wrapper.sh $(LIBEXEC)/condor_gpu_discovery $(NUM_MAX_GPU_DIVIDE)

## Ideally, this would be read from condor_gpu_discovery
DEVICE_MEMORY_PER_GPU = 40000
MACHINE_RESOURCE_GPUMemoryMB = $(DEVICE_MEMORY_PER_GPU) * 8

# Split GPU slots
SLOT_TYPE_1_PARTITIONABLE = TRUE
NUM_SLOTS_TYPE_1 = 8
SLOT_TYPE_1 = GPUMemoryMB=$(DEVICE_MEMORY_PER_GPU), GPUs=$(NUM_MAX_GPU_DIVIDE), auto

# Multi GPU slot
SLOT_TYPE_2_PARTITIONABLE = TRUE
NUM_SLOTS_TYPE_2 = 1
SLOT_TYPE_2 = Cpus=auto, Memory=auto, Disk=auto, Swap=auto, GPUs=8

SLOT1TO8_CONDITION = ( My.SlotID < 9 && ( Target.RequestGPUs == 1 && ( Target.RequestGPUMemoryMB > 0 && Target.RequestGPUMemoryMB < $(DEVICE_MEMORY_PER_GPU) ) ))
SLOT9_CONDITION = ( My.SlotID == 9 && ( Target.RequestGPUs > 1 || Target.RequestGPUMemoryMB =?= undefined || Target.RequestGPUMemoryMB == 0 || Target.RequestGPUMemoryMB >= $(DEVICE_MEMORY_PER_GPU) ) )
START = $(START) && ( $(SLOT1TO8_CONDITION) || $(SLOT9_CONDITION) )