[HTCondor-users] assigning correct GPU on multi-GPU hosts

[Henning Fehrmann <henning.fehrmann@xxxxxxxxxx>]

Hi,

we are using HTCondor v 8.8.9. We have few nodes which host
different kinds of GPUs. If I start interactive jobs I specify the GPU
of interest in the submit file in the 'Requirements' line. E.g.:

Requirements = (TARGET.CUDA1DeviceName == "GeForce GTX 1660 Ti")

The Negotiator assigns a correct node hosting my chosen GPU for this
job. If this is the first job running on this node it seems that
automatically the setting is as follows:

CUDA_VISIBLE_DEVICES=0
_CONDOR_AssignedGPUs=CUDA0

regardless which GPU has been chosen.

In my example this actually points to the other GPU.

Is there a way for the submitter to get the correct CUDA_VISIBLE_DEVICES
and _CONDOR_AssignedGPUs variables on the startd host?

I haven't tested it with non-interactive jobs though.

Cheers,
Henning