[HTCondor-users] CUDA_VISIBLE_DEVICES not in the environment

["Beyer, Christoph" <christoph.beyer@xxxxxxx>]

Hi,

I am struggling a bit with the parallel usage of GPUs as I mentioned earlier. As a matter of fact part of my problems result from  
CUDA_VISIBLE_DEVICES not being set in the job environment 

I use the gpu-feature which expands as expected to: 

[root@batchg010 condor]# condor_config_val use feature:gpus
use FEATURE:GPUs is
	MACHINE_RESOURCE_INVENTORY_GPUs=$(LIBEXEC)/condor_gpu_discovery -properties $(GPU_DISCOVERY_EXTRA)
	ENVIRONMENT_FOR_AssignedGPUs=GPU_DEVICE_ORDINAL=/(CUDA|OCL)//  CUDA_VISIBLE_DEVICES
	ENVIRONMENT_VALUE_FOR_UnAssignedGPUs=10000

I am running a jobwrapper but also in the jobwrapper environment I do not see a sign of CUDA_VISIBLE_DEVICES being set, same thing in the environment once the job is running. 

Subsequently I get all 4 GPUs in a single gpu-slot: 

/usr/libexec/condor/condor_gpu_discovery
DetectedGPUs="CUDA0, CUDA1, CUDA2, CUDA3"


Is there an additional trick that I missed ? 

This on 

$CondorVersion: 8.9.1 Apr 17 2019 BuildID: 466671 PackageID: 8.9.1-1 $




-- 
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx