Re: [HTCondor-users] Enabling "use feature: GPUs" using environment variable

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 11/2/2017 5:02 PM, Vladimir Brik wrote:
> Hello.
> 
> Is it possible to enable "use feature:  GPUs" configuration setting
> using an _CONDOR_... environment variable?
> 
> Vlad

Hi Vlad,

It is not obvious, but a couple of ideas come to mind that I am hoping
can work for your situation.

-- IDEA #1

Realize that all config templates like "use feature:GPUs" do is expand to a bunch of regular name=value pairs, and also realize that 
you can use condor_config_val to tell you what the template expands into.
Like so:

  $ condor_config_val use feature:GPUs
  use FEATURE:GPUs is
        MACHINE_RESOURCE_INVENTORY_GPUs=$(LIBEXEC)/condor_gpu_discovery -properties $(GPU_DISCOVERY_EXTRA)
        ENVIRONMENT_FOR_AssignedGPUs=GPU_DEVICE_ORDINAL=/(CUDA|OCL)//  CUDA_VISIBLE_DEVICES
        ENVIRONMENT_VALUE_FOR_UnAssignedGPUs=10000

So you could certainly just hardcode the above three knobs as _condor_ environment variables, but
one advantage of using a config template is perhaps the next version of HTCondor may improve 
how the template is defined.  So instead you probably want a script to first call condor_config_val 
to see how the feature is defined, and then set the environment variables dynamically.  Like so:

  #!/bin/bash
  # Set a bunch of environment vars so HTCondor thinks 'use feature:GPUs' is set
  # Warning: Mountain Dew drinker line to follow. 
  source <(condor_config_val use feature:GPUs | sed '1d; s/\t//; s/=/;/' | awk -F\; '{printf "export _condor_%s=\x27%s\x27\n",$1,$2}')
  # At this point, any HTCondor process subsequently launched will think use feature:GPUs is set

-- IDEA #2

You could setup your condor_config in advance to conditionally use feature gpus based on an environment variable.

So if you add the following to condor_config:

  # Support GPUs only if environment variable WANT_GPU_FEATURES is defined
  if defined $ENV(WANT_GPU_FEATURES)
     use feature:GPUs
  endif

Now things can work like so:

  $ condor_config_val -dump GPUs

  $ WANT_GPU_FEATURES=True condor_config_val -dump GPUs
  ENVIRONMENT_FOR_AssignedGPUs = GPU_DEVICE_ORDINAL=/(CUDA|OCL)//  CUDA_VISIBLE_DEVICES
  ENVIRONMENT_VALUE_FOR_UnAssignedGPUs = 10000
  MACHINE_RESOURCE_INVENTORY_GPUs = $(LIBEXEC)/condor_gpu_discovery -properties $(GPU_DISCOVERY_EXTRA)

Hope the above helps,
regards,
Todd