Re: [HTCondor-users] Enabling "use feature: GPUs" using environment variable

[Zach Miller <zmiller@xxxxxxxxxxx>]

Hi Vlad,

Just curious as a followup, did one of Toddâs ideas work out for you?


Cheers,
-zach


On 11/2/17, 7:06 PM, "HTCondor-users on behalf of Todd Tannenbaum" <htcondor-users-bounces@xxxxxxxxxxx on behalf of tannenba@xxxxxxxxxxx> wrote:

    On 11/2/2017 5:02 PM, Vladimir Brik wrote:
    > Hello.
    > 
    > Is it possible to enable "use feature:  GPUs" configuration setting
    > using an _CONDOR_... environment variable?
    > 
    > Vlad
    
    Hi Vlad,
    
    It is not obvious, but a couple of ideas come to mind that I am hoping
    can work for your situation.
    
    -- IDEA #1
    
    Realize that all config templates like "use feature:GPUs" do is expand to a bunch of regular name=value pairs, and also realize that 
    you can use condor_config_val to tell you what the template expands into.
    Like so:
    
      $ condor_config_val use feature:GPUs
      use FEATURE:GPUs is
            MACHINE_RESOURCE_INVENTORY_GPUs=$(LIBEXEC)/condor_gpu_discovery -properties $(GPU_DISCOVERY_EXTRA)
            ENVIRONMENT_FOR_AssignedGPUs=GPU_DEVICE_ORDINAL=/(CUDA|OCL)//  CUDA_VISIBLE_DEVICES
            ENVIRONMENT_VALUE_FOR_UnAssignedGPUs=10000
    
    So you could certainly just hardcode the above three knobs as _condor_ environment variables, but
    one advantage of using a config template is perhaps the next version of HTCondor may improve 
    how the template is defined.  So instead you probably want a script to first call condor_config_val 
    to see how the feature is defined, and then set the environment variables dynamically.  Like so:
    
      #!/bin/bash
      # Set a bunch of environment vars so HTCondor thinks 'use feature:GPUs' is set
      # Warning: Mountain Dew drinker line to follow. 
      source <(condor_config_val use feature:GPUs | sed '1d; s/\t//; s/=/;/' | awk -F\; '{printf "export _condor_%s=\x27%s\x27\n",$1,$2}')
      # At this point, any HTCondor process subsequently launched will think use feature:GPUs is set
    
    -- IDEA #2
    
    You could setup your condor_config in advance to conditionally use feature gpus based on an environment variable.
    
    So if you add the following to condor_config:
    
      # Support GPUs only if environment variable WANT_GPU_FEATURES is defined
      if defined $ENV(WANT_GPU_FEATURES)
         use feature:GPUs
      endif
    
    Now things can work like so:
    
      $ condor_config_val -dump GPUs
    
      $ WANT_GPU_FEATURES=True condor_config_val -dump GPUs
      ENVIRONMENT_FOR_AssignedGPUs = GPU_DEVICE_ORDINAL=/(CUDA|OCL)//  CUDA_VISIBLE_DEVICES
      ENVIRONMENT_VALUE_FOR_UnAssignedGPUs = 10000
      MACHINE_RESOURCE_INVENTORY_GPUs = $(LIBEXEC)/condor_gpu_discovery -properties $(GPU_DISCOVERY_EXTRA)
    
    Hope the above helps,
    regards,
    Todd
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/