Mailing List ArchivesPublic Access |
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesPublic Access |
|
|
As for The negotiation, the only thing that matters there is the numeric value of GPUs. Changing the AssignedGPUs value that the negotiator sees won’t have any effect. I did a little digging, and it appears that a restart is required to set a GPU even for partitionable slots. I guess I remembered that wrong.
I can’t think if any way to do what you want dynamically other than something really hacky –
Use startd cron to set the attribute for in-use gpus into all slots, and then add something into the START _expression_ that makes it go to false when the AssignedGPUs value is the same as your offline GPUs. You want this to still be TRUE
in the partitionable slot when there is at lease one free GPU, but go to false when the dynamic slot ends up being assigned an in-use GPU.
You will get some false matches that get rejected at the last minute by the STARTD that way. But that’s the best I can think of. -tj From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx]
On Behalf Of Michael Pelletier Unfortunately according to 3.3.11, changes to OFFLINE_MACHINE_RESOURCE_<name> require a restart, rather than a reconfig. I’d like to do this dynamically, so that the negotiator can try to avoid GPUs in use by
jobs which haven’t yet been brought under HTCondor’s purview. Would the negotiator’s assignment work as expected if the AssignedGPUs string were made into some sort of ugly strcat() _expression_ based on the GPUs’ state attributes, and an ugly sum _expression_ for Gpus? So
that if, say, the CUDA0UtilizationPercent is 100, the “CUDA0” would be omitted from the partitionable slot’s AssignedGPUs attribute if it wasn’t assigned by HTCondor, and GPUs would be one fewer? -Michael Pelletier. From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx]
On Behalf Of John M Knoeller You can configure OFFLINE_MACHINE_RESOURCE_GPUS = CUDA0 to prevent HTCondor from assigning that GPU to a slot. -tj From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx]
On Behalf Of Michael Pelletier Hi folks, I’m working on getting a new exec node stood up with multiple GPUs, for use by job which need dedicated GPU assignment – a first in our pools. Other jobs I’ve dealt with had an internal lock and queue mechanism to be able to share all the
GPUs on the system, so I didn’t need to worry about HTCondor assignments. I’d like to be able to prevent HTCondor from assigning a GPU that’s already in use by a non-HTCondor process to one of its jobs. I wrote a wrapper for nvidia-smi which pulls in an ad like so: hostname$ /user/condor/libexec/condor_nvidia_probe CUDA0FreeGlobalMemory = 2441 CUDA0UtilizationPct = 100 CUDA1FreeGlobalMemory = 4031 CUDA1UtilizationPct = 0 CUDA2FreeGlobalMemory = 4031 CUDA2UtilizationPct = 0 CUDA3FreeGlobalMemory = 4031 CUDA3UtilizationPct = 0 CUDAFreeGlobalMemory = 14534 CUDAUtilization = 25.0 -- hostname$
So in the above case, I’d like to prevent any HTCondor job from being assigned the CUDA0 device since it’s 100% used, and preferably advertise one fewer GPU available on the system. Is there any means to do this? I’ve been mulling the kinds
of expressions I think I might need and my brain is starting to hurt a bit. Michael V. Pelletier |
