Re: [HTCondor-users] GPU ressource usage

["Beyer, Christoph" <christoph.beyer@xxxxxxx>]

Hi Todd,

thanks for that information I remember now that I knew that myself a while ago :( 

Anyway I just installed 8.9.1 on one of the GPU-nodes and everything looks good so far: 

use feature: GPUsMonitor

leads to: 

STARTD_CRON_GPUs_MONITOR_EXECUTABLE = $(LIBEXEC)/condor_gpu_utilization
STARTD_CRON_GPUs_MONITOR_METRICS = SUM:GPUs, PEAK:GPUsMemory
STARTD_CRON_GPUs_MONITOR_MODE = WaitForExit
STARTD_CRON_GPUs_MONITOR_PERIOD = 1
STARTD_CRON_JOBLIST = NODEHEALTH GPUs_MONITOR

While a job is running I can see that GPUs_MONITOR is actually running, but there is no output to be found in the classadd of the running job nor in the history file. 

Do I need 8.9 on the scheduler too to make it a success or did I miss anything else ? 

Best
Christoph

-- 
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

----- UrsprÃngliche Mail -----
Von: "Todd Tannenbaum" <tannenba@xxxxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Gesendet: Montag, 29. April 2019 15:08:54
Betreff: Re: [HTCondor-users] GPU ressource usage

> On Apr 29, 2019, at 7:30 AM, Beyer, Christoph <christoph.beyer@xxxxxxx> wrote:
> 
> Hi,
> 
> I just aded a couple of GPU-hosts in my pool and I wonder if there is an easy way to get information about the actual usage of the gPU possibly out of the history file of the job ? 
> 
> Maybe there is some tweaking needed to do this but I would definetely like to know if and how intense the gPU tagged jobs actually used the GPU .... 
> 
> Any hints someone ? :) 
> 
> Best
> Christoph
> 

Hi Christoph,

HTCondor 8.8.2+ (and thus 8.9.1+) includes GPU monitoring fixes, and monitors both GPU core processing utilization and memory utilization .  To enable GPU monitoring in either release,
add the following line to your configuration:

use feature: GPUsMonitor

You should then get some helpful GPU utilization attributes in the history classad. 

Best regards
Todd



> -- 
> Christoph Beyer
> DESY Hamburg
> IT-Department
> 
> Notkestr. 85
> Building 02b, Room 009
> 22607 Hamburg
> 
> phone:+49-(0)40-8998-2317
> mail: christoph.beyer@xxxxxxx
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/