Re: [HTCondor-users] HT cores utilized to 100% although HT core count is false

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hui Greg,

thanks for the feedback! Yes, that makes sense

Yes, that makes sense. Our intention was to "soft switch off" HT on 
these nodes as they are attracting primarily user jobs (that can be from 
a wide range of workflows...). In parallel, we have nodes of the same 
series attracting primarily Grid workflows. Since Grid jobs are not 
sensitive to latency we went 100% for HT to get "most" throughout - but 
for the users the idea was, that a physical core per requested core 
would make the users more happy.
And since deactivating HT on the CPUs is much more work than just 
deploying a new sw configuration (for moving a node between both job 
classes), we tried to tune it via Condor.

But I see, that the time slot CPU shaping is not the way to achieve that 
(and binding jobs to a cpuset might be even worse/inflexible ...)

Cheers and thanks
  Thomas

On 15/01/2021 17.51, Greg Thain wrote:
> Hi Thomas:
>
> When you sent
>
> COUNT_HYPERTHREADED_CPUS = false
>
> HTCondor will only advertise as many cores as there are physical cores.  
> Whether the kernel will choose to schedule processes only on the 
> physical cores is kind of up to the Linux kernel. If you absolutely 
> want to prohibit the kernel from ever running a process using 
> hyperthreads, it might be best to disable hyperthreading in the BIOS, 
> but I understand that's more work than merely setting a condor knob.
>
> As you see from your cgroups, an HTCondor with root will set 
> cpu.shares. Note that cpu.shares isn't a hard limit, but only comes 
> into play when there is contention. That is, let's say on your machine 
> you have 48 slots, all running jobs that have requested and been 
> allocated one core each. If 47 of those jobs are idle, (maybe waiting 
> on I/O), but one job launched 96 cpu-bound threads, the linux kernel 
> schedule may run all 96 of those threads concurrently. If the 47 idle 
> jobs suddenly become cpu-bound again, the Linux scheduler will throttle 
> the 96 thread job back to one core.
>
> Now, whether to use or disable hyperthreads depends on your needs.  
> Enabling hyperthreads, in general, increases throughput, at the cost of 
> performance and per job memory of individual jobs. There is no free lunch.
>
> -greg
>
> On 1/15/21 8:30 AM, Thomas Hartmann wrote:
> > Hi all,
> >
> > I am currently wondering about a few nodes, that have a utilization of 
> > all (HT) cores but should only be using only 50%, i.e., just the 
> > physical core count.
> >
> > The nodes have AMD Epycs with HT/SMT cores active - but since we have
> > Â COUNT_HYPERTHREAD_CPUS = false
> > set, Condor should be using only 50% of the (virtual) core count [1], 
> > or?.
> >
> > What worries me a bit is, that the CPU time shares of the jobs look 
> > good [2], i.e., currently just <48 single core jobs with a relative 
> > '100' weight. However, I am not sure anymore, how the kernel is 
> > distributing the CPU time slots here, if the parent relative share is 
> > 100%(?) of the overall(??) time share?
> >
> > Is the CPU time weighting maybe misleading here, if one tries to 
> > 'match' only for the physical core count?
> >
> > Cheers and thanks for ideas,
> > Â Thomas
> >
> >
> >
> > [1]
> > COUNT_HYPERTHREAD_CPUS = false
> > ...
> > DETECTED_CORES = 96
> > DETECTED_CPUS = 48
> > DETECTED_MEMORY = 257656
> > DETECTED_PHYSICAL_CPUS = 48
> > ..
> > NUM_CPUS = $(DETECTED_CPUS)
> >
> >
> > [2]
> > [root@batch1071 htcondor]# cat /sys/fs/cgroup/cpu,cpuacct/cpu.shares
> > 1024
> > [root@batch1071 htcondor]# cat 
> > /sys/fs/cgroup/cpu,cpuacct/htcondor/cpu.shares
> > 1024
> > [root@batch1071 htcondor]# cat 
> > /sys/fs/cgroup/cpu,cpuacct/htcondor/condor_var_lib_condor_execute_slot*/cpu.shares 
> > | sort | wc -l
> > 45
> >
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message tohtcondor-users-request@xxxxxxxxxxx  with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature