Re: [HTCondor-users] How to tie slots to cpus ?

[valerio@xxxxxxxxxx]

So the use of cgroup cpusets/cpuacct to accomplish NUMA-aware thread allocation is mandatory I think.

Valerio

On Sun, 2021-10-10 at 08:00 -0500, tpdownes@xxxxxxxxx wrote:

Greg almost certainly wrote the thing. I must have seen (and searches for) the old ENFORCE_CPU_AFFINITY and SLOT settings Iâve used in past.

That said, I only see that ASSIGN_CPU_AFFINITY claims to pin jobs to cores, not that it will do so in a NUMA-aware way.

Tom

On Sat, Oct 9, 2021 at 9:17 PM <valerio@xxxxxxxxxx> wrote:

On Fri, 2021-09-03 at 11:07 -0500, tpdownes@xxxxxxxxx wrote:

I think it's worth adding to Greg's response that I don't believe ASSIGN_CPU_AFFINITY will, all on its own, map jobs to NUMA nodes. I believe your best bet here is to create N partitionable slots, with N equal to the number of physical CPUs you have. Then combine ASSIGN_CPU_AFFINITY and SLOT<N>_CPU_AFFINITY so that each slot is mapped to NUMA nodes.

If you want to polish the doorknob, you should also look into setting cgroup cpusets on HTCondor so that it has "exclusive" access where it can and that the other top-level cgroups (e.g. system.slice) do not have access to very many cores. There obviously has to be some overlap unless you're willing to reduce the # of cores available to HTCondor.

https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpusets.html

PS: you should consider reducing the "cpuquota" available to HTCondor in its cgroup. It's always good to have 0.25 core available for ssh!

https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html

Tom

The 9.2.0 manual says that ASSIGN_CPU_AFFINITY replaces both ENFORCE_CPU_AFFINITY and SLOT<N>_CPU_AFFINITY.

Is SLOT<N>_CPU_AFFINITY still a valid configuration variable?

https://htcondor.readthedocs.io/en/latest/admin-manual/configuration-macros.html#ASSIGN_CPU_AFFINITY

Thanks,

Valerio

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to 

htcondor-users-request@xxxxxxxxxxx

 with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/