Re: [Condor-users] How to filter out hyperthreading CPU cores

[Hermann Fuchs <hermann.fuchs@xxxxxxxxxxxxxxxx>]

Hello

It seems condor is unable to determine the hyperthreading CPUs.

Here is my output.
_CONDOR_TOOL_DEBUG=D_FULLDEBUG condor_config_val -debug NUM_CPUS
11/09/11 08:19:44 Reading from /proc/cpuinfo
11/09/11 08:19:44 Found: Physical-IDs:True; Core-IDs:True
11/09/11 08:19:44 Analyzing 8 processors using IDs...
11/09/11 08:19:44 Looking at processor #0 (PID:0, CID:0):
11/09/11 08:19:44 Comparing P#0   and P#1  : pid:0!=1 or  cid:0!=0
(match=No)
11/09/11 08:19:44 Comparing P#0   and P#2  : pid:0!=0 or  cid:0!=1
(match=No)
11/09/11 08:19:44 Comparing P#0   and P#3  : pid:0!=1 or  cid:0!=1
(match=No)
11/09/11 08:19:44 Comparing P#0   and P#4  : pid:0!=0 or  cid:0!=2
(match=No)
11/09/11 08:19:44 Comparing P#0   and P#5  : pid:0!=1 or  cid:0!=2
(match=No)
11/09/11 08:19:44 Comparing P#0   and P#6  : pid:0!=0 or  cid:0!=3
(match=No)
11/09/11 08:19:44 Comparing P#0   and P#7  : pid:0!=1 or  cid:0!=3
(match=No)
11/09/11 08:19:44 ncpus = 1
11/09/11 08:19:44 P0: match->1
11/09/11 08:19:44 Looking at processor #1 (PID:1, CID:0):
11/09/11 08:19:44 Comparing P#1   and P#2  : pid:1!=0 or  cid:0!=1
(match=No)
11/09/11 08:19:44 Comparing P#1   and P#3  : pid:1!=1 or  cid:0!=1
(match=No)
11/09/11 08:19:44 Comparing P#1   and P#4  : pid:1!=0 or  cid:0!=2
(match=No)
11/09/11 08:19:44 Comparing P#1   and P#5  : pid:1!=1 or  cid:0!=2
(match=No)
11/09/11 08:19:44 Comparing P#1   and P#6  : pid:1!=0 or  cid:0!=3
(match=No)
11/09/11 08:19:44 Comparing P#1   and P#7  : pid:1!=1 or  cid:0!=3
(match=No)
11/09/11 08:19:44 ncpus = 2
11/09/11 08:19:44 P1: match->1
11/09/11 08:19:44 Looking at processor #2 (PID:0, CID:1):
11/09/11 08:19:44 Comparing P#2   and P#3  : pid:0!=1 or  cid:1!=1
(match=No)
11/09/11 08:19:44 Comparing P#2   and P#4  : pid:0!=0 or  cid:1!=2
(match=No)
11/09/11 08:19:44 Comparing P#2   and P#5  : pid:0!=1 or  cid:1!=2
(match=No)
11/09/11 08:19:44 Comparing P#2   and P#6  : pid:0!=0 or  cid:1!=3
(match=No)
11/09/11 08:19:44 Comparing P#2   and P#7  : pid:0!=1 or  cid:1!=3
(match=No)
11/09/11 08:19:44 ncpus = 3
11/09/11 08:19:44 P2: match->1
11/09/11 08:19:44 Looking at processor #3 (PID:1, CID:1):
11/09/11 08:19:44 Comparing P#3   and P#4  : pid:1!=0 or  cid:1!=2
(match=No)
11/09/11 08:19:44 Comparing P#3   and P#5  : pid:1!=1 or  cid:1!=2
(match=No)
11/09/11 08:19:44 Comparing P#3   and P#6  : pid:1!=0 or  cid:1!=3
(match=No)
11/09/11 08:19:44 Comparing P#3   and P#7  : pid:1!=1 or  cid:1!=3
(match=No)
11/09/11 08:19:44 ncpus = 4
11/09/11 08:19:44 P3: match->1
11/09/11 08:19:44 Looking at processor #4 (PID:0, CID:2):
11/09/11 08:19:44 Comparing P#4   and P#5  : pid:0!=1 or  cid:2!=2
(match=No)
11/09/11 08:19:44 Comparing P#4   and P#6  : pid:0!=0 or  cid:2!=3
(match=No)
11/09/11 08:19:44 Comparing P#4   and P#7  : pid:0!=1 or  cid:2!=3
(match=No)
11/09/11 08:19:44 ncpus = 5
11/09/11 08:19:44 P4: match->1
11/09/11 08:19:44 Looking at processor #5 (PID:1, CID:2):
11/09/11 08:19:44 Comparing P#5   and P#6  : pid:1!=0 or  cid:2!=3
(match=No)
11/09/11 08:19:44 Comparing P#5   and P#7  : pid:1!=1 or  cid:2!=3
(match=No)
11/09/11 08:19:44 ncpus = 6
11/09/11 08:19:44 P5: match->1
11/09/11 08:19:44 Looking at processor #6 (PID:0, CID:3):
11/09/11 08:19:44 Comparing P#6   and P#7  : pid:0!=1 or  cid:3!=3
(match=No)
11/09/11 08:19:44 ncpus = 7
11/09/11 08:19:44 P6: match->1
11/09/11 08:19:44 Looking at processor #7 (PID:1, CID:3):
11/09/11 08:19:44 ncpus = 8
11/09/11 08:19:44 P7: match->1
11/09/11 08:19:44 Using IDs: 8 processors, 8 CPUs, 0 HTs

It is a
Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz

Is this an operating system issue? We are running Ubunut 11.04.

Best regards,
Hermann

On Tue, 2011-11-08 at 22:48 -0500, Matthew Farrellee wrote:
> I don't know specifically, but you can run "env 
> _CONDOR_TOOL_DEBUG=D_FULLDEBUG condor_config_val -debug NUM_CPUS" to get 
> a look at how condor is detecting the number of CPUs.
> 
> Best,
> 
> 
> matt
> 
> On 10/21/2011 08:21 AM, Hermann Fuchs wrote:
> > Hello
> >
> > It seems
> > COUNT_HYPERTHREAD_CPUS=FALSE
> > has no effect. Still all virtual hyperthreaded cores show up.
> > Using NUM_CPUS=4
> > works as expected.
> > However I would prefer a more automated way of detection.
> >
> > Do you have any idea why COUNT_HYPERTHREAD_CPUS=FALSE could not work?
> > On this machine Ubuntu 11.04 64bit is employed.
> >
> > Yours,
> > Hermann
> >
> > On Fri, 2011-10-21 at 07:31 -0400, Matthew Farrellee wrote:
> >> On 10/21/2011 04:38 AM, Hermann Fuchs wrote:
> >>> Hello
> >>>
> >>> In our condor cluster we have several workstations configured to use
> >>> hyperthreading (e.g. virtual CPU cores from INTEL CPUs).
> >>> Condor seems to recognise virtual hyperthreaded CPU cores as real
> > ones
> >>> and evenly divides recources.
> >>>
> >>> For example a workstation with a four core Intel CPU would then
> > appear
> >>> as an eight core workstation, which we would like to avoid.
> >>>
> >>> We do not want to use such virtual CPU cores but can not enforce
> > their
> >>> hardware sided (e.g in the BIOS) deactivation. Is there a way that
> >>> condor somehow "filters" them out and announces only real CPU cores?
> >>>
> >>> Can condor distinguish real cores from virtual ones?
> >>> I suppose manually declaring 4 CPU cores may not work because all 4
> >>> declared cores might be running on virtual cores...
> >>>
> >>> Yours,
> >>> Hermann
> >>
> >> You could configure NUM_CPUS=4, but try COUNT_HYPERTHREAD_CPUS=FALSE
> > first.
> >>
> >> http://www.cs.wisc.edu/condor/manual/v7.6/3_3Configuration.html#17736
> >>
> >> COUNT_HYPERTHREAD_CPUS
> >>       This macro controls how Condor sees hyper threaded processors.
> > When
> >> set to True (the default), it includes virtual CPUs in the default
> > value
> >> of NUM_CPUS. On dedicated cluster nodes, counting virtual CPUs can
> >> sometimes improve total throughput at the expense of individual job
> >> speed. However, counting them on desktop workstations can interfere
> > with
> >> interactive job performance.
> >>
> >> Best,
> >>
> >>
> >> matt
> >>
> >
> 
> 

-- 
-------------
DI Hermann Fuchs
Div. Medical Radiation Physics
Department of Radiotherapy
Med. Univ. Vienna / AKH Vienna
Währinger Gürtel 18-20
A-1090 Wien

Tel.  + 43 / 1 / 40 400 7271
Mail. hermann.fuchs@xxxxxxxxxxxxxxxx