[Gems-users] error on L2 misses while multiple benchmarks running

[Pau Erola <pauerola@xxxxxxxxxx>]

Greetings,

I'm working with MOESI_CMP_directory protocol on 8core single processor 
machine with PT_to_PT interconnection network. We are launching 8 
SPEC2000 benchmarcks on Solaris9 with their corresponding skip-cycles 
and warmups. With <<con0.input "prstat\n">> we can see them running
422 root       42M   28M cpu1    59  -20   0:05:13  12% facerec_base.au
423 root      211M  197M cpu7    59  -20   0:03:39  12% swim_base.aug16
424 root      157M   47M cpu5    59  -20   0:01:13  12% galgel_base.aug
425 root     3016K 2520K cpu6    59  -20   0:00:51  11% twolf_base.aug1
426 root       28M   27M run     59  -20   0:00:38  11% equake_base.aug
427 root     4280K 3736K cpu3    59  -20   0:00:34  10% art_base.aug16a
428 root       72M   58M cpu2    59  -20   0:00:34  10% mgrid_base.aug1
429 root       96M   96M cpu4    59  -20   0:00:19 7.6% mcf_base.aug16a


After running Ruby for 200M cycles we see that only 4 first cores have 
L2 misses (512Kb, 16way assoc)
Total_misses: 1657738
total_misses: 1657738 [ 536341 467195 489785 145520 18157 249 286 205 ]
user_misses: 1630461 [ 531623 466042 488376 144420 0 0 0 0 ]
supervisor_misses: 27277 [ 4718 1153 1409 1100 18157 249 286 205 ]

instruction_executed: 981861968 [ 26401973 46018520 35723729 79469230 
194243888 200001684 200000001 200002943 ]
simics_cycles_executed: 1600173064 [ 200021633 200021633 200021633 
200021633 200021633 200021633 200021633 200021633 ]
cycles_per_instruction: 1.62973 [ 7.57601 4.34655 5.59913 2.51697 
1.02974 1.0001 1.00011 1.00009 ]
misses_per_thousand_instructions: 1.68836 [ 20.3144 10.1523 13.7104 
1.83115 0.0934753 0.00124499 0.00143 0.00102498 ]

We have monitored with printf L2 cache misses in Profiler.C and in 
Sequencer.C and seems that only 0 to 3 cores produce user misses
2u  0u  3u  1u  2u  0u  1u  3u  0u  2u  1u  3u  2u  1u  0u  3u  2u  1u 
0u  3u  1u  0u  2u  3u  1u  0u  2u  3u  0u  2u  3u  0u  1u  2u  3u  0u 
1u  2u  3u  0u  1u  2u  3u  0u  1u  2u  1u  0u  3u  2u  1u  0u  3u  1u 
2u  0u  3u  1u  0u  2u  3u  0u  2u  3u  0u  2u  3u  1u  0u  2u  3u  0u 
2u  1u  3u  0u  2u  1u  3u  0u  1u  2u  3u  1u  0u  2u  3u  1u  0u  2u 
1u  3u  0u  1u  3u  2u  0u  2u  3u  0u  1u  2u  3u  0u  1u  0u  3u  2u 
1u  0u  2u  3u  1u  0u  2u  1u  3u  0u  1u  3u  2u  0u  1u  3u  2u  0u 
1u  3u  0u  2u  3u  0u  2u  3u  0u  1u  2u  0u  3u  1u  2u  0u  1u  3u

We have tried to change Solaris scheduler, priorities, launch 
applications with multiple users,... without successful. It seems that
after loading Ruby only 4 cores work.

Any help will be very grateful.


Many thanks,
Pau