[Gems-users] Question about network topology and Simics core mapping

[Hamid Reza Khaleghzadeh <khaleghzadeh@xxxxxxxxx>]

Hi

I have executed Fmm splash benchmark with 3 processes on Ruby. Ruby
network topology was considered hierarchical switch where all
latencies are same (topology is attached at following):

Network Configuration
---------------------
network: SIMPLE_NETWORK
topology: HIERARCHICAL_SWITCH

virtual_net_0: active, unordered
virtual_net_1: active, unordered
virtual_net_2: active, unordered
virtual_net_3: inactive
virtual_net_4: inactive

--- Begin Topology Print ---

Topology print ONLY indicates the _NETWORK_ latency between two machines
It does NOT include the latency within the machines

L2Cache-0 Network Latencies
  L2Cache-0 -> L2Cache-1 net_lat: 9
  L2Cache-0 -> L2Cache-2 net_lat: 9
  L2Cache-0 -> L2Cache-3 net_lat: 9
  L2Cache-0 -> L1Cache-0 net_lat: 9
  L2Cache-0 -> L1Cache-1 net_lat: 9
  L2Cache-0 -> L1Cache-2 net_lat: 9
  L2Cache-0 -> L1Cache-3 net_lat: 9
  L2Cache-0 -> L1Cache-4 net_lat: 9
  L2Cache-0 -> L1Cache-5 net_lat: 9
  L2Cache-0 -> L1Cache-6 net_lat: 9
  L2Cache-0 -> L1Cache-7 net_lat: 9
  L2Cache-0 -> Directory-0 net_lat: 9
  L2Cache-0 -> Directory-1 net_lat: 9
  L2Cache-0 -> Directory-2 net_lat: 9
  L2Cache-0 -> Directory-3 net_lat: 9

L2Cache-1 Network Latencies
  L2Cache-1 -> L2Cache-0 net_lat: 9
  L2Cache-1 -> L2Cache-2 net_lat: 9
  L2Cache-1 -> L2Cache-3 net_lat: 9
  L2Cache-1 -> L1Cache-0 net_lat: 9
  L2Cache-1 -> L1Cache-1 net_lat: 9
  L2Cache-1 -> L1Cache-2 net_lat: 9
  L2Cache-1 -> L1Cache-3 net_lat: 9
  L2Cache-1 -> L1Cache-4 net_lat: 9
  L2Cache-1 -> L1Cache-5 net_lat: 9
  L2Cache-1 -> L1Cache-6 net_lat: 9
  L2Cache-1 -> L1Cache-7 net_lat: 9
  L2Cache-1 -> Directory-0 net_lat: 9
  L2Cache-1 -> Directory-1 net_lat: 9
  L2Cache-1 -> Directory-2 net_lat: 9
  L2Cache-1 -> Directory-3 net_lat: 9

L2Cache-2 Network Latencies
  L2Cache-2 -> L2Cache-0 net_lat: 9
  L2Cache-2 -> L2Cache-1 net_lat: 9
  L2Cache-2 -> L2Cache-3 net_lat: 9
  L2Cache-2 -> L1Cache-0 net_lat: 9
  L2Cache-2 -> L1Cache-1 net_lat: 9
  L2Cache-2 -> L1Cache-2 net_lat: 9
  L2Cache-2 -> L1Cache-3 net_lat: 9
  L2Cache-2 -> L1Cache-4 net_lat: 9
  L2Cache-2 -> L1Cache-5 net_lat: 9
  L2Cache-2 -> L1Cache-6 net_lat: 9
  L2Cache-2 -> L1Cache-7 net_lat: 9
  L2Cache-2 -> Directory-0 net_lat: 9
  L2Cache-2 -> Directory-1 net_lat: 9
  L2Cache-2 -> Directory-2 net_lat: 9
  L2Cache-2 -> Directory-3 net_lat: 9

L2Cache-3 Network Latencies
  L2Cache-3 -> L2Cache-0 net_lat: 9
  L2Cache-3 -> L2Cache-1 net_lat: 9
  L2Cache-3 -> L2Cache-2 net_lat: 9
  L2Cache-3 -> L1Cache-0 net_lat: 9
  L2Cache-3 -> L1Cache-1 net_lat: 9
  L2Cache-3 -> L1Cache-2 net_lat: 9
  L2Cache-3 -> L1Cache-3 net_lat: 9
  L2Cache-3 -> L1Cache-4 net_lat: 9
  L2Cache-3 -> L1Cache-5 net_lat: 9
  L2Cache-3 -> L1Cache-6 net_lat: 9
  L2Cache-3 -> L1Cache-7 net_lat: 9
  L2Cache-3 -> Directory-0 net_lat: 9
  L2Cache-3 -> Directory-1 net_lat: 9
  L2Cache-3 -> Directory-2 net_lat: 9
  L2Cache-3 -> Directory-3 net_lat: 9

L1Cache-0 Network Latencies
  L1Cache-0 -> L2Cache-0 net_lat: 9
  L1Cache-0 -> L2Cache-1 net_lat: 9
  L1Cache-0 -> L2Cache-2 net_lat: 9
  L1Cache-0 -> L2Cache-3 net_lat: 9
  L1Cache-0 -> L1Cache-1 net_lat: 9
  L1Cache-0 -> L1Cache-2 net_lat: 9
  L1Cache-0 -> L1Cache-3 net_lat: 9
  L1Cache-0 -> L1Cache-4 net_lat: 9
  L1Cache-0 -> L1Cache-5 net_lat: 9
  L1Cache-0 -> L1Cache-6 net_lat: 9
  L1Cache-0 -> L1Cache-7 net_lat: 9
  L1Cache-0 -> Directory-0 net_lat: 9
  L1Cache-0 -> Directory-1 net_lat: 9
  L1Cache-0 -> Directory-2 net_lat: 9
  L1Cache-0 -> Directory-3 net_lat: 9

L1Cache-1 Network Latencies
  L1Cache-1 -> L2Cache-0 net_lat: 9
  L1Cache-1 -> L2Cache-1 net_lat: 9
  L1Cache-1 -> L2Cache-2 net_lat: 9
  L1Cache-1 -> L2Cache-3 net_lat: 9
  L1Cache-1 -> L1Cache-0 net_lat: 9
  L1Cache-1 -> L1Cache-2 net_lat: 9
  L1Cache-1 -> L1Cache-3 net_lat: 9
  L1Cache-1 -> L1Cache-4 net_lat: 9
  L1Cache-1 -> L1Cache-5 net_lat: 9
  L1Cache-1 -> L1Cache-6 net_lat: 9
  L1Cache-1 -> L1Cache-7 net_lat: 9
  L1Cache-1 -> Directory-0 net_lat: 9
  L1Cache-1 -> Directory-1 net_lat: 9
  L1Cache-1 -> Directory-2 net_lat: 9
  L1Cache-1 -> Directory-3 net_lat: 9

L1Cache-2 Network Latencies
  L1Cache-2 -> L2Cache-0 net_lat: 9
  L1Cache-2 -> L2Cache-1 net_lat: 9
  L1Cache-2 -> L2Cache-2 net_lat: 9
  L1Cache-2 -> L2Cache-3 net_lat: 9
  L1Cache-2 -> L1Cache-0 net_lat: 9
  L1Cache-2 -> L1Cache-1 net_lat: 9
  L1Cache-2 -> L1Cache-3 net_lat: 9
  L1Cache-2 -> L1Cache-4 net_lat: 9
  L1Cache-2 -> L1Cache-5 net_lat: 9
  L1Cache-2 -> L1Cache-6 net_lat: 9
  L1Cache-2 -> L1Cache-7 net_lat: 9
  L1Cache-2 -> Directory-0 net_lat: 9
  L1Cache-2 -> Directory-1 net_lat: 9
  L1Cache-2 -> Directory-2 net_lat: 9
  L1Cache-2 -> Directory-3 net_lat: 9

L1Cache-3 Network Latencies
  L1Cache-3 -> L2Cache-0 net_lat: 9
  L1Cache-3 -> L2Cache-1 net_lat: 9
  L1Cache-3 -> L2Cache-2 net_lat: 9
  L1Cache-3 -> L2Cache-3 net_lat: 9
  L1Cache-3 -> L1Cache-0 net_lat: 9
  L1Cache-3 -> L1Cache-1 net_lat: 9
  L1Cache-3 -> L1Cache-2 net_lat: 9
  L1Cache-3 -> L1Cache-4 net_lat: 9
  L1Cache-3 -> L1Cache-5 net_lat: 9
  L1Cache-3 -> L1Cache-6 net_lat: 9
  L1Cache-3 -> L1Cache-7 net_lat: 9
  L1Cache-3 -> Directory-0 net_lat: 9
  L1Cache-3 -> Directory-1 net_lat: 9
  L1Cache-3 -> Directory-2 net_lat: 9
  L1Cache-3 -> Directory-3 net_lat: 9

L1Cache-4 Network Latencies
  L1Cache-4 -> L2Cache-0 net_lat: 9
  L1Cache-4 -> L2Cache-1 net_lat: 9
  L1Cache-4 -> L2Cache-2 net_lat: 9
  L1Cache-4 -> L2Cache-3 net_lat: 9
  L1Cache-4 -> L1Cache-0 net_lat: 9
  L1Cache-4 -> L1Cache-1 net_lat: 9
  L1Cache-4 -> L1Cache-2 net_lat: 9
  L1Cache-4 -> L1Cache-3 net_lat: 9
  L1Cache-4 -> L1Cache-5 net_lat: 9
  L1Cache-4 -> L1Cache-6 net_lat: 9
  L1Cache-4 -> L1Cache-7 net_lat: 9
  L1Cache-4 -> Directory-0 net_lat: 9
  L1Cache-4 -> Directory-1 net_lat: 9
  L1Cache-4 -> Directory-2 net_lat: 9
  L1Cache-4 -> Directory-3 net_lat: 9

L1Cache-5 Network Latencies
  L1Cache-5 -> L2Cache-0 net_lat: 9
  L1Cache-5 -> L2Cache-1 net_lat: 9
  L1Cache-5 -> L2Cache-2 net_lat: 9
  L1Cache-5 -> L2Cache-3 net_lat: 9
  L1Cache-5 -> L1Cache-0 net_lat: 9
  L1Cache-5 -> L1Cache-1 net_lat: 9
  L1Cache-5 -> L1Cache-2 net_lat: 9
  L1Cache-5 -> L1Cache-3 net_lat: 9
  L1Cache-5 -> L1Cache-4 net_lat: 9
  L1Cache-5 -> L1Cache-6 net_lat: 9
  L1Cache-5 -> L1Cache-7 net_lat: 9
  L1Cache-5 -> Directory-0 net_lat: 9
  L1Cache-5 -> Directory-1 net_lat: 9
  L1Cache-5 -> Directory-2 net_lat: 9
  L1Cache-5 -> Directory-3 net_lat: 9

L1Cache-6 Network Latencies
  L1Cache-6 -> L2Cache-0 net_lat: 9
  L1Cache-6 -> L2Cache-1 net_lat: 9
  L1Cache-6 -> L2Cache-2 net_lat: 9
  L1Cache-6 -> L2Cache-3 net_lat: 9
  L1Cache-6 -> L1Cache-0 net_lat: 9
  L1Cache-6 -> L1Cache-1 net_lat: 9
  L1Cache-6 -> L1Cache-2 net_lat: 9
  L1Cache-6 -> L1Cache-3 net_lat: 9
  L1Cache-6 -> L1Cache-4 net_lat: 9
  L1Cache-6 -> L1Cache-5 net_lat: 9
  L1Cache-6 -> L1Cache-7 net_lat: 9
  L1Cache-6 -> Directory-0 net_lat: 9
  L1Cache-6 -> Directory-1 net_lat: 9
  L1Cache-6 -> Directory-2 net_lat: 9
  L1Cache-6 -> Directory-3 net_lat: 9

L1Cache-7 Network Latencies
  L1Cache-7 -> L2Cache-0 net_lat: 9
  L1Cache-7 -> L2Cache-1 net_lat: 9
  L1Cache-7 -> L2Cache-2 net_lat: 9
  L1Cache-7 -> L2Cache-3 net_lat: 9
  L1Cache-7 -> L1Cache-0 net_lat: 9
  L1Cache-7 -> L1Cache-1 net_lat: 9
  L1Cache-7 -> L1Cache-2 net_lat: 9
  L1Cache-7 -> L1Cache-3 net_lat: 9
  L1Cache-7 -> L1Cache-4 net_lat: 9
  L1Cache-7 -> L1Cache-5 net_lat: 9
  L1Cache-7 -> L1Cache-6 net_lat: 9
  L1Cache-7 -> Directory-0 net_lat: 9
  L1Cache-7 -> Directory-1 net_lat: 9
  L1Cache-7 -> Directory-2 net_lat: 9
  L1Cache-7 -> Directory-3 net_lat: 9

Directory-0 Network Latencies
  Directory-0 -> L2Cache-0 net_lat: 9
  Directory-0 -> L2Cache-1 net_lat: 9
  Directory-0 -> L2Cache-2 net_lat: 9
  Directory-0 -> L2Cache-3 net_lat: 9
  Directory-0 -> L1Cache-0 net_lat: 9
  Directory-0 -> L1Cache-1 net_lat: 9
  Directory-0 -> L1Cache-2 net_lat: 9
  Directory-0 -> L1Cache-3 net_lat: 9
  Directory-0 -> L1Cache-4 net_lat: 9
  Directory-0 -> L1Cache-5 net_lat: 9
  Directory-0 -> L1Cache-6 net_lat: 9
  Directory-0 -> L1Cache-7 net_lat: 9
  Directory-0 -> Directory-1 net_lat: 9
  Directory-0 -> Directory-2 net_lat: 9
  Directory-0 -> Directory-3 net_lat: 9

Directory-1 Network Latencies
  Directory-1 -> L2Cache-0 net_lat: 9
  Directory-1 -> L2Cache-1 net_lat: 9
  Directory-1 -> L2Cache-2 net_lat: 9
  Directory-1 -> L2Cache-3 net_lat: 9
  Directory-1 -> L1Cache-0 net_lat: 9
  Directory-1 -> L1Cache-1 net_lat: 9
  Directory-1 -> L1Cache-2 net_lat: 9
  Directory-1 -> L1Cache-3 net_lat: 9
  Directory-1 -> L1Cache-4 net_lat: 9
  Directory-1 -> L1Cache-5 net_lat: 9
  Directory-1 -> L1Cache-6 net_lat: 9
  Directory-1 -> L1Cache-7 net_lat: 9
  Directory-1 -> Directory-0 net_lat: 9
  Directory-1 -> Directory-2 net_lat: 9
  Directory-1 -> Directory-3 net_lat: 9

Directory-2 Network Latencies
  Directory-2 -> L2Cache-0 net_lat: 9
  Directory-2 -> L2Cache-1 net_lat: 9
  Directory-2 -> L2Cache-2 net_lat: 9
  Directory-2 -> L2Cache-3 net_lat: 9
  Directory-2 -> L1Cache-0 net_lat: 9
  Directory-2 -> L1Cache-1 net_lat: 9
  Directory-2 -> L1Cache-2 net_lat: 9
  Directory-2 -> L1Cache-3 net_lat: 9
  Directory-2 -> L1Cache-4 net_lat: 9
  Directory-2 -> L1Cache-5 net_lat: 9
  Directory-2 -> L1Cache-6 net_lat: 9
  Directory-2 -> L1Cache-7 net_lat: 9
  Directory-2 -> Directory-0 net_lat: 9
  Directory-2 -> Directory-1 net_lat: 9
  Directory-2 -> Directory-3 net_lat: 9

Directory-3 Network Latencies
  Directory-3 -> L2Cache-0 net_lat: 9
  Directory-3 -> L2Cache-1 net_lat: 9
  Directory-3 -> L2Cache-2 net_lat: 9
  Directory-3 -> L2Cache-3 net_lat: 9
  Directory-3 -> L1Cache-0 net_lat: 9
  Directory-3 -> L1Cache-1 net_lat: 9
  Directory-3 -> L1Cache-2 net_lat: 9
  Directory-3 -> L1Cache-3 net_lat: 9
  Directory-3 -> L1Cache-4 net_lat: 9
  Directory-3 -> L1Cache-5 net_lat: 9
  Directory-3 -> L1Cache-6 net_lat: 9
  Directory-3 -> L1Cache-7 net_lat: 9
  Directory-3 -> Directory-0 net_lat: 9
  Directory-3 -> Directory-1 net_lat: 9
  Directory-3 -> Directory-2 net_lat: 9

I have defined a CMP that contains 4 chips. Each chip contains 2
cores. Coherency protocol is MOESI_CMP_DIRECTORY.
I ran Fmm two times. First, 3 processes of Fmm was mapped on cores
1,2,3 and second time, they mapped on cores 3,4,5. I expected that
ruby cycles of run-1 equals with ruby cycles of run-2 but ruby cycles
of run-1 is 16816271 and run-2 is 18876464. How do you legitimise
this?

By the way, I have another question. When a CMP is defined that
contains 4 chips where each chip has 2 cores and one L2, Could you
tell me how simics cores are mapped to ruby? In other words, There are
four L2 caches that each of them is shared by 2 cores. Simics core
ID's are [0 , 7]. Could you tell me which cores are connected to same
L2?

Sorry to bother you.



-- 
Hamid Reza Khaleghzadeh