Re: [Gems-users] Please help to check my scripts for cache warmingup

[ruzhu kao <kaoruzhu@xxxxxxxxx>]

Hi Faraz:
In warm and run simics scripts, I use @microbench.start_ruby(debug=0)
show below to set these parameters.
#################################################
# Configure simics and load ruby                #
#################################################
@microbench.start_ruby(debug=0)

Start_ruby will set these parameters and you can check its contents in
 gen-scripts/microbench.py.





2009/10/27 Faraz Ahmad <faraz@xxxxxxxxxx>:
> If you exit simics and re-execute it to run the warm simics script, you must
> call instruction-fetch-mode, i/dstc-disable etc before loading ruby. I see
> that happening only in your cold script right now.
>
>
>
>
> ----- Original Message -----
> From: "ruzhu kao" <kaoruzhu@xxxxxxxxx>
> To: "Gems Users" <gems-users@xxxxxxxxxxx>
> Sent: Tuesday, October 27, 2009 2:43 AM
> Subject: Re: [Gems-users] Please help to check my scripts for cache
> warmingup
>
>
> Hi Faraz:
>   Thanks for your suggestion.
>
> I have tested cold simics script without setting
> instruction-fetch-mode and i/dstc-disable. It seems no difference at
> least the result dump from ruby about the XACT_BREAKDOWN_CYCLE maybe
> it just affect simics running environment.
>
> I use the similar way  to gen-scripts.py to generate a sh script to
> run the workload. I create a new sh to call simics like this way:
> #############################################################
> cd /opt/develop/gems/gems-2.1/simics/home/MESI_CMP_filter_directory
> ./simics -echo -verbose -no-log -no-win -stall -x
> ../../../gen-scripts/microbench-cold.simics
> ###############################################################################
> I changed the microbench-cold.simics from microbench.simics just make
> it call ssca2-cold.simics. The warm scripts are use the same calling
> pattern.
>
>  In microbench.start_ruby I used it to test ssca in my clean
> checkpoint and there is no problem to count the XACT_BREAKDOWN_CYCLE,
> so I assume it would have no problem to load and set ruby correctly.
>  Do I miss any other point. Please point my mistake.
>
> Thanks in advanced.
>
> Sincerely your:
> Ruzhu
>
> 2009/10/27 Faraz Ahmad <faraz@xxxxxxxxxx>:
>> The following steps may help:
>> 1. In cold simics script, you do not need to execute
>> instruction-fetch-mode,
>> instruction-fetch-trace, istc-disable, dstc-disable commands because you
>> are
>> not loading ruby.
>> 2. In warm simics script, you should execute above commands because now
>> you
>> want to load ruby. Also make sure that you start simics with the -stall
>> flag.
>> 3. In microbench.start_ruby, make sure that it loads the ruby module, sets
>> memory and cache parameters correctly and executes ruby0.init.
>>
>>
>> ----- Original Message -----
>> From: "ruzhu kao" <kaoruzhu@xxxxxxxxx>
>> To: "Gems Users" <gems-users@xxxxxxxxxxx>
>> Sent: Monday, October 26, 2009 9:44 PM
>> Subject: [Gems-users] Please help to check my scripts for cache warming up
>>
>>
>>> Hi all:
>>> I have read the relating topics in ISCA-tutorial and this list. I
>>> assume it needs three steps and then I write these scripts for cold,
>>> warm and run respectively. However, it seems that I should miss some
>>> points which lead to some failure. The warm and run steps take a long
>>> time and show strange behavior. Can you help to check my scripts and
>>> point out the problem.
>>>
>>> #***************below is the cold simics script******************#
>>> @import microbench, workloads, mfacet
>>>
>>> @processors = 16
>>> @ruby_proc_map = (1, 2, 2, 4, 4, 8, 8, 8, 8, 16, 16, 16, 16, 16, 16, 16,
>>> 16)
>>> @if(processors <= 16):
>>> ruby_procs = ruby_proc_map[processors]
>>> proc_no = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
>>> elif (processors <= 32):
>>> ruby_procs = 32
>>> proc_no = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
>>> 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]
>>> else:
>>> processors = -1
>>>
>>> @transactions = 1
>>> @dump_interval = 1
>>>
>>> @print "running simics with %d processors" % ruby_procs
>>>
>>> #############################################
>>> # Load a clean checkpoint #
>>> #############################################
>>> @mfacet.run_sim_command('read-configuration
>>> "../../../checkpoints/conf_files/2G_16p/step_2.conf"')
>>> @mfacet.run_sim_command('magic-break-enable')
>>>
>>>
>>> @sys.path.append("../../gen-scripts")
>>>
>>> @hostpath = "/host" + os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/"
>>>
>>>
>>> @command_lines = [
>>> "bash\n",
>>> "cp " + hostpath + "ssca2" + " ./\n",
>>> "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib\n",
>>> "export PATH=$PATH:/usr/local/bin\n",
>>> "./ssca2 -s13 -i1.0 -u1.0 -l3 -p3 -t16\n" ]
>>>
>>> @mfacet.console_commands(command_lines,"#")
>>>
>>> ############################################################################
>>> # Copy the binary and execute it in vm till the first magic breakpoint
>>> #
>>> ############################################################################
>>> @mfacet.run_sim_command('instruction-fetch-mode instruction-fetch-trace')
>>> @mfacet.run_sim_command('istc-disable')
>>> @mfacet.run_sim_command('dstc-disable')
>>> @mfacet.run_sim_command('cpu-switch-time 1')
>>> @mfacet.run_sim_command('c')
>>> @print "encounter the first magic breakpoint"
>>>
>>> ######################################################################
>>> # Continue the executing because it is just for a cold checkpoint #
>>> ######################################################################
>>> @mfacet.setup_run_for_n_transactions(transactions, dump_interval)
>>> @print "running ssca2 on simics + ruby with %d transactions" %
>>> transactions
>>> @mfacet.run_sim_command('c')
>>> @print "encounter the second magic breakpoint"
>>>
>>> #############################################################
>>> # Execute till the third breakpoint #
>>> #############################################################
>>> @mfacet.run_sim_command('c')
>>> @print "encounter the third magic breakpoint"
>>>
>>> ##############################################################
>>> # Save a cold checkpoint [App]-cold-[clean-checkpoint].conf #
>>> ##############################################################
>>> @mfacet.run_sim_command('write-configuration
>>> "../../../checkpoints/conf_files/2G_16p/ssca2-cold-step_2.conf"')
>>>
>>>
>>> quit
>>> #-----------------
>>> #***************above is the cold simics script******************#
>>>
>>> -------------------------------------------------------------------------------------------------------
>>>
>>>
>>> #***************below is the warm simics script******************#
>>> @import microbench, workloads, mfacet
>>>
>>> @env_dict = workloads.prepare_env_dictionary(simics = 1)
>>> @processors = 16
>>> @ruby_proc_map = (1, 2, 2, 4, 4, 8, 8, 8, 8, 16, 16, 16, 16, 16, 16, 16,
>>> 16)
>>> @if(processors <= 16):
>>> ruby_procs = ruby_proc_map[processors]
>>> proc_no = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
>>> elif (processors <= 32):
>>> ruby_procs = 32
>>> proc_no = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
>>> 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]
>>> else:
>>> processors = -1
>>>
>>> @print "running simics with %d processors" % ruby_procs
>>>
>>>
>>> @lock_type = 'TM'
>>> @transactions = 1
>>> @dump_interval = 1
>>> @env_dict = workloads.prepare_env_dictionary(simics = 1)
>>> @visualizer = 0
>>>
>>> #############################################
>>> # Load the cold checkpoint #
>>> #############################################
>>> @mfacet.run_sim_command('read-configuration
>>> "../../../checkpoints/conf_files/2G_16p/ssca2-cold-step_2.conf"')
>>> @mfacet.run_sim_command('magic-break-enable')
>>>
>>>
>>> @sys.path.append("../../gen-scripts")
>>>
>>> @hostpath = "/host" + os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/"
>>>
>>>
>>> @command_lines = [
>>> "bash\n",
>>> "cp " + hostpath + "ssca2" + " ./\n",
>>> "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib\n",
>>> "export PATH=$PATH:/usr/local/bin\n",
>>> "psrset -i\n",
>>> "./ssca2 -s13 -i1.0 -u1.0 -l3 -p3 -t16\n" ]
>>>
>>> @mfacet.console_commands(command_lines,"#")
>>>
>>> ############################################################################
>>> # Copy the binary and execute it in vm till the first magic breakpoint
>>> #
>>> ############################################################################
>>> @mfacet.run_sim_command('c')
>>> @print "encounter the first magic breakpoint"
>>>
>>> #################################################
>>> # Configure simics and load ruby #
>>> #################################################
>>> @microbench.start_ruby(debug=0)
>>>
>>> ######################################################################
>>> # Warmup the cache to run the parallel part of the application #
>>> ######################################################################
>>> @mfacet.setup_run_for_n_transactions(transactions, dump_interval)
>>> @print "running ssca2 on simics + ruby with %d transactions" %
>>> transactions
>>> @mfacet.run_sim_command('c')
>>> @print "encounter the second magic breakpoint"
>>>
>>> #########################################
>>> # Save the cache file #
>>> #########################################
>>> @cache_filename_prefix = os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/" + "ssca2-warm"
>>> @mfacet.run_sim_command('ruby0.save-caches %s.cache.save' %
>>> cache_filename_prefix)
>>>
>>> @filename_prefix = os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/" +
>>> "ssca2-warm-dump"
>>> @if visualizer == 1:
>>> mfacet.run_sim_command('ruby0.xact-visualizer-file %s.visual' %
>>> filename_prefix)
>>> @mfacet.run_sim_command('ruby0.dump-stats %s.stats' % filename_prefix)
>>>
>>> ##############################################################
>>> # Save a warm checkpoint [App]-warm-[clean-checkpoint].conf #
>>> ##############################################################
>>> @mfacet.run_sim_command('write-configuration
>>> "../../../checkpoints/conf_files/2G_16p/ssca2-warm-step_2.conf"')
>>>
>>>
>>> quit
>>> #-----------------
>>> #***************above is the warm simics script******************#
>>>
>>> ---------------------------------------------------------------------------------------------------
>>>
>>>
>>> #***************below is the run simics script******************#
>>> @import microbench, workloads, mfacet
>>>
>>> @env_dict = workloads.prepare_env_dictionary(simics = 1)
>>> #@processors = int(workloads.get_var(env_dict, "PROCESSORS"))
>>> @processors = 16
>>> # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
>>> @ruby_proc_map = (1, 2, 2, 4, 4, 8, 8, 8, 8, 16, 16, 16, 16, 16, 16, 16,
>>> 16)
>>> @if(processors <= 16):
>>> ruby_procs = ruby_proc_map[processors]
>>> proc_no = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
>>> elif (processors <= 32):
>>> ruby_procs = 32
>>> proc_no = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
>>> 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]
>>> else:
>>> processors = -1
>>>
>>> @print "running simics with %d processors" % ruby_procs
>>>
>>>
>>> @lock_type = 'TM'
>>> @transactions = 1
>>> @dump_interval = 1
>>> @env_dict = workloads.prepare_env_dictionary(simics = 1)
>>> @visualizer = 0
>>>
>>>
>>>
>>> @mfacet.run_sim_command('read-configuration
>>> "../../../checkpoints/conf_files/2G_16p/ssca2-warm-step_2.conf"')
>>> @mfacet.run_sim_command('magic-break-enable')
>>>
>>>
>>> @sys.path.append("../../gen-scripts")
>>>
>>> @hostpath = "/host" + os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/"
>>>
>>>
>>> @command_lines = [
>>> "bash\n",
>>> "cp " + hostpath + "ssca2" + " ./\n",
>>> "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib\n",
>>> "export PATH=$PATH:/usr/local/bin\n",
>>> "psrset -i\n",
>>> "./ssca2 -s13 -i1.0 -u1.0 -l3 -p3 -t16\n" ]
>>>
>>> @mfacet.console_commands(command_lines,"#")
>>> @mfacet.run_sim_command('c')
>>> @print "first magic break point"
>>>
>>> @microbench.start_ruby(debug=0)
>>>
>>> ###############################
>>> # Load cache data #
>>> ###############################
>>> @warmed_cache_file = os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/" +
>>> "ssca2-warm.cache.save"
>>> @mfacet.run_sim_command('ruby0.load-caches %s' % warmed_cache_file)
>>> @mfacet.run_sim_command('ruby0.clear-stats')
>>> @mfacet.setup_run_for_n_transactions(transactions, dump_interval)
>>> @print "running ssca2 on simics + ruby with %d transactions" %
>>> transactions
>>> @mfacet.run_sim_command('c')
>>>
>>> @print "second magic break point"
>>> @filename_prefix = os.getcwd() +
>>> "/../../../microbenchmarks/transactional/stamp/ssca2/" +
>>> "ssca2-warmed-result"
>>> @if visualizer == 1:
>>> mfacet.run_sim_command('ruby0.xact-visualizer-file %s.visual' %
>>> filename_prefix)
>>> @mfacet.run_sim_command('ruby0.dump-stats %s.stats' % filename_prefix)
>>>
>>>
>>> quit
>>> #-----------------
>>> #***************above is the run simics script******************#
>>>
>>> -----------------------------------------------------------------------------------------------------
>>>
>>> When I use run simics scripts load step_2.conf, the stats file
>>> generated shows counters work well like this.
>>> XACT CYCLE BREAKDOWN
>>> XACT_BREAKDOWN_NON_TRANS_CYCLES: 48557592
>>> XACT_BREAKDOWN_TRANS_CYCLES: 3398260
>>> XACT_BREAKDOWN_GOOD_TRANS_CYCLES: 3357808
>>> XACT_BREAKDOWN_ABORTING_CYCLES: 89996
>>> XACT_BREAKDOWN_COMMITING_CYCLES: 0
>>> XACT_BREAKDOWN_BACKOFF_CYCLES: 5051407
>>> XACT_BREAKDOWN_BARRIER_CYCLES: 22622226
>>> XACT_BREAKDOWN_STALL_CYCLES: 110658
>>> However, when I check the stats file from warm scripts, these counters
>>> look strange like this:
>>> XACT CYCLE BREAKDOWN
>>> XACT_BREAKDOWN_NON_TRANS_CYCLES: 122136
>>> XACT_BREAKDOWN_TRANS_CYCLES: 0
>>> XACT_BREAKDOWN_GOOD_TRANS_CYCLES: 0
>>> XACT_BREAKDOWN_ABORTING_CYCLES: 0
>>> XACT_BREAKDOWN_COMMITING_CYCLES: 0
>>> XACT_BREAKDOWN_BACKOFF_CYCLES: 0
>>> XACT_BREAKDOWN_BARRIER_CYCLES: 672
>>> XACT_BREAKDOWN_STALL_CYCLES: 0
>>>
>>> Any suggestions would be welcomed.
>>> Thanks in advance.
>>>
>>>
>>> Sincerely yours:
>>> Ruzhu
>>> _______________________________________________
>>> Gems-users mailing list
>>> Gems-users@xxxxxxxxxxx
>>> https://lists.cs.wisc.edu/mailman/listinfo/gems-users
>>> Use Google to search the GEMS Users mailing list by adding
>>> "site:https://lists.cs.wisc.edu/archive/gems-users/"; to your search.
>>>
>>>
>>
>> _______________________________________________
>> Gems-users mailing list
>> Gems-users@xxxxxxxxxxx
>> https://lists.cs.wisc.edu/mailman/listinfo/gems-users
>> Use Google to search the GEMS Users mailing list by adding
>> "site:https://lists.cs.wisc.edu/archive/gems-users/"; to your search.
>>
>>
> _______________________________________________
> Gems-users mailing list
> Gems-users@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/gems-users
> Use Google to search the GEMS Users mailing list by adding
> "site:https://lists.cs.wisc.edu/archive/gems-users/"; to your search.
>
>
> _______________________________________________
> Gems-users mailing list
> Gems-users@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/gems-users
> Use Google to search the GEMS Users mailing list by adding "site:https://lists.cs.wisc.edu/archive/gems-users/"; to your search.
>
>