Re: [Gems-users] about throughout

[Dan Gibson <degibson@xxxxxxxx>]

� wrote:
> How to evaluate the benchmarks' performance? CPI? or transactions? I have seen
> gems advises to use transactions.
>   
In multithreaded environments, metrics involving instructions per 
[anything] break down because of dynamic interaction between threads -- 
IPC, for instance, implicitly grants equal importance to all 
instructions -- instructions committed by CPU19's idle loop and CPU0's 
database transaction aren't differentiated. Of course, the next time the 
benchmark is run, it could be CPU 19 executing the transaction. To 
handle this case, we break benchmarks into /transactions/, or logical 
units of work. In database workloads, these can correspond 1-to-1 with 
database transactions. For web servers, we use client requests as 
'transactions'.
The use of transactions serves as a proxy for goodput, the true 
meaningful forward progress, of the application. Combined with a measure 
of wall clock time (RUBY_CYCLEs), it is possible to make meaningful 
performance comparisons between different configurations, given that the 
workload runs for a large number of transactions, to account for 
variation between transactions. We usually try to simulate a second or 
so of the target's execution, and we use multiple runs of each 
configuration/benchmark to achieve 95% confidence intervals to account 
for random variation.
> As I have build only 3 workloads Apache,JBB and DSS, I can understand using
> transactions to evaluate JBB and DSS, but what about Apache? I take notice of the
> setup_run_for_n_transactions in all *_build.py. How can I understand this?
>   
I'd suggest client requests as proxies for transactions.
> Another question, can we say A performs better than B because of more finished
> transactions in the same ruby cycles (of course, the same benchmark and the same
> ruby confiuration)? But, i think there is an important precondition that the
> cycles per transaction or instructions per transaction are almost the same, am I
> right? How can I evaluate A and B with different cycles per transaction (A's
> cycles per transaction almost 10 times than B) 
>   
You definitely do not want to count instructions per transaction as a 
meaningful metric, because synchronization vastly affects instruction 
counts. Cycle per transaction is also misleading, because generally 
multiple transactions will be in flight in a given 'cycle'. We prefer to 
run for /large/ numbers of transactions, and measure the RUBY_CYCLEs 
elapsed to complete our target transaction count.
One must recall that in most multithreaded codes, there is a huge set of 
correct executions, and metrics shouldn't favor one execution over another.
Regards,
Dan
> _______________________________________________
> Gems-users mailing list
> Gems-users@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/gems-users
> Use Google to search the GEMS Users mailing list by adding "site:https://lists.cs.wisc.edu/archive/gems-users/"; to your search.
>
>
>   
--
http://www.cs.wisc.edu/~gibson [esc]:wq!