rfc:better_benchmarks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
rfc:better_benchmarks [2009/04/21 09:48] – Add link for 3d-raytrace pbiggarrfc:better_benchmarks [2009/07/20 13:10] – Added AES olafurw
Line 92: Line 92:
   * [[http://www.roadsend.com/home/index.php?pageID=benchmarks|Roadsend]]   * [[http://www.roadsend.com/home/index.php?pageID=benchmarks|Roadsend]]
     * Short, doesnt take command line parameters, is varied, but simple.     * Short, doesnt take command line parameters, is varied, but simple.
-    * Many benchmarks are copied from the language shootout +    * Many benchmarks are copied from the language shootout (old version thereof, apparently) 
-    * This seems to be the only test with data that needs to be run-through. This makes it a somewhat better test than the others, it seems.+    * Some tests come with input data
  
 ==== Large benchmarks  ==== ==== Large benchmarks  ====
Line 144: Line 144:
     * Not very useful if it is IO bound     * Not very useful if it is IO bound
     * Lots of data sets: can use any PHP package     * Lots of data sets: can use any PHP package
 +      * NOTE: Seems to require graphviz
  
   * [[http://westhoffswelt.de/projects/phpWhirl.html|phpWhirl]]   * [[http://westhoffswelt.de/projects/phpWhirl.html|phpWhirl]]
Line 169: Line 170:
       * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/3d-raytrace.js|3d-raytrace]] (ignoring the actual drawing from JS)       * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/3d-raytrace.js|3d-raytrace]] (ignoring the actual drawing from JS)
  
 +===== Benchmarking in other languages =====
 +
 +* Python: http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Performance
 +
 +* Ruby: http://groups.google.com/group/ruby-benchmark-suite
 +
 +===== Desired Benchmark Features =====
 +
 +==== For command-line apps applications ====
 +  * Run-time
 +  * Memory usage
 +  * Hardware performance counters (if available), using PAPI
 +  * Simulated hardware statistics, using cachegrind
 +    * These should be combined into a single representative number, using some hardware model.
 +  * Ability to compare all of these over two runs
 +  * Support for other PHP implementations
 +  * Benchmark characterisation:
 +    * This is hard to do properly, so best to do it badly and give coarse grained information.
 +
 +*TODO*
 +
 +==== For web-apps ====
 +
 +  * Requests per second
 +  * Memory usage
 +  * Bottleneck (is it scripting, DB, or network)
 +  * Total time for request
  
 ====== Status ====== ====== Status ======
  
-  * Richards test has been ported but has an error that is yet to be resolved - Ólafur W (07.04.2009) +===== Completed Tests ===== 
-  * Currently working on porting DeltaBlue - Ólafur W (07.04.2009)+ 
 +  * raytracer (28.04.2009) 
 +  * deltablue (04.05.2009) 
 +  * crypto    (07.06.2009) 
 +  * whirl & j4p5 
 +  * crypto-md5 (29.06.2009) 
 +  * richards (29.06.2009) 
 +  * Crypto-AES 
 + 
 +===== Still unfinished ===== 
  
 ====== Collaboration ====== ====== Collaboration ======
Line 201: Line 239:
   * Ólafur Waage - olafurw [at] gmail.com   * Ólafur Waage - olafurw [at] gmail.com
   * Michiaki Tatsubori - mich [at] acm.org   * Michiaki Tatsubori - mich [at] acm.org
 +  * Alexander Hjalmarsson - hjalle [at] sgh.se
 +  * Davide Mendolia - idaf1er [at] gmail.com
  
rfc/better_benchmarks.txt · Last modified: 2017/09/22 13:28 by 127.0.0.1

Page Tools