rfc:better_benchmarks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
rfc:better_benchmarks [2009/04/21 09:48] – Add link for 3d-raytrace pbiggarrfc:better_benchmarks [2017/09/22 13:28] (current) – external edit 127.0.0.1
Line 92: Line 92:
   * [[http://www.roadsend.com/home/index.php?pageID=benchmarks|Roadsend]]   * [[http://www.roadsend.com/home/index.php?pageID=benchmarks|Roadsend]]
     * Short, doesnt take command line parameters, is varied, but simple.     * Short, doesnt take command line parameters, is varied, but simple.
-    * Many benchmarks are copied from the language shootout +    * Many benchmarks are copied from the language shootout (old version thereof, apparently) 
-    * This seems to be the only test with data that needs to be run-through. This makes it a somewhat better test than the others, it seems.+    * Some tests come with input data
  
 ==== Large benchmarks  ==== ==== Large benchmarks  ====
Line 144: Line 144:
     * Not very useful if it is IO bound     * Not very useful if it is IO bound
     * Lots of data sets: can use any PHP package     * Lots of data sets: can use any PHP package
 +      * NOTE: Seems to require graphviz
  
   * [[http://westhoffswelt.de/projects/phpWhirl.html|phpWhirl]]   * [[http://westhoffswelt.de/projects/phpWhirl.html|phpWhirl]]
Line 169: Line 170:
       * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/3d-raytrace.js|3d-raytrace]] (ignoring the actual drawing from JS)       * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/3d-raytrace.js|3d-raytrace]] (ignoring the actual drawing from JS)
  
 +===== Benchmarking in other languages =====
 +
 +* Python: http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Performance
 +
 +* Ruby: http://groups.google.com/group/ruby-benchmark-suite
 +
 +===== Desired Benchmark Features =====
 +
 +==== For command-line apps applications ====
 +  * Run-time
 +  * Memory usage
 +  * Hardware performance counters (if available), using PAPI
 +  * Simulated hardware statistics, using cachegrind
 +    * These should be combined into a single representative number, using some hardware model.
 +  * Ability to compare all of these over two runs
 +  * Support for other PHP implementations
 +  * Benchmark characterisation:
 +    * This is hard to do properly, so best to do it badly and give coarse grained information.
 +
 +*TODO*
 +
 +==== For web-apps ====
 +
 +  * Requests per second
 +  * Memory usage
 +  * Bottleneck (is it scripting, DB, or network)
 +  * Total time for request
  
 ====== Status ====== ====== Status ======
  
-  Richards test has been ported but has an error that is yet to be resolved - Ólafur W (07.04.2009) +===== Completed Tests ===== 
-  * Currently working on porting DeltaBlue - Ólafur W (07.04.2009)+ 
 +  raytracer (28.04.2009) 
 +  * deltablue (04.05.2009) 
 +  * crypto    (07.06.2009) 
 +  * whirl & j4p5 
 +  * crypto-md5 (29.06.2009) 
 +  * richards (29.06.2009) 
 +  * Crypto-AES 
 + 
 +===== Still unfinished ===== 
  
 ====== Collaboration ====== ====== Collaboration ======
Line 201: Line 239:
   * Ólafur Waage - olafurw [at] gmail.com   * Ólafur Waage - olafurw [at] gmail.com
   * Michiaki Tatsubori - mich [at] acm.org   * Michiaki Tatsubori - mich [at] acm.org
 +  * Alexander Hjalmarsson - hjalle [at] sgh.se
 +  * Davide Mendolia - idaf1er [at] gmail.com
  
rfc/better_benchmarks.1240307311.txt.gz · Last modified: 2017/09/22 13:28 (external edit)