rfc:better_benchmarks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
rfc:better_benchmarks [2009/04/07 14:23] – Added Status olafurwrfc:better_benchmarks [2017/09/22 13:28] (current) – external edit 127.0.0.1
Line 49: Line 49:
     * Develop simple CLI benchmarks with the criteria "better than bench.php"     * Develop simple CLI benchmarks with the criteria "better than bench.php"
       * The best idea is probably to port existing javascript benchmarks (see bottom of page)       * The best idea is probably to port existing javascript benchmarks (see bottom of page)
-    * It might be good to do this as part of the 2009 test-fest?+    * Tools to analyse and compare test runs.
  
   * Build a larger suite based on the infrastructure created from the first part   * Build a larger suite based on the infrastructure created from the first part
Line 92: Line 92:
   * [[http://www.roadsend.com/home/index.php?pageID=benchmarks|Roadsend]]   * [[http://www.roadsend.com/home/index.php?pageID=benchmarks|Roadsend]]
     * Short, doesnt take command line parameters, is varied, but simple.     * Short, doesnt take command line parameters, is varied, but simple.
-    * Many benchmarks are copied from the language shootout +    * Many benchmarks are copied from the language shootout (old version thereof, apparently) 
-    * This seems to be the only test with data that needs to be run-through. This makes it a somewhat better test than the others, it seems.+    * Some tests come with input data
  
 ==== Large benchmarks  ==== ==== Large benchmarks  ====
Line 103: Line 103:
     * Requires a number of machines: for DB, front-end and clients. Difficult to deploy. Results difficult to replicate for other developers.     * Requires a number of machines: for DB, front-end and clients. Difficult to deploy. Results difficult to replicate for other developers.
     * This benchmarks the entire stack, not just PHP. If PHP is not the bottleneck, then the 'improvements' being tested will not appear. This can be an advantage - small improvements may not increase the speed of an web application in real life. However, it obscures what we are trying to measure - the speed of the PHP implementation.     * This benchmarks the entire stack, not just PHP. If PHP is not the bottleneck, then the 'improvements' being tested will not appear. This can be an advantage - small improvements may not increase the speed of an web application in real life. However, it obscures what we are trying to measure - the speed of the PHP implementation.
 +    * More information in paper: [[http://open.spec.org/workshops/2008/sanfrancisco/papers/Warner.doc|SPECweb2005®™ in the Real World: Using Internet Information Server (IIS) and PHP]] by Warner and Worley.
 +
  
   * [[http://jmob.objectweb.org/rubbos.html|RUBBoS: Bulletin Board Benchmark]]   * [[http://jmob.objectweb.org/rubbos.html|RUBBoS: Bulletin Board Benchmark]]
Line 142: Line 144:
     * Not very useful if it is IO bound     * Not very useful if it is IO bound
     * Lots of data sets: can use any PHP package     * Lots of data sets: can use any PHP package
 +      * NOTE: Seems to require graphviz
  
   * [[http://westhoffswelt.de/projects/phpWhirl.html|phpWhirl]]   * [[http://westhoffswelt.de/projects/phpWhirl.html|phpWhirl]]
Line 154: Line 157:
     * These will probably take a day each to port     * These will probably take a day each to port
     * The data sets are built into the applications     * The data sets are built into the applications
 +    * The sunspider ports are probably easier to work with
  
   * Sunspider: http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/   * Sunspider: http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/
     * The data sets are (//I think//) built into the applications     * The data sets are (//I think//) built into the applications
 +    * The most useful benchmarks, in order, appear to be (includes V8 benchmarks):
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/v8-richards.js|Richards]] (V8)
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/v8-deltablue.js|Deltablue]] (V8)
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/v8-crypto.js|V8-Crypto]] (V8)
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/date-format-xparb.js|xparb]]
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/crypto-aes.js|Crypto-AES]]
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/crypto-md5.js|Crypto-MD5]]
 +      * [[http://svn.webkit.org/repository/webkit/trunk/SunSpider/tests/3d-raytrace.js|3d-raytrace]] (ignoring the actual drawing from JS)
 +
 +===== Benchmarking in other languages =====
 +
 +* Python: http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Performance
 +
 +* Ruby: http://groups.google.com/group/ruby-benchmark-suite
 +
 +===== Desired Benchmark Features =====
 +
 +==== For command-line apps applications ====
 +  * Run-time
 +  * Memory usage
 +  * Hardware performance counters (if available), using PAPI
 +  * Simulated hardware statistics, using cachegrind
 +    * These should be combined into a single representative number, using some hardware model.
 +  * Ability to compare all of these over two runs
 +  * Support for other PHP implementations
 +  * Benchmark characterisation:
 +    * This is hard to do properly, so best to do it badly and give coarse grained information.
 +
 +*TODO*
 +
 +==== For web-apps ====
 +
 +  * Requests per second
 +  * Memory usage
 +  * Bottleneck (is it scripting, DB, or network)
 +  * Total time for request
  
 ====== Status ====== ====== Status ======
  
-  Richards test has been ported but has an error that is yet to be resolved - Ólafur W (07.04.2009) +===== Completed Tests ===== 
-  * Currently working on porting DeltaBlue - Ólafur W (07.04.2009)+ 
 +  raytracer (28.04.2009) 
 +  * deltablue (04.05.2009) 
 +  * crypto    (07.06.2009) 
 +  * whirl & j4p5 
 +  * crypto-md5 (29.06.2009) 
 +  * richards (29.06.2009) 
 +  * Crypto-AES 
 + 
 +===== Still unfinished ===== 
  
 ====== Collaboration ====== ====== Collaboration ======
Line 185: Line 235:
 Add your name here if you want to help. Add your name here if you want to help.
  
-  * Paul Biggar - paul.biggar [at] gmail.com - Proposer +  * Paul Biggar - paul.biggar [at] gmail.com 
-  * Nuno Lopes - nlopess [at] php.net - Proposer+  * Nuno Lopes - nlopess [at] php.net
   * Ólafur Waage - olafurw [at] gmail.com   * Ólafur Waage - olafurw [at] gmail.com
   * Michiaki Tatsubori - mich [at] acm.org   * Michiaki Tatsubori - mich [at] acm.org
 +  * Alexander Hjalmarsson - hjalle [at] sgh.se
 +  * Davide Mendolia - idaf1er [at] gmail.com
  
rfc/better_benchmarks.1239114180.txt.gz · Last modified: 2017/09/22 13:28 (external edit)