rfc:better_benchmarks
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
rfc:better_benchmarks [2009/03/09 04:30] – Fixed typos in a link notation. mich | rfc:better_benchmarks [2009/07/20 13:10] – Added AES olafurw | ||
---|---|---|---|
Line 3: | Line 3: | ||
* Date: 2009-02-01 | * Date: 2009-02-01 | ||
* Author: Paul Biggar < | * Author: Paul Biggar < | ||
- | * Status: | + | * Status: |
**Contributors: | **Contributors: | ||
Line 49: | Line 49: | ||
* Develop simple CLI benchmarks with the criteria " | * Develop simple CLI benchmarks with the criteria " | ||
* The best idea is probably to port existing javascript benchmarks (see bottom of page) | * The best idea is probably to port existing javascript benchmarks (see bottom of page) | ||
- | * It might be good to do this as part of the 2009 test-fest? | + | * Tools to analyse and compare |
* Build a larger suite based on the infrastructure created from the first part | * Build a larger suite based on the infrastructure created from the first part | ||
Line 92: | Line 92: | ||
* [[http:// | * [[http:// | ||
* Short, doesnt take command line parameters, is varied, but simple. | * Short, doesnt take command line parameters, is varied, but simple. | ||
- | * Many benchmarks are copied from the language shootout | + | * Many benchmarks are copied from the language shootout |
- | * This seems to be the only test with data that needs to be run-through. This makes it a somewhat better test than the others, it seems. | + | * Some tests come with input data |
==== Large benchmarks | ==== Large benchmarks | ||
Line 103: | Line 103: | ||
* Requires a number of machines: for DB, front-end and clients. Difficult to deploy. Results difficult to replicate for other developers. | * Requires a number of machines: for DB, front-end and clients. Difficult to deploy. Results difficult to replicate for other developers. | ||
* This benchmarks the entire stack, not just PHP. If PHP is not the bottleneck, then the ' | * This benchmarks the entire stack, not just PHP. If PHP is not the bottleneck, then the ' | ||
+ | * More information in paper: [[http:// | ||
+ | |||
* [[http:// | * [[http:// | ||
Line 112: | Line 114: | ||
* Written in PHP 5. Has DB backends for MySQL, Oracle and SQLServer | * Written in PHP 5. Has DB backends for MySQL, Oracle and SQLServer | ||
* TODO: take a closer look to see if it can be used | * TODO: take a closer look to see if it can be used | ||
+ | |||
+ | * [[http:// | ||
+ | * An auction site application modeled after eBay. | ||
+ | * Implementation variations in Java servlets and EJB are also available and compared in Middleware 2003 paper [[[http:// | ||
===== Applications which could make good web-app benchmarks ===== | ===== Applications which could make good web-app benchmarks ===== | ||
Line 138: | Line 144: | ||
* Not very useful if it is IO bound | * Not very useful if it is IO bound | ||
* Lots of data sets: can use any PHP package | * Lots of data sets: can use any PHP package | ||
+ | * NOTE: Seems to require graphviz | ||
* [[http:// | * [[http:// | ||
Line 150: | Line 157: | ||
* These will probably take a day each to port | * These will probably take a day each to port | ||
* The data sets are built into the applications | * The data sets are built into the applications | ||
+ | * The sunspider ports are probably easier to work with | ||
* Sunspider: http:// | * Sunspider: http:// | ||
* The data sets are (//I think//) built into the applications | * The data sets are (//I think//) built into the applications | ||
+ | * The most useful benchmarks, in order, appear to be (includes V8 benchmarks): | ||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | |||
+ | ===== Benchmarking in other languages ===== | ||
+ | |||
+ | * Python: http:// | ||
+ | |||
+ | * Ruby: http:// | ||
+ | |||
+ | ===== Desired Benchmark Features ===== | ||
+ | |||
+ | ==== For command-line apps applications ==== | ||
+ | * Run-time | ||
+ | * Memory usage | ||
+ | * Hardware performance counters (if available), using PAPI | ||
+ | * Simulated hardware statistics, using cachegrind | ||
+ | * These should be combined into a single representative number, using some hardware model. | ||
+ | * Ability to compare all of these over two runs | ||
+ | * Support for other PHP implementations | ||
+ | * Benchmark characterisation: | ||
+ | * This is hard to do properly, so best to do it badly and give coarse grained information. | ||
+ | |||
+ | *TODO* | ||
+ | |||
+ | ==== For web-apps ==== | ||
+ | |||
+ | * Requests per second | ||
+ | * Memory usage | ||
+ | * Bottleneck (is it scripting, DB, or network) | ||
+ | * Total time for request | ||
+ | |||
+ | ====== Status ====== | ||
+ | |||
+ | ===== Completed Tests ===== | ||
+ | |||
+ | * raytracer (28.04.2009) | ||
+ | * deltablue (04.05.2009) | ||
+ | * crypto | ||
+ | * whirl & j4p5 | ||
+ | * crypto-md5 (29.06.2009) | ||
+ | * richards (29.06.2009) | ||
+ | * Crypto-AES | ||
+ | |||
+ | ===== Still unfinished ===== | ||
+ | |||
====== Collaboration ====== | ====== Collaboration ====== | ||
Line 176: | Line 235: | ||
Add your name here if you want to help. | Add your name here if you want to help. | ||
- | * Paul Biggar - paul.biggar [at] gmail.com | + | * Paul Biggar - paul.biggar [at] gmail.com |
- | * Nuno Lopes - nlopess [at] php.net | + | * Nuno Lopes - nlopess [at] php.net |
* Ólafur Waage - olafurw [at] gmail.com | * Ólafur Waage - olafurw [at] gmail.com | ||
* Michiaki Tatsubori - mich [at] acm.org | * Michiaki Tatsubori - mich [at] acm.org | ||
+ | * Alexander Hjalmarsson - hjalle [at] sgh.se | ||
+ | * Davide Mendolia - idaf1er [at] gmail.com | ||
rfc/better_benchmarks.txt · Last modified: 2017/09/22 13:28 by 127.0.0.1