rfc:better_benchmarks
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
| rfc:better_benchmarks [2009/04/21 09:27] – Add link to SPECWeb2005 paper pbiggar | rfc:better_benchmarks [2009/07/20 13:10] – Added AES olafurw | ||
|---|---|---|---|
| Line 49: | Line 49: | ||
| * Develop simple CLI benchmarks with the criteria " | * Develop simple CLI benchmarks with the criteria " | ||
| * The best idea is probably to port existing javascript benchmarks (see bottom of page) | * The best idea is probably to port existing javascript benchmarks (see bottom of page) | ||
| - | * It might be good to do this as part of the 2009 test-fest? | + | * Tools to analyse and compare |
| * Build a larger suite based on the infrastructure created from the first part | * Build a larger suite based on the infrastructure created from the first part | ||
| Line 92: | Line 92: | ||
| * [[http:// | * [[http:// | ||
| * Short, doesnt take command line parameters, is varied, but simple. | * Short, doesnt take command line parameters, is varied, but simple. | ||
| - | * Many benchmarks are copied from the language shootout | + | * Many benchmarks are copied from the language shootout |
| - | * This seems to be the only test with data that needs to be run-through. This makes it a somewhat better test than the others, it seems. | + | * Some tests come with input data |
| ==== Large benchmarks | ==== Large benchmarks | ||
| Line 144: | Line 144: | ||
| * Not very useful if it is IO bound | * Not very useful if it is IO bound | ||
| * Lots of data sets: can use any PHP package | * Lots of data sets: can use any PHP package | ||
| + | * NOTE: Seems to require graphviz | ||
| * [[http:// | * [[http:// | ||
| Line 156: | Line 157: | ||
| * These will probably take a day each to port | * These will probably take a day each to port | ||
| * The data sets are built into the applications | * The data sets are built into the applications | ||
| + | * The sunspider ports are probably easier to work with | ||
| * Sunspider: http:// | * Sunspider: http:// | ||
| * The data sets are (//I think//) built into the applications | * The data sets are (//I think//) built into the applications | ||
| + | * The most useful benchmarks, in order, appear to be (includes V8 benchmarks): | ||
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | * [[http:// | ||
| + | |||
| + | ===== Benchmarking in other languages ===== | ||
| + | |||
| + | * Python: http:// | ||
| + | |||
| + | * Ruby: http:// | ||
| + | |||
| + | ===== Desired Benchmark Features ===== | ||
| + | |||
| + | ==== For command-line apps applications ==== | ||
| + | * Run-time | ||
| + | * Memory usage | ||
| + | * Hardware performance counters (if available), using PAPI | ||
| + | * Simulated hardware statistics, using cachegrind | ||
| + | * These should be combined into a single representative number, using some hardware model. | ||
| + | * Ability to compare all of these over two runs | ||
| + | * Support for other PHP implementations | ||
| + | * Benchmark characterisation: | ||
| + | * This is hard to do properly, so best to do it badly and give coarse grained information. | ||
| + | |||
| + | *TODO* | ||
| + | |||
| + | ==== For web-apps ==== | ||
| + | |||
| + | * Requests per second | ||
| + | * Memory usage | ||
| + | * Bottleneck (is it scripting, DB, or network) | ||
| + | * Total time for request | ||
| ====== Status ====== | ====== Status ====== | ||
| - | | + | ===== Completed Tests ===== |
| - | * Currently working on porting DeltaBlue - Ólafur W (07.04.2009) | + | |
| + | | ||
| + | * deltablue (04.05.2009) | ||
| + | * crypto | ||
| + | * whirl & j4p5 | ||
| + | * crypto-md5 (29.06.2009) | ||
| + | * richards (29.06.2009) | ||
| + | * Crypto-AES | ||
| + | |||
| + | ===== Still unfinished ===== | ||
| ====== Collaboration ====== | ====== Collaboration ====== | ||
| Line 191: | Line 239: | ||
| * Ólafur Waage - olafurw [at] gmail.com | * Ólafur Waage - olafurw [at] gmail.com | ||
| * Michiaki Tatsubori - mich [at] acm.org | * Michiaki Tatsubori - mich [at] acm.org | ||
| + | * Alexander Hjalmarsson - hjalle [at] sgh.se | ||
| + | * Davide Mendolia - idaf1er [at] gmail.com | ||
rfc/better_benchmarks.txt · Last modified: 2017/09/22 13:28 by 127.0.0.1