The PHP CLI benchmark is a script that runs several algorithms and test runs and gives back results in forms of memory consumption or cache misses depending on which tool that is used. The algorithms that are used are similar (or clones) to the ones used in the V8 javascript benchmark and the Sunspider javascript benchmark.

Current available tools:

cachegrind, memusage

The whole benchmark requires PEAR.

For the tool memusage, linux with kernel 2.6.14 or newer is required. (/proc/PID/smaps)

For the tool cachegrind, the software valgrind (http://valgrind.org/) is required.

bench.php --tool memusage --complog -v

This would do a benchmark run in two default test-directories (microtests and tests) and print a total memory usage for the run. It will also (with the -v) display each tests memory usage. --complog will create a serialized log file that will be useful when comparing runs. The comparison log file will look like “bench_PHPVERSION_TOOL_PID.txt”.

bench.php --tool memusage --path microtests -q --complog

This run will just include the microtests path and be quiet (-q, not produce output). There will be a logfile created after this run that can be used for comparison.

bench.php --tool cachegrind --complog

This will do a cachegrind benchmark and show results of the valgrind tool cachegrind. This requires valgrind.

There are also other options available such as:

• --php PATH

Where PATH is a path to a specific PHP Binary. This is useful when running several php versions on the same machine. Default is just “php”. All files in this path that should be tested need to be names test_XXX.php.

• --memory-limit MEM

Where MEM is the maximum memory usage for the test files to consume (individually). The value is in MB, the default is 128.

• --debug

A switch to set to a slightly more verbose debug mode.

• --help

Print the default help message.

• --q

Quite mode (No output)

• --tool TOOL

Select which TOOL that is used. Default is none and current available options are memusage and cachegrind.

• --log FILE

Produces an additional logfile of the output. Default is benchlog.txt, if FILE is set, it will write to FILE.

• --complog

Writes a serialized log file for comparing runs. Looks like bench_PHPVERSION_TOOL_PID.txt

• --comparison FILES

An individual option that compares the FILES selected. FILES can be “bench_*_memusage_*.txt” or selected files, separated by spaces. They must be benched with the same tool.

The comparison output might be changed, but this is how it looks like currently. The number in the parenthesis is the percentage of the “best” value. If the best runtime is 10s, 10s would become 100%. Since I've only used 5.3.0RC4 for comparing, all results are pretty much the same.

To compare the results it's possible to use the --comparison option. In example, if you would like to compare results from 5.X with memory usage the command could be following:

bench.php --comparison bench_5.*_memusage_*

The results will look like this:

bench.php --comparison bench_5.*_memusage_*
--------------- bench.php Comparison --------------
PHP version     PHP uname                           Runtime           Private RSS    
5.3.0RC4        Linux 2.6.28-11-generic i686        147s (101%)       117.5MB (100%)  
5.3.0RC4        Linux 2.6.28-11-generic i686        147.6s (101%)     117.5MB (100%)  
5.3.0RC4        Linux 2.6.28-11-generic i686        146.3s (101%)     117.5MB (100%)  
5.3.0RC4        Linux 2.6.28-11-generic i686        145.4s (100%)     117.6MB (100%)  
--------------- bench.php End Comparison -----------

Private RSS is the summary of the processes memory usage and runtime is just the total runtime.

This is how the cachegrind comparison looks like currently:

bench.php --comparison bench_5.*_cachegrind_*
--------------- bench.php Comparison --------------
PHP version     PHP uname                           Branch Mispred.        Instructions         Cache misses        
5.3.0RC4        Linux 2.6.28-11-generic i686        1.394690e+7 (100%)     7.312482e+9 (100%)   2.419207e+7 (100%)  
5.3.0RC4        Linux 2.6.28-11-generic i686        1.394695e+7 (100%)     7.312483e+9 (100%)   2.419216e+7 (100%)  
--------------- bench.php End Comparison -----------

The web application benchmark works by deploying three virtual machines using XEN. The machines that are deployed are:

Web server with apache and php5 with mysql (in future it will be possible to switch version and compile from scratch).

Client server with php5-cli with curl. This serves as the machine which makes the requests for the server.

Database server with mysql-server. This server is used by the web server.

To be able to run the web applications benchmark, you will have to have a xen environment working along with xen-tools. Xen-tools is used to create and deploy the virtual machines and set up the network connection along with their roles. Their roles are specified in individual files, which can be found in /benchweb/roles/.

To be able to deploy the virtual machines, there are a couple of things that needs to be known. Currently, the config.php file contains information that need to be modified in order to get the network settings to work. In this file, you will have to modify the gateway, netmask and the $HOME_DIR to the directory where the deployment files are, such as config.php and deploy.php.

After those settings have been configured, you can call deploy.php with your starting IP. The IP's will take a range of 3, which means that it will need 3 IP adresses in a row to work. This call:

./deploy 192.168.0.150

Will setup the webserver at 192.168.0.150, the client at 192.168.0.151 and the database server at 192.168.0.152. The creation of these servers take a while because of its nature. It downloads the packages needed and will, when created, boot the images and when they appear to be running, the script will execute some commands that set up their environments. The speed of the installation is very dependent on the speed of your internet connection. This is the setup.php script that is called and it will change some configuration-files and make some adjustments to the system to make it ready for benchmarking.

When the deploy.php script has finished, the servers are ready for installing the web applications (currently only phpBB).

• config.php

Configuration before the deployment of the virtual machines

• deploy.php START-IP

Deployment script that deploys the three virtual machines to START-IP, START-IP + 1 and START-IP + 2.

• functions.php

Functions used for the deployment and upload/execution of commands.

• installers/

This is were the applications are stored

• machines/

This is were machine-specific files are stored, such as configuration-files and such.

• roles/

Here are the role-files places (files that choose which packages that are installed after the deployment)

• setup.php

This is a file that's included when all the servers has booted up in the deployment script and this performs configurations and additional setups.

• shutdown.php

This shutdowns (destroys) the virtual machines and removes the created files (ips.txt which contains the IP-adresses of the VM's) and the known_hosts file that is created when ssh and scp commands are executed.

To install and setup phpBB3 for benchmarking you can change directory to “installers/phpbb/”. The setup file in this directory will download phpBB 3.0.5 and deploy it and setup a slightly modified version of it that enables benchmarking (allows faster postings) along with a database that is setup with some categories. The setup file will also move the bench-directory to the client, which will serve as the files that are used when benchmarking phpBB3.

To execute a benchmark and get the value of requests per second, you simply do:

php setup.php
php benchmark.php Number_of_user_scenarios concurrency

Where Number_of_user_scenarios are a number of record of requests that will be made. A scenario when a user registers can be described like this:

User requests the register-page.
User accepts the terms.
User fills in the information and posts the data.

This scenario is made up of three requests but will only count as one scenario. It's basically the same for creating a thread, posting a reply and viewing a thread.

If you want to configure the size of posts or frequency of creating threads, registering or such you can find the configuration file inside of the /installers/phpbb/bench/config.php file. This file has to be modified before you run the setup.php file, since it is uploaded to the server during this call.

It is fully possible to ssh to the virtual machines and modify files, configurations or anything just like a normal server.