PHPT roundup talk
This wikipage contains all extra information for the talk prepared for phptestfest 2009. They heavily rely on content written by other people before. Every resource of the talk, even the slides (in different formats) will be in this place. Other testfesters can use and modify this talk to save precious time.
How is PHP Tested
What is a PHPT
A phpt test is a little script used by the php internal and quality assurance teams to test PHP's functionality. It can be used with new releases to make sure they can do all the things that previous releases can, or to help find bugs in current releases. By writing phpt tests you are helping to make PHP more stable.
All that is really needed to write a phpt test is
- a basic understanding of the PHP language
- a text editor
- a way to execute the code
Required Packages (debian)
- the make commando available
- Bison (before php 5.2)
- lcov / ggcov
Get the source
First things first. For all the testfest fun you need to checkout all the sourcecode for PP 5.3
For easy handling of the PHP CVS repo. you can setup some variables in you ./~cvsrc file
cvs -z3 update -d -P checkout -P diff -u
Login to CVS
cvs -d :pserver:firstname.lastname@example.org:/repository login
- Username: cvsread
- password: phpfi
Testfest 2009 is all about the PHP 5.3 release. So checkout the right part of the source tree
cvs -d :pserver:email@example.com:/repository checkout -r PHP_5_3 php5
Now you have all the sourcecode you neeed to start testing in a folder php5.
Compile and run the tests
Running phpt tests is part of the build process.
Start by preparing the configuration of your build by executing the buildconf
Everything is now set up to write PHPT tests or just explore and hack the PHP source code. If you want to configure and compile PHP, use the following commands:
./configure && make && make test
If you also want to enable code coverage results, use the following command:
./configure --enable-gcov && make && make lcov
The build will run for a while, strongly depending on the performance of your computer.
While files are compiled the output will look like this:
/bin/sh /home/sschuermann/php/php5/libtool --silent --preserve-dup-deps --mode=compile cc -Isapi/cgi/ -I/home/sschuermann/php/php5/sapi/cgi/ .....
As soon as the main test process starts output will look like this:
PASS Test array_combine() function : usage variations - different arrays(Bug#43424) [ext/standard/tests/array/array_combine_variation3.phpt]
When there are one or more failing tests a special output is presented to you.
===================================================================== EXPECTED FAILED TEST SUMMARY --------------------------------------------------------------------- ob_start(): Ensure unerasable buffer cannot be flushed by ob_flush(). [tests/output/ob_start_basic_unerasable_005.phpt] Inconsistencies when accessing protected members [Zend/tests/access_modifiers_008.phpt] Inconsistencies when accessing protected members - 2 [Zend/tests/access_modifiers_009.phpt] Bug #42718 (unsafe_raw filter not applied when configured as default filter) [ext/filter/tests/bug42718.phpt] SimpleXML: array casting bug [ext/simplexml/tests/034.phpt] ===================================================================== You may have found a problem in PHP. We would like to send this report automatically to the PHP QA team, to give us a better understanding of how the test cases are doing. If you don't want to send it immediately, you can choose "s" to save the report to a file that you can send us later. Do you want to send this report now? [Yns]:
There is a set of things that need to be prepared before you can actually start execute the tests.
Which "php" executable "make test" look for
You must use TEST_PHP_EXECUTABLE environment variable to explicitly select the php executable to be used to run the tests. That can either be the CLI or CGI executable. Command “make test” executes “run-tests.php” script with “php” binary. Some test scripts such as session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all tests.
You can re-run the same Tests as executed in
make test when you execute the file run-tests.php
It is possible to execute a single test or tests residing in one directory. Again run-tests.php helps here.
A single Test
php run-tests.php tests/func/test010.phpt
A directory with tests
php run-tests.php tests/func/
From here on it's easy to add own tests.
--TEST-- Hello world test --FILE-- <?php echo "Hello"; ?> --CLEAN-- --EXPECT-- Hello
Write good Tests...
There are certain guidelines you can follow when writing tests. There are several guidelines that you can follow to get working test code that is solid and makes sense. Several basics apply in general
- KISS Keep the Test short in general.
- Test one functionality at a time
- Try to make execute the test fast, waiting for the next eclipse to proof the date() function fails will slow down build times
- Document the outcome. People using your tests will not always be experts for that specific tests
- Place Links to resources as they help understand the test. A test is as much about documentation as on prooving fail or working condition of a function
- Extension docs
- Bug reports
- If you are testing on a bug, make sure you got the bottom line of it
When a test uses external resources, such as a file containing test data, the test is no longer self contained. Consequently, there is not enough information to understand the tested functionality, making it hard to use that test as documentation. Moreover, using external resources introduces hidden de- pendencies: if some force changes or deletes such a re- source, tests start failing. Chances for this increase when more tests use the same resource. The use of external re- sources can be eliminated using the refactoring Inline Re- source. If external resources are needed, you can apply Setup External Resource to remove hidden dependen- cies.
- storing data in data files
- Store Data in the Test
- Write Setup Code
Test code that makes optimistic assumptions about the ex- istence (or absence) and state of external resources (such as particular directories or database tables) can cause non- deterministic behavior in test outcomes. The situation where tests run fine at one time and fail miserably the other time is not a situation you want to find yourself in. Use Setup External Resource to allocate and/or initialize all resources that are used.
- Files in /tmp (there is no /tmp in windows)
- extensions installed
- Special variants/setups of extensions installed
- Write Setup Code for the resource
- Do not re-use resources through multiple tests
Test Run Wars
Such wars arise when the tests run fine as long as you are the only one testing but fail when more programmers run them. This is most likely caused by resource interference: some tests in your suite allocate resources such as tempo- rary files that are also used by others. Apply Make Re- source Unique (3) to overcome interference.
- Creating files in a public directory (/tmp/ if choosen right ^^) under a fixed file name
- Adding Random or user based Value to file name
When a test method checks several methods of the object to be tested, it is hard to read and understand, and therefore more difficult to use as documentation. Moreover, it makes tests more dependent on each other and harder to maintain. The solution is simple: separate the test code into test methods that test only one method using Fowler’s Extract Method (F:110), using a meaningful name highlighting the purpose of the test. Note that splitting into smaller methods can slow down the tests due to increased setup/teardown overhead.
- Tests of different behaviour in one test
- More than one method is tested
- Split into several smaller tests
... for Coverage
... to find bugs
Naming conventions for Tests ...
Phpt tests follow a very strict naming convention. This is done to easily identify what each phpt test is for.
... on bugs
The tests for Bugs contain the id of the bug according to the bug id in the bugtracker
- Pattern bug<bugid>.phpt
... basic behaviour
Basic funcion behaviour is a simple call to the function checking on the normal behaviour of it.
- Pattern <functionname>_basic.phpt
... error behaviour
This kind of test covers the fnctions behaviour in case on an error. Do not mix this up with a bug.
- Pattern <functionname>_error.phpt
... variations in function behaviour
- Pattern <functionname>_variation.phpt
... on extensions
- Pattern <extname><no>.phpt
phpt Test Basics http://qa.php.net/write-test.php
Guide to anonymous CVS checkout http://de3.php.net/anoncvs.php
Zoes 2008 Testfest Talk http://www.phplondon.org/conference/2008/media/docs/TestOrDie_Zoe_Slattery.pdf
Sebastian Bergmanns Slides 2008 http://www.slideshare.net/sebastian_bergmann/php-testfest-cologne?src=embed
Requirements for tests on mac http://felix.phpbelgium.be/blog/2008/05/22/writing-phpt-tests-on-your-mac/