This wikipage contains all extra information for the talk prepared for phptestfest 2009. They heavily rely on content written by other people before. Every resource of the talk, even the slides (in different formats) will be in this place. Other testfesters can use and modify this talk to save precious time.
A phpt test is a little script used by the php internal and quality assurance teams to test PHP's functionality. It can be used with new releases to make sure they can do all the things that previous releases can, or to help find bugs in current releases. By writing phpt tests you are helping to make PHP more stable.
All that is really needed to write a phpt test is
Required Packages (debian)
First things first. For all the testfest fun you need to checkout all the sourcecode for PP 5.3
For easy handling of the PHP CVS repo. you can setup some variables in you ./~cvsrc file
cvs -z3 update -d -P checkout -P diff -u
cvs -d :pserver:cvsread@cvs.php.net:/repository login
Testfest 2009 is all about the PHP 5.3 release. So checkout the right part of the source tree
cvs -d :pserver:cvsread@cvs.php.net:/repository checkout -r PHP_5_3 php5
Now you have all the sourcecode you neeed to start testing in a folder php5.
Running phpt tests is part of the build process.
Start by preparing the configuration of your build by executing the buildconf
./buildconf
Everything is now set up to write PHPT tests or just explore and hack the PHP source code. If you want to configure and compile PHP, use the following commands:
./configure && make && make test
If you also want to enable code coverage results, use the following command:
./configure --enable-gcov && make && make lcov
The build will run for a while, strongly depending on the performance of your computer.
While files are compiled the output will look like this:
/bin/sh /home/sschuermann/php/php5/libtool --silent --preserve-dup-deps --mode=compile cc -Isapi/cgi/ -I/home/sschuermann/php/php5/sapi/cgi/ .....
As soon as the main test process starts output will look like this:
PASS Test array_combine() function : usage variations - different arrays(Bug#43424) [ext/standard/tests/array/array_combine_variation3.phpt]
When there are one or more failing tests a special output is presented to you.
===================================================================== EXPECTED FAILED TEST SUMMARY --------------------------------------------------------------------- ob_start(): Ensure unerasable buffer cannot be flushed by ob_flush(). [tests/output/ob_start_basic_unerasable_005.phpt] Inconsistencies when accessing protected members [Zend/tests/access_modifiers_008.phpt] Inconsistencies when accessing protected members - 2 [Zend/tests/access_modifiers_009.phpt] Bug #42718 (unsafe_raw filter not applied when configured as default filter) [ext/filter/tests/bug42718.phpt] SimpleXML: array casting bug [ext/simplexml/tests/034.phpt] ===================================================================== You may have found a problem in PHP. We would like to send this report automatically to the PHP QA team, to give us a better understanding of how the test cases are doing. If you don't want to send it immediately, you can choose "s" to save the report to a file that you can send us later. Do you want to send this report now? [Yns]:
There is a set of things that need to be prepared before you can actually start execute the tests.
You must use TEST_PHP_EXECUTABLE environment variable to explicitly select the php executable to be used to run the tests. That can either be the CLI or CGI executable. Command “make test” executes “run-tests.php” script with “php” binary. Some test scripts such as session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all tests.
TEST_PHP_EXECUTABLE=sapi/cli/php
and
export TEST_PHP_EXECUTABLE
You can re-run the same Tests as executed in make test when you execute the file run-tests.php
php run-tests.php
It is possible to execute a single test or tests residing in one directory. Again run-tests.php helps here.
A single Test
php run-tests.php tests/func/test010.phpt
A directory with tests
php run-tests.php tests/func/
From here on it's easy to add own tests.
--TEST-- Hello world test --FILE-- <?php echo "Hello"; ?> --CLEAN-- --EXPECT-- Hello
There are certain guidelines you can follow when writing tests. There are several guidelines that you can follow to get working test code that is solid and makes sense. Several basics apply in general
When a test uses external resources, such as a file containing test data, the test is no longer self contained. Consequently, there is not enough information to understand the tested functionality, making it hard to use that test as documentation. Moreover, using external resources introduces hidden de- pendencies: if some force changes or deletes such a re- source, tests start failing. Chances for this increase when more tests use the same resource. The use of external re- sources can be eliminated using the refactoring Inline Re- source. If external resources are needed, you can apply Setup External Resource to remove hidden dependen- cies.
Examples:
Avoid by
Test code that makes optimistic assumptions about the ex- istence (or absence) and state of external resources (such as particular directories or database tables) can cause non- deterministic behavior in test outcomes. The situation where tests run fine at one time and fail miserably the other time is not a situation you want to find yourself in. Use Setup External Resource to allocate and/or initialize all resources that are used.
Examples:
Avoid By
Such wars arise when the tests run fine as long as you are the only one testing but fail when more programmers run them. This is most likely caused by resource interference: some tests in your suite allocate resources such as tempo- rary files that are also used by others. Apply Make Re- source Unique (3) to overcome interference.
Examples:
Avoid by:
When a test method checks several methods of the object to be tested, it is hard to read and understand, and therefore more difficult to use as documentation. Moreover, it makes tests more dependent on each other and harder to maintain. The solution is simple: separate the test code into test methods that test only one method using Fowler’s Extract Method (F:110), using a meaningful name highlighting the purpose of the test. Note that splitting into smaller methods can slow down the tests due to increased setup/teardown overhead.
Examples:
Avoid
Phpt tests follow a very strict naming convention. This is done to easily identify what each phpt test is for.
The tests for Bugs contain the id of the bug according to the bug id in the bugtracker
Basic funcion behaviour is a simple call to the function checking on the normal behaviour of it.
This kind of test covers the fnctions behaviour in case on an error. Do not mix this up with a bug.
phpt Test Basics http://qa.php.net/write-test.php
Guide to anonymous CVS checkout http://de3.php.net/anoncvs.php
Zoes 2008 Testfest Talk http://www.phplondon.org/conference/2008/media/docs/TestOrDie_Zoe_Slattery.pdf
Sebastian Bergmanns Slides 2008 http://www.slideshare.net/sebastian_bergmann/php-testfest-cologne?src=embed
Requirements for tests on mac http://felix.phpbelgium.be/blog/2008/05/22/writing-phpt-tests-on-your-mac/