====== PHPT roundup talk ====== This wikipage contains all extra information for the talk prepared for [[qa:testfest-2009|phptestfest 2009]]. They heavily rely on content written by other people before. Every resource of the talk, even the slides (in different formats) will be in this place. Other testfesters can use and modify this talk to save precious time. [[http://tinyurl.com/c9bqf4]] ===== How is PHP Tested ===== ==== What is a PHPT ==== A phpt test is a little script used by the php internal and quality assurance teams to test PHP's functionality. It can be used with new releases to make sure they can do all the things that previous releases can, or to help find bugs in current releases. By writing phpt tests you are helping to make PHP more stable. ==== Requirements ==== All that is really needed to write a phpt test is * a basic understanding of the PHP language * a text editor * a way to execute the code Required Packages (debian) * the make commando available * autoconf * gcc * flex-old * libxml2-dev * re2c * Bison (before php 5.2) * lcov / ggcov ===== Get the source ===== First things first. For all the testfest fun you need to checkout all the sourcecode for PP 5.3 ==== CVS Configuration ==== For easy handling of the PHP CVS repo. you can setup some variables in you ./~cvsrc file cvs -z3 update -d -P checkout -P diff -u ==== Login to CVS ==== cvs -d :pserver:cvsread@cvs.php.net:/repository login * Username: cvsread * password: phpfi ==== Checkout ==== Testfest 2009 is all about the PHP 5.3 release. So checkout the right part of the source tree cvs -d :pserver:cvsread@cvs.php.net:/repository checkout -r PHP_5_3 php5 Now you have all the sourcecode you neeed to start testing in a folder php5. ===== Compile and run the tests ===== Running phpt tests is part of the build process. Start by preparing the configuration of your build by executing the buildconf ./buildconf Everything is now set up to write PHPT tests or just explore and hack the PHP source code. If you want to configure and compile PHP, use the following commands: ./configure && make && make test If you also want to enable code coverage results, use the following command: ./configure --enable-gcov && make && make lcov The build will run for a while, strongly depending on the performance of your computer. While files are compiled the output will look like this: /bin/sh /home/sschuermann/php/php5/libtool --silent --preserve-dup-deps --mode=compile cc -Isapi/cgi/ -I/home/sschuermann/php/php5/sapi/cgi/ ..... As soon as the main test process starts output will look like this: PASS Test array_combine() function : usage variations - different arrays(Bug#43424) [ext/standard/tests/array/array_combine_variation3.phpt] When there are one or more failing tests a special output is presented to you. ===================================================================== EXPECTED FAILED TEST SUMMARY --------------------------------------------------------------------- ob_start(): Ensure unerasable buffer cannot be flushed by ob_flush(). [tests/output/ob_start_basic_unerasable_005.phpt] Inconsistencies when accessing protected members [Zend/tests/access_modifiers_008.phpt] Inconsistencies when accessing protected members - 2 [Zend/tests/access_modifiers_009.phpt] Bug #42718 (unsafe_raw filter not applied when configured as default filter) [ext/filter/tests/bug42718.phpt] SimpleXML: array casting bug [ext/simplexml/tests/034.phpt] ===================================================================== You may have found a problem in PHP. We would like to send this report automatically to the PHP QA team, to give us a better understanding of how the test cases are doing. If you don't want to send it immediately, you can choose "s" to save the report to a file that you can send us later. Do you want to send this report now? [Yns]: ===== Execute tests ===== There is a set of things that need to be prepared before you can actually start execute the tests. ==== Which "php" executable "make test" look for ==== You must use TEST_PHP_EXECUTABLE environment variable to explicitly select the php executable to be used to run the tests. That can either be the CLI or CGI executable. Command "make test" executes "run-tests.php" script with "php" binary. Some test scripts such as session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all tests. TEST_PHP_EXECUTABLE=sapi/cli/php and export TEST_PHP_EXECUTABLE ==== All Tests ==== You can re-run the same Tests as executed in ''make test'' when you execute the file run-tests.php php run-tests.php ==== Single Tests ==== It is possible to execute a single test or tests residing in one directory. Again run-tests.php helps here. A single Test php run-tests.php tests/func/test010.phpt A directory with tests php run-tests.php tests/func/ ===== Write Tests ===== From here on it's easy to add own tests. --TEST-- Hello world test --FILE-- --CLEAN-- --EXPECT-- Hello ===== Write good Tests... ===== There are certain guidelines you can follow when writing tests. There are several guidelines that you can follow to get working test code that is solid and makes sense. Several basics apply in general * KISS Keep the Test short in general. * Test one functionality at a time * Try to make execute the test fast, waiting for the next eclipse to proof the date() function fails will slow down build times * Document the outcome. People using your tests will not always be experts for that specific tests * Place Links to resources as they help understand the test. A test is as much about documentation as on prooving fail or working condition of a function * Extension docs * Bug reports * If you are testing on a bug, make sure you got the bottom line of it ==== Mystery Guests ==== // When a test uses external resources, such as a file containing test data, the test is no longer self contained. Consequently, there is not enough information to understand the tested functionality, making it hard to use that test as documentation. Moreover, using external resources introduces hidden de- pendencies: if some force changes or deletes such a re- source, tests start failing. Chances for this increase when more tests use the same resource. The use of external re- sources can be eliminated using the refactoring Inline Re- source. If external resources are needed, you can apply Setup External Resource to remove hidden dependen- cies.// Examples: * storing data in data files Avoid by * Store Data in the Test * Write Setup Code ==== Resource Optimism ==== //Test code that makes optimistic assumptions about the ex- istence (or absence) and state of external resources (such as particular directories or database tables) can cause non- deterministic behavior in test outcomes. The situation where tests run fine at one time and fail miserably the other time is not a situation you want to find yourself in. Use Setup External Resource to allocate and/or initialize all resources that are used.// Examples: * Files in /tmp (there is no /tmp in windows) * extensions installed * Special variants/setups of extensions installed Avoid By * Write Setup Code for the resource * Do not re-use resources through multiple tests ==== Test Run Wars ==== //Such wars arise when the tests run fine as long as you are the only one testing but fail when more programmers run them. This is most likely caused by resource interference: some tests in your suite allocate resources such as tempo- rary files that are also used by others. Apply Make Re- source Unique (3) to overcome interference. // Examples: * Creating files in a public directory (/tmp/ if choosen right ^^) under a fixed file name Avoid by: * Adding Random or user based Value to file name ==== Eager Test ==== //When a test method checks several methods of the object to be tested, it is hard to read and understand, and therefore more difficult to use as documentation. Moreover, it makes tests more dependent on each other and harder to maintain. The solution is simple: separate the test code into test methods that test only one method using Fowler’s Extract Method (F:110), using a meaningful name highlighting the purpose of the test. Note that splitting into smaller methods can slow down the tests due to increased setup/teardown overhead.// Examples: * Tests of different behaviour in one test * More than one method is tested Avoid * Split into several smaller tests ==== ... for Coverage ==== ==== ... to find bugs ==== ==== Naming conventions for Tests ... ==== Phpt tests follow a very strict naming convention. This is done to easily identify what each phpt test is for. === ... on bugs === The tests for Bugs contain the id of the bug according to the bug id in the bugtracker * Pattern bug.phpt * bug17123.phpt === ... basic behaviour === Basic funcion behaviour is a simple call to the function checking on the normal behaviour of it. * Pattern _basic.phpt * dba_open_basic.phpt === ... error behaviour === This kind of test covers the fnctions behaviour in case on an error. Do not mix this up with a bug. * Pattern _error.phpt * dba_open_error.phpt === ... variations in function behaviour === * Pattern _variation.phpt * dba_open_variation.phpt === ... on extensions === * Pattern .phpt * dba_003.phpt ===== Resources ===== phpt Test Basics [[http://qa.php.net/write-test.php]] Guide to anonymous CVS checkout [[http://de3.php.net/anoncvs.php]] Zoes 2008 Testfest Talk [[http://www.phplondon.org/conference/2008/media/docs/TestOrDie_Zoe_Slattery.pdf]] Sebastian Bergmanns Slides 2008 [[ http://www.slideshare.net/sebastian_bergmann/php-testfest-cologne?src=embed]] Requirements for tests on mac [[http://felix.phpbelgium.be/blog/2008/05/22/writing-phpt-tests-on-your-mac/]]