PHPT roundup talk

This wikipage contains all extra information for the talk prepared for phptestfest 2009. They heavily rely on content written by other people before. Every resource of the talk, even the slides (in different formats) will be in this place. Other testfesters can use and modify this talk to save precious time.

http://tinyurl.com/c9bqf4

A phpt test is a little script used by the php internal and quality assurance teams to test PHP's functionality. It can be used with new releases to make sure they can do all the things that previous releases can, or to help find bugs in current releases. By writing phpt tests you are helping to make PHP more stable.

All that is really needed to write a phpt test is

• a basic understanding of the PHP language
• a text editor
• a way to execute the code

Required Packages (debian)

• the make commando available
• autoconf
• gcc
• flex-old
• libxml2-dev
• re2c
• Bison (before php 5.2)
• lcov / ggcov

First things first. For all the testfest fun you need to checkout all the sourcecode for PP 5.3

For easy handling of the PHP CVS repo. you can setup some variables in you ./~cvsrc file

cvs -z3
update -d -P
checkout -P
diff -u

cvs -d :pserver:cvsread@cvs.php.net:/repository login

• Username: cvsread
• password: phpfi

Testfest 2009 is all about the PHP 5.3 release. So checkout the right part of the source tree

cvs -d :pserver:cvsread@cvs.php.net:/repository checkout -r PHP_5_3 php5

Now you have all the sourcecode you neeed to start testing in a folder php5.

Running phpt tests is part of the build process.

Start by preparing the configuration of your build by executing the buildconf

./buildconf

Everything is now set up to write PHPT tests or just explore and hack the PHP source code. If you want to configure and compile PHP, use the following commands:

./configure && make && make test

If you also want to enable code coverage results, use the following command:

./configure --enable-gcov && make && make lcov

The build will run for a while, strongly depending on the performance of your computer.

While files are compiled the output will look like this:

/bin/sh /home/sschuermann/php/php5/libtool --silent --preserve-dup-deps --mode=compile cc  -Isapi/cgi/ -I/home/sschuermann/php/php5/sapi/cgi/ .....

As soon as the main test process starts output will look like this:

PASS Test array_combine() function : usage variations - different arrays(Bug#43424) [ext/standard/tests/array/array_combine_variation3.phpt] 

When there are one or more failing tests a special output is presented to you.

=====================================================================
EXPECTED FAILED TEST SUMMARY
---------------------------------------------------------------------
ob_start(): Ensure unerasable buffer cannot be flushed by ob_flush(). [tests/output/ob_start_basic_unerasable_005.phpt]
Inconsistencies when accessing protected members [Zend/tests/access_modifiers_008.phpt]
Inconsistencies when accessing protected members - 2 [Zend/tests/access_modifiers_009.phpt]
Bug #42718 (unsafe_raw filter not applied when configured as default filter) [ext/filter/tests/bug42718.phpt]
SimpleXML: array casting bug [ext/simplexml/tests/034.phpt]
=====================================================================

You may have found a problem in PHP.
We would like to send this report automatically to the
PHP QA team, to give us a better understanding of how
the test cases are doing. If you don't want to send it
immediately, you can choose "s" to save the report to
a file that you can send us later.
Do you want to send this report now? [Yns]: 

There is a set of things that need to be prepared before you can actually start execute the tests.

You must use TEST_PHP_EXECUTABLE environment variable to explicitly select the php executable to be used to run the tests. That can either be the CLI or CGI executable. Command “make test” executes “run-tests.php” script with “php” binary. Some test scripts such as session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all tests.

TEST_PHP_EXECUTABLE=sapi/cli/php

and

export TEST_PHP_EXECUTABLE

You can re-run the same Tests as executed in make test when you execute the file run-tests.php

php run-tests.php

It is possible to execute a single test or tests residing in one directory. Again run-tests.php helps here.

A single Test

php run-tests.php tests/func/test010.phpt

A directory with tests

php run-tests.php tests/func/

From here on it's easy to add own tests.

--TEST--
Hello world test
--FILE--
<?php
echo "Hello";
?>
--CLEAN--
--EXPECT--
Hello

There are certain guidelines you can follow when writing tests. There are several guidelines that you can follow to get working test code that is solid and makes sense. Several basics apply in general

• KISS Keep the Test short in general.
• Test one functionality at a time
• Try to make execute the test fast, waiting for the next eclipse to proof the date() function fails will slow down build times
• Document the outcome. People using your tests will not always be experts for that specific tests
• Place Links to resources as they help understand the test. A test is as much about documentation as on prooving fail or working condition of a function
  • Extension docs
  • Bug reports
• If you are testing on a bug, make sure you got the bottom line of it

When a test uses external resources, such as a file containing test data, the test is no longer self contained. Consequently, there is not enough information to understand the tested functionality, making it hard to use that test as documentation. Moreover, using external resources introduces hidden de- pendencies: if some force changes or deletes such a re- source, tests start failing. Chances for this increase when more tests use the same resource. The use of external re- sources can be eliminated using the refactoring Inline Re- source. If external resources are needed, you can apply Setup External Resource to remove hidden dependen- cies.

Examples:

• storing data in data files

Avoid by

• Store Data in the Test
• Write Setup Code

Test code that makes optimistic assumptions about the ex- istence (or absence) and state of external resources (such as particular directories or database tables) can cause non- deterministic behavior in test outcomes. The situation where tests run fine at one time and fail miserably the other time is not a situation you want to find yourself in. Use Setup External Resource to allocate and/or initialize all resources that are used.

Examples:

• Files in /tmp (there is no /tmp in windows)
• extensions installed
• Special variants/setups of extensions installed

Avoid By

• Write Setup Code for the resource
• Do not re-use resources through multiple tests

Such wars arise when the tests run fine as long as you are the only one testing but fail when more programmers run them. This is most likely caused by resource interference: some tests in your suite allocate resources such as tempo- rary files that are also used by others. Apply Make Re- source Unique (3) to overcome interference.

Examples:

• Creating files in a public directory (/tmp/ if choosen right ^^) under a fixed file name

Avoid by:

• Adding Random or user based Value to file name

When a test method checks several methods of the object to be tested, it is hard to read and understand, and therefore more difficult to use as documentation. Moreover, it makes tests more dependent on each other and harder to maintain. The solution is simple: separate the test code into test methods that test only one method using Fowler’s Extract Method (F:110), using a meaningful name highlighting the purpose of the test. Note that splitting into smaller methods can slow down the tests due to increased setup/teardown overhead.

Examples:

• Tests of different behaviour in one test
• More than one method is tested

Avoid

• Split into several smaller tests

Phpt tests follow a very strict naming convention. This is done to easily identify what each phpt test is for.

The tests for Bugs contain the id of the bug according to the bug id in the bugtracker

• Pattern bug<bugid>.phpt
• bug17123.phpt

Basic funcion behaviour is a simple call to the function checking on the normal behaviour of it.

• Pattern <functionname>_basic.phpt
• dba_open_basic.phpt

This kind of test covers the fnctions behaviour in case on an error. Do not mix this up with a bug.

• Pattern <functionname>_error.phpt
• dba_open_error.phpt

• Pattern <functionname>_variation.phpt
• dba_open_variation.phpt

• Pattern <extname><no>.phpt
• dba_003.phpt

phpt Test Basics http://qa.php.net/write-test.php

Guide to anonymous CVS checkout http://de3.php.net/anoncvs.php

Zoes 2008 Testfest Talk http://www.phplondon.org/conference/2008/media/docs/TestOrDie_Zoe_Slattery.pdf

Sebastian Bergmanns Slides 2008 http://www.slideshare.net/sebastian_bergmann/php-testfest-cologne?src=embed

Requirements for tests on mac http://felix.phpbelgium.be/blog/2008/05/22/writing-phpt-tests-on-your-mac/