qa:testfesttalk

This is an old revision of the document!


PHPT roundup talk

This wikipage contains all extra information for the talk prepared for phptestfest 2009. They heavily rely on content written by other people before. Every resource of the talk, even the slides (in different formats) will be in this place. Other testfesters can use and modify this talk to save precious time.

http://tinyurl.com/c9bqf4

How is PHP Tested

What is a PHPT

A phpt test is a little script used by the php internal and quality assurance teams to test PHP's functionality. It can be used with new releases to make sure they can do all the things that previous releases can, or to help find bugs in current releases. By writing phpt tests you are helping to make PHP more stable.

Requirements

All that is really needed to write a phpt test is

  • a basic understanding of the PHP language
  • a text editor
  • a way to execute the code

Required Packages (debian)

  • the make commando available
  • autoconf
  • gcc
  • bison or flex
  • libxml2-dev
  • re2c
  • lcov

Get the source

First things first. For all the testfest fun you need to checkout all the sourcecode for PP 5.3

CVS Configuration

For easy handling of the PHP CVS repo. you can setup some variables in you ./~cvsrc file

cvs -z3
update -d -P
checkout -P
diff -u

Login to CVS

cvs -d :pserver:cvsread@cvs.php.net:/repository login
  • Username: cvsread
  • password: phpfi

Checkout

Testfest 2009 is all about the PHP 5.3 release. So checkout the right part of the source tree

cvs -d :pserver:cvsread@cvs.php.net:/repository checkout -r PHP_5_3 php5

Now you have all the sourcecode you neeed to start testing in a folder php5.

Compile and run the tests

Running phpt tests is part of the build process.

Start by preparing the configuration of your build by executing the buildconf

./buildconf

Everything is now set up to write PHPT tests or just explore and hack the PHP source code. If you want to configure and compile PHP, use the following commands:

./configure && make && make test

If you also want to enable code coverage results, use the following command:

./configure --enable-gcov && make && make lcov

The build will run for a while, strongly depending on the performance of your computer.

While files are compiled the output will look like this:

/bin/sh /home/sschuermann/php/php5/libtool --silent --preserve-dup-deps --mode=compile cc  -Isapi/cgi/ -I/home/sschuermann/php/php5/sapi/cgi/ .....

As soon as the main test process starts output will look like this:

PASS Test array_combine() function : usage variations - different arrays(Bug#43424) [ext/standard/tests/array/array_combine_variation3.phpt] 

When there are one or more failing tests a special output is presented to you.

=====================================================================
EXPECTED FAILED TEST SUMMARY
---------------------------------------------------------------------
ob_start(): Ensure unerasable buffer cannot be flushed by ob_flush(). [tests/output/ob_start_basic_unerasable_005.phpt]
Inconsistencies when accessing protected members [Zend/tests/access_modifiers_008.phpt]
Inconsistencies when accessing protected members - 2 [Zend/tests/access_modifiers_009.phpt]
Bug #42718 (unsafe_raw filter not applied when configured as default filter) [ext/filter/tests/bug42718.phpt]
SimpleXML: array casting bug [ext/simplexml/tests/034.phpt]
=====================================================================

You may have found a problem in PHP.
We would like to send this report automatically to the
PHP QA team, to give us a better understanding of how
the test cases are doing. If you don't want to send it
immediately, you can choose "s" to save the report to
a file that you can send us later.
Do you want to send this report now? [Yns]: 

Execute tests

There is a set of things that need to be prepared before you can actually start execute the tests.

Which "php" executable "make test" look for

You must use TEST_PHP_EXECUTABLE environment variable to explicitly select the php executable to be used to run the tests. That can either be the CLI or CGI executable. Command “make test” executes “run-tests.php” script with “php” binary. Some test scripts such as session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all tests.

TEST_PHP_EXECUTABLE=sapi/cli/php

and

export TEST_PHP_EXECUTABLE

All Tests

You can re-run the same Tests as executed in make test when you execute the file run-tests.php

php run-tests.php

Single Tests

It is possible to execute a single test or tests residing in one directory. Again run-tests.php helps here.

A single Test

php run-tests.php tests/func/test010.phpt

A directory with tests

php run-tests.php tests/func/

Write Tests

From here on it's easy to add own tests.

--TEST--
Hello world test
--FILE--
<?php
echo "Hello";
?>
--CLEAN--
--EXPECT--
Hello

Write good Tests...

There are certain guidelines you can follow when writing tests. There are several guidelines that you can follow to get working test code that is solid and makes sense. Several basics apply in general

  • KISS Keep the Test short in general.
  • Test one functionality at a time
  • Try to make execute the test fast, waiting for the next eclipse to proof the date() function fails will slow down build times
  • Document the outcome. People using your tests will not always be experts for that specific tests
  • Place Links to resources as they help understand the test. A test is as much about documentation as on prooving fail or working condition of a function
    • Extension docs
    • Bug reports
  • If you are testing on a bug, make sure you got the bottom line of it

Mystery Guests

When a test uses external resources, such as a file containing test data, the test is no longer self contained. Consequently, there is not enough information to understand the tested functionality, making it hard to use that test as documentation. Moreover, using external resources introduces hidden de- pendencies: if some force changes or deletes such a re- source, tests start failing. Chances for this increase when more tests use the same resource. The use of external re- sources can be eliminated using the refactoring Inline Re- source. If external resources are needed, you can apply Setup External Resource to remove hidden dependen- cies.

Examples:

  • storing data in data files

Avoid by

  • Store Data in the Test
  • Write Setup Code

Resource Optimism

Test code that makes optimistic assumptions about the ex- istence (or absence) and state of external resources (such as particular directories or database tables) can cause non- deterministic behavior in test outcomes. The situation where tests run fine at one time and fail miserably the other time is not a situation you want to find yourself in. Use Setup External Resource to allocate and/or initialize all resources that are used.

Examples:

  • Files in /tmp (there is no /tmp in windows)
  • extensions installed
  • Special variants/setups of extensions installed

Avoid By

  • Write Setup Code for the resource
  • Do not re-use resources through multiple tests

Test Run Wars

Such wars arise when the tests run fine as long as you are the only one testing but fail when more programmers run them. This is most likely caused by resource interference: some tests in your suite allocate resources such as tempo- rary files that are also used by others. Apply Make Re- source Unique (3) to overcome interference.

Examples:

  • Creating files in a public directory (/tmp/ if choosen right ^^) under a fixed file name

Avoid by:

  • Adding Random or user based Value to file name

Eager Test

When a test method checks several methods of the object to be tested, it is hard to read and understand, and therefore more difficult to use as documentation. Moreover, it makes tests more dependent on each other and harder to maintain. The solution is simple: separate the test code into test methods that test only one method using Fowler’s Extract Method (F:110), using a meaningful name highlighting the purpose of the test. Note that splitting into smaller methods can slow down the tests due to increased setup/teardown overhead.

Examples:

  • Tests of different behaviour in one test
  • More than one method is tested

Avoid

  • Split into several smaller tests

... for Coverage

... to find bugs

Naming conventions for Tests ...

Phpt tests follow a very strict naming convention. This is done to easily identify what each phpt test is for.

... on bugs

The tests for Bugs contain the id of the bug according to the bug id in the bugtracker

  • Pattern bug<bugid>.phpt
  • bug17123.phpt

... basic behaviour

Basic funcion behaviour is a simple call to the function checking on the normal behaviour of it.

  • Pattern <functionname>_basic.phpt
  • dba_open_basic.phpt

... error behaviour

This kind of test covers the fnctions behaviour in case on an error. Do not mix this up with a bug.

  • Pattern <functionname>_error.phpt
  • dba_open_error.phpt

... variations in function behaviour

  • Pattern <functionname>_variation.phpt
  • dba_open_variation.phpt

... on extensions

  • Pattern <extname><no>.phpt
  • dba_003.phpt

Resources

qa/testfesttalk.1240001762.txt.gz · Last modified: 2017/09/22 13:28 (external edit)