Lessons Learned from PHPT Writing

I decided to write this mainly because I won’t be writing tests as a full time job any more so am using this to pass the responsibility over to the new testers! But also with the impending TestFests and general focus on testing in the community thought it would be worthwhile to share my knowledge. Some of the points below are quite specific to the extensions that I’ve written tests for (array, ctype, dir, ereg, imap, mbstring, mysql and pcre), but hopefully will be useful for testing in general and should highlight the kind of things that should be focused on to produce a good test suite. If anyone wishes to add to, change or edit this list then please feel free, I’d be interested to see what other people have to say!

I'd like to thank Zoe Slattery (zoe), Steve Seear (stevseea) and Robin Fernandes (robinf) for their help with producing this.

• Keep basic tests basic: test a function with the minimum number of arguments, maximum number of arguments, and, if any flags or options can be set, each one of those (include any checks to ensure that they have been set).
• Include files, although annoying for debugging reasons, save massive amounts of code reuse and keep tests clean and consistent.
  • If using a function from an include file, write a comment explaining what the function is doing to save a reader’s time.
  • Use include files for variables that will change, for example passwords and names of databases and email servers. If using these, remember to generalise expected output.
• Read the documentation, write tests that reflect what the documentation says the function should do.
  • For example, in the directory functions the resource parameter in some functions is explained to be “opened using opendir”, try passing a resource opened with fopen(), this according to the documentation shouldn’t work (alas, it did).
  • Use titles (in the --TEST-- section) that explain what the test is actually doing! These should be one line only
• Tests should be intuitive, if something very complicated is happening separate a test out.
  • If being non-intuitive is unavoidable, COMMENTS ARE USEFUL.
• Lay tests out in an easy to read fashion, remember that other people might actually read them at some point.
• If the documentation says that behaviour is unpredictable, don’t write a test for that scenario.

• When testing different data types for arguments that require integers, be careful of supplying resources. If no error message is returned, there is a chance that the resource has been converted to an integer which will be unpredictable, thus making the expected output impossible to create.
• When testing arguments that relate to the length or offset of something, pass negative numbers, as well as numbers less and greater than the length.
• If a function takes a resource as an argument, test different types of resources (file system, directory, database connection etc.) in case there is no type check.

This list is quite specific to using EXPECTF, the same ideas apply to EXPECTREGEX though!

• Run a test multiple times to check if the expected output needs to be generalised to accommodate for variable output.
• Don’t put comments in the test explaining what the expected output should be, that’s what the expected output section is for .
• Use echo statements to separate parts of the expected output (e.g. a counter if using iterations etc.)
• Remember that the expected output has to reflect what will come out of a test if everything is working. For example a test with a statement conditionally echoed out that has not been included in the expected output saying that something is not available, with a comment next to it saying “don’t fail if this happens” is nonsensical, the test will fail if this happens as the string is not in the expected output.
• If var_dumping a string where some of the string is generalised (i.e. replaced with %d/%s etc), check the string length is not also variable.

Example:

string(%d) "The next bit might vary in length: %s"

• Change numbers generated in resources and objects to %d.

Example:

object(C)#%d (1) {
   ["p"]=>
   bool(true)
}

resource(%d) of type (stream)

casting a resource to a string: you don’t know how many digits the number will be, so character count has to be %d too!

string(%d) "Resource id #%d"

• Try to write the expected output yourself, instead of copying and pasting from a console. This ensures that you don’t miss any odd/unexpected behaviour.
• Watch out for line endings on different platforms and how that effects the character length of a string
• Avoid using %a unless it really is unavoidable. I mean REALLY unavoidable.
• Watch out for error messages in the expected output:
  • Change the file location to %s
  • Line numbers should be changed to %d, though this can mean bugs don’t get noticed (for example all errors saying “on line 0”). If leaving the line number in, ensure it’s the right one!
  • Some errors are platform specific, if they’re being generated using strerror(errno) in the source code then change the error to %s.

These are specific to extensions I have tested, but could probably be used in other extensions too.

MBString:

• ‘unexpected’ behaviour can come from not remembering byte length vs character length in strings.
  • e.g. in mb_ereg the byte length of a matched part of a string is returned instead of the character length, so for example when 15 multibyte characters are matched, mb_ereg returned int(35). (see mb_ereg_basic.phpt).
• base64 encode any multibyte strings that appear in the expected output.

Anything requiring an external server (e.g. MySQL, IMAP):

• Use an include file where users can set their own username, password and server name as variables, then ensure that you use the variables throughout the tests.
• These variables should be overwritable by enviroment variable so that each tester could specify his own setup e.g.:

$host = !empty($_ENV['TEST_HOST']) ? $_ENV['TEST_HOST'] : 'localhost';

• Whenever any of these appear in the expected output, remember to replace them with %s (if using EXPECTF).
• If something doesn’t appear to be working as expected, it could be that the server you’re using doesn’t support this functionality (to save bogus bugs).
• If possible, test tests on more than one server before committing.

This script was written by Zoe Slattery (thank you!) and can be found in the PHP 5_3 source code in scripts/dev/generate_phpt.php. Although it’s in the 5_3 source code, I think I’m right in saying that it can be used to create tests for all versions of PHP. I use it for all the tests I have written and have found it to be incredibly useful! The basic format of it is that you pass a function name and a type of test (either basic, error or variation) and it will generate a template for the test. Below is how I have it set up on my machine (windows), and how I use it.

Use on command Line:

php.exe generate_phpt.php –h

will give a good overview of how to use the test case generator, there are things that I set every time I use it so have that in a batch file.

gen.bat file:

cd <place to store generated tests>
php.exe <location of generate_phpt.php script> -s <location of source code> -f %*

I then type the function name and the type of test I want, e.g.

gen.bat array_key_exists –b

This will generate a basic test for array_key_exists. All I need to do now is fill out the EXPECTF section of the file!

Variation tests:

gen.bat array_key_exists –v

will generate variation tests. There will be one for each of the functions argument and an empty test. The numbered variations will iterate over an array of different data types and pass them as an argument, these tests are very useful! Again all that needs doing with these is changing the title (in the -- TEST -- section), add a comment explaining what the test is doing and filling out the EXPECTF section. With the blank variation test is a template for any other variation tests the function needs.

If you concentrate on a certain extension you will probably write several tests for this extension. When it's done you can submit those tests to the TestFest website, but be sure, that every test is working fine. You can do so by running:

make test TESTS=/path/to/php5/ext/yourextension/tests