qa:phptlessonslearned
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
qa:phptlessonslearned [2008/04/18 15:52] – jmessa | qa:phptlessonslearned [2017/09/22 13:28] (current) – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | === Lessons Learned from PHPT Writing === | + | ====== Lessons Learned from PHPT Writing |
I decided to write this mainly because I won’t be writing tests as a full time job any more so am using this to pass the responsibility over to the new testers! But also with the impending TestFests and general focus on testing in the community thought it would be worthwhile to share my knowledge. Some of the points below are quite specific to the extensions that I’ve written tests for (array, ctype, dir, ereg, imap, mbstring, mysql and pcre), but hopefully will be useful for testing in general and should highlight the kind of things that should be focused on to produce a good test suite. If anyone wishes to add to, change or edit this list then please feel free, I’d be interested to see what other people have to say! | I decided to write this mainly because I won’t be writing tests as a full time job any more so am using this to pass the responsibility over to the new testers! But also with the impending TestFests and general focus on testing in the community thought it would be worthwhile to share my knowledge. Some of the points below are quite specific to the extensions that I’ve written tests for (array, ctype, dir, ereg, imap, mbstring, mysql and pcre), but hopefully will be useful for testing in general and should highlight the kind of things that should be focused on to produce a good test suite. If anyone wishes to add to, change or edit this list then please feel free, I’d be interested to see what other people have to say! | ||
Line 5: | Line 5: | ||
I'd like to thank Zoe Slattery (zoe), Steve Seear (stevseea) and Robin Fernandes (robinf) for their help with producing this. | I'd like to thank Zoe Slattery (zoe), Steve Seear (stevseea) and Robin Fernandes (robinf) for their help with producing this. | ||
- | == General: == | + | ===== General: |
* Keep basic tests basic: test a function with the minimum number of arguments, maximum number of arguments, and, if any flags or options can be set, each one of those (include any checks to ensure that they have been set). | * Keep basic tests basic: test a function with the minimum number of arguments, maximum number of arguments, and, if any flags or options can be set, each one of those (include any checks to ensure that they have been set). | ||
Line 19: | Line 19: | ||
* If the documentation says that behaviour is unpredictable, | * If the documentation says that behaviour is unpredictable, | ||
- | == Argument Types: == | + | ===== Argument Types: |
* When testing different data types for arguments that require integers, be careful of supplying resources. If no error message is returned, there is a chance that the resource has been converted to an integer which will be unpredictable, | * When testing different data types for arguments that require integers, be careful of supplying resources. If no error message is returned, there is a chance that the resource has been converted to an integer which will be unpredictable, | ||
Line 26: | Line 26: | ||
- | == Expected Output: == | + | |
+ | ===== Expected Output: | ||
//This list is quite specific to using EXPECTF, the same ideas apply to EXPECTREGEX though!// | //This list is quite specific to using EXPECTF, the same ideas apply to EXPECTREGEX though!// | ||
Line 56: | Line 57: | ||
* Watch out for error messages in the expected output: | * Watch out for error messages in the expected output: | ||
* Change the file location to %s | * Change the file location to %s | ||
- | | + | |
- | * Some errors are platform specific, if they’re being generated using '' | + | * Some errors are platform specific, if they’re being generated using '' |
- | == Specific Extensions: == | + | |
+ | ===== Specific Extensions: | ||
//These are specific to extensions I have tested, but could probably be used in other extensions too.// | //These are specific to extensions I have tested, but could probably be used in other extensions too.// | ||
Line 70: | Line 72: | ||
**Anything requiring an external server (e.g. MySQL, IMAP): ** | **Anything requiring an external server (e.g. MySQL, IMAP): ** | ||
* Use an include file where users can set their own username, password and server name as variables, then ensure that you use the variables throughout the tests. | * Use an include file where users can set their own username, password and server name as variables, then ensure that you use the variables throughout the tests. | ||
+ | * These variables should be overwritable by enviroment variable so that each tester could specify his own setup e.g.: | ||
+ | |||
+ | $host = !empty($_ENV[' | ||
+ | |||
* Whenever any of these appear in the expected output, remember to replace them with %s (if using EXPECTF). | * Whenever any of these appear in the expected output, remember to replace them with %s (if using EXPECTF). | ||
* If something doesn’t appear to be working as expected, it could be that the server you’re using doesn’t support this functionality (to save bogus bugs). | * If something doesn’t appear to be working as expected, it could be that the server you’re using doesn’t support this functionality (to save bogus bugs). | ||
* If possible, test tests on more than one server before committing. | * If possible, test tests on more than one server before committing. | ||
- | == Test Case Generator: == | + | |
+ | ===== Test Case Generator: | ||
This script was written by Zoe Slattery (thank you!) and can be found in the PHP 5_3 source code in // | This script was written by Zoe Slattery (thank you!) and can be found in the PHP 5_3 source code in // | ||
**Use on command Line:** | **Use on command Line:** | ||
+ | |||
'' | '' | ||
+ | |||
will give a good overview of how to use the test case generator, there are things that I set every time I use it so have that in a batch file. | will give a good overview of how to use the test case generator, there are things that I set every time I use it so have that in a batch file. | ||
**gen.bat file:** | **gen.bat file:** | ||
+ | |||
cd <place to store generated tests> | cd <place to store generated tests> | ||
php.exe < | php.exe < | ||
I then type the function name and the type of test I want, e.g. | I then type the function name and the type of test I want, e.g. | ||
+ | |||
'' | '' | ||
+ | |||
This will generate a basic test for array_key_exists. All I need to do now is fill out the EXPECTF section of the file! | This will generate a basic test for array_key_exists. All I need to do now is fill out the EXPECTF section of the file! | ||
** Variation tests: ** | ** Variation tests: ** | ||
+ | |||
'' | '' | ||
+ | |||
will generate variation tests. There will be one for each of the functions argument and an empty test. The numbered variations will iterate over an array of different data types and pass them as an argument, these tests are very useful! Again all that needs doing with these is changing the title (in the -- TEST -- section), add a comment explaining what the test is doing and filling out the EXPECTF section. | will generate variation tests. There will be one for each of the functions argument and an empty test. The numbered variations will iterate over an array of different data types and pass them as an argument, these tests are very useful! Again all that needs doing with these is changing the title (in the -- TEST -- section), add a comment explaining what the test is doing and filling out the EXPECTF section. | ||
With the blank variation test is a template for any other variation tests the function needs. | With the blank variation test is a template for any other variation tests the function needs. | ||
+ | |||
+ | ===== Submitting Tests ===== | ||
+ | |||
+ | If you concentrate on a certain extension you will probably write several tests for this extension. When it's done you can submit those tests to the [[https:// | ||
+ | |||
+ | make test TESTS=/ | ||
+ | |||
+ | |||
qa/phptlessonslearned.1208533955.txt.gz · Last modified: 2017/09/22 13:28 (external edit)