qa:phptlessonslearned

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
qa:phptlessonslearned [2008/04/18 15:52] jmessaqa:phptlessonslearned [2017/09/22 13:28] (current) – external edit 127.0.0.1
Line 1: Line 1:
-=== Lessons Learned from PHPT Writing ===+====== Lessons Learned from PHPT Writing ======
  
 I decided to write this mainly because I won’t be writing tests as a full time job any more so am using this to pass the responsibility over to the new testers! But also with the impending TestFests and general focus on testing in the community thought it would be worthwhile to share my knowledge. Some of the points below are quite specific to the extensions that I’ve written tests for (array, ctype, dir, ereg, imap, mbstring, mysql and pcre), but hopefully will be useful for testing in general and should highlight the kind of things that should be focused on to produce a good test suite. If anyone wishes to add to, change or edit this list then please feel free, I’d be interested to see what other people have to say! I decided to write this mainly because I won’t be writing tests as a full time job any more so am using this to pass the responsibility over to the new testers! But also with the impending TestFests and general focus on testing in the community thought it would be worthwhile to share my knowledge. Some of the points below are quite specific to the extensions that I’ve written tests for (array, ctype, dir, ereg, imap, mbstring, mysql and pcre), but hopefully will be useful for testing in general and should highlight the kind of things that should be focused on to produce a good test suite. If anyone wishes to add to, change or edit this list then please feel free, I’d be interested to see what other people have to say!
Line 5: Line 5:
 I'd like to thank Zoe Slattery (zoe), Steve Seear (stevseea) and Robin Fernandes (robinf) for their help with producing this. I'd like to thank Zoe Slattery (zoe), Steve Seear (stevseea) and Robin Fernandes (robinf) for their help with producing this.
  
-== General: ==+===== General: =====
  
   * Keep basic tests basic: test a function with the minimum number of arguments, maximum number of arguments, and, if any flags or options can be set, each one of those (include any checks to ensure that they have been set).   * Keep basic tests basic: test a function with the minimum number of arguments, maximum number of arguments, and, if any flags or options can be set, each one of those (include any checks to ensure that they have been set).
Line 19: Line 19:
   * If the documentation says that behaviour is unpredictable, don’t write a test for that scenario.   * If the documentation says that behaviour is unpredictable, don’t write a test for that scenario.
  
-== Argument Types: ==+===== Argument Types: =====
  
   * When testing different data types for arguments that require integers, be careful of supplying resources. If no error message is returned, there is a chance that the resource has been converted to an integer which will be unpredictable, thus making the expected output impossible to create.   * When testing different data types for arguments that require integers, be careful of supplying resources. If no error message is returned, there is a chance that the resource has been converted to an integer which will be unpredictable, thus making the expected output impossible to create.
Line 26: Line 26:
  
  
-== Expected Output: ==+ 
 +===== Expected Output: =====
  
 //This list is quite specific to using EXPECTF, the same ideas apply to EXPECTREGEX though!// //This list is quite specific to using EXPECTF, the same ideas apply to EXPECTREGEX though!//
Line 56: Line 57:
   * Watch out for error messages in the expected output:   * Watch out for error messages in the expected output:
     * Change the file location to %s     * Change the file location to %s
-  * Line numbers should be changed to %d, though this can mean bugs don’t get noticed (for example all errors saying “on line 0”). If leaving the line number in, ensure it’s the right one! +    * Line numbers should be changed to %d, though this can mean bugs don’t get noticed (for example all errors saying “on line 0”). If leaving the line number in, ensure it’s the right one! 
-  * Some errors are platform specific, if they’re being generated using ''strerror(errno)'' in the source code then change the error to %s.+    * Some errors are platform specific, if they’re being generated using ''strerror(errno)'' in the source code then change the error to %s.
  
-== Specific Extensions: ==+ 
 +===== Specific Extensions: =====
  
 //These are specific to extensions I have tested, but could probably be used in other extensions too.// //These are specific to extensions I have tested, but could probably be used in other extensions too.//
Line 70: Line 72:
 **Anything requiring an external server (e.g. MySQL, IMAP): ** **Anything requiring an external server (e.g. MySQL, IMAP): **
   * Use an include file where users can set their own username, password and server name as variables, then ensure that you use the variables throughout the tests.   * Use an include file where users can set their own username, password and server name as variables, then ensure that you use the variables throughout the tests.
 +  * These variables should be overwritable by enviroment variable so that each tester could specify his own setup e.g.:
 +
 +  $host = !empty($_ENV['TEST_HOST']) ? $_ENV['TEST_HOST'] : 'localhost';
 +
   * Whenever any of these appear in the expected output, remember to replace them with %s (if using EXPECTF).   * Whenever any of these appear in the expected output, remember to replace them with %s (if using EXPECTF).
   * If something doesn’t appear to be working as expected, it could be that the server you’re using doesn’t support this functionality (to save bogus bugs).   * If something doesn’t appear to be working as expected, it could be that the server you’re using doesn’t support this functionality (to save bogus bugs).
   * If possible, test tests on more than one server before committing.   * If possible, test tests on more than one server before committing.
  
-== Test Case Generator: ==+ 
 +===== Test Case Generator: =====
  
 This script was written by Zoe Slattery (thank you!) and can be found in the PHP 5_3 source code in //scripts/dev/generate_phpt.php//. Although it’s in the 5_3 source code, I think I’m right in saying that it can be used to create tests for **all** versions of PHP. I use it for all the tests I have written and have found it to be incredibly useful! The basic format of it is that you pass a function name and a type of test (either basic, error or variation) and it will generate a template for the test. Below is how I have it set up on my machine (windows), and how I use it. This script was written by Zoe Slattery (thank you!) and can be found in the PHP 5_3 source code in //scripts/dev/generate_phpt.php//. Although it’s in the 5_3 source code, I think I’m right in saying that it can be used to create tests for **all** versions of PHP. I use it for all the tests I have written and have found it to be incredibly useful! The basic format of it is that you pass a function name and a type of test (either basic, error or variation) and it will generate a template for the test. Below is how I have it set up on my machine (windows), and how I use it.
  
 **Use on command Line:** **Use on command Line:**
 +
 ''php.exe generate_phpt.php –h'' ''php.exe generate_phpt.php –h''
 +
 will give a good overview of how to use the test case generator, there are things that I set every time I use it so have that in a batch file. will give a good overview of how to use the test case generator, there are things that I set every time I use it so have that in a batch file.
  
 **gen.bat file:** **gen.bat file:**
 +
   cd <place to store generated tests>   cd <place to store generated tests>
   php.exe <location of generate_phpt.php script> -s <location of source code> -f %*   php.exe <location of generate_phpt.php script> -s <location of source code> -f %*
  
 I then type the function name and the type of test I want, e.g. I then type the function name and the type of test I want, e.g.
 +
 ''  gen.bat array_key_exists –b '' ''  gen.bat array_key_exists –b ''
 +
 This will generate a basic test for array_key_exists. All I need to do now is fill out the EXPECTF section of the file! This will generate a basic test for array_key_exists. All I need to do now is fill out the EXPECTF section of the file!
  
 ** Variation tests: ** ** Variation tests: **
 +
 ''  gen.bat array_key_exists –v'' ''  gen.bat array_key_exists –v''
 +
 will generate variation tests. There will be one for each of the functions argument and an empty test. The numbered variations will iterate over an array of different data types and pass them as an argument, these tests are very useful! Again all that needs doing with these is changing the title (in the -- TEST -- section), add a comment explaining what the test is doing and filling out the EXPECTF section. will generate variation tests. There will be one for each of the functions argument and an empty test. The numbered variations will iterate over an array of different data types and pass them as an argument, these tests are very useful! Again all that needs doing with these is changing the title (in the -- TEST -- section), add a comment explaining what the test is doing and filling out the EXPECTF section.
 With the blank variation test is a template for any other variation tests the function needs. With the blank variation test is a template for any other variation tests the function needs.
 +
 +===== Submitting Tests =====
 +
 +If you concentrate on a certain extension you will probably write several tests for this extension. When it's done you can submit those tests to the  [[https://phptestfest.org/|TestFest website]], but be sure, that every test is working fine. You can do so by running:
 +
 +  make test TESTS=/path/to/php5/ext/yourextension/tests
 +
 +
  
qa/phptlessonslearned.1208533955.txt.gz · Last modified: 2017/09/22 13:28 (external edit)