Requirements for run-tests.php

There are three main reasons to re-write (or re-factor) the existing run-tests.php code, these are:

• The number of core PHP test cases has increased from about 3000 at the beginning of 2007 up to about 8000 today. Running 8000 tests sequentially is taking several minutes and this is a dis-incentive to running the tests. Ultimately we will have have many thousands of test cases for PHP so the problem will get worse. The only solution is to be able to run tests in parallel, achieving this with the existing PHP4 based code is extremely difficult.
• We should have a better way to collect output from test runs than we currently have (they are mailed to the qa-reports list). A better solution would be to have a web page or set of web pages that show the last known status of PHP test pass and fails on as many platforms as possible. Doing this will be much easier if we can make run-tests.php produce XML output.
• The run-tests.php code has been around since 2000. It is in a single file and has had many additions and changes over the years. As a result the logic which started off clear and simple has become tangled and will become increasingly difficult to maintain. Therefore, ease of maintenance is the final reason for considering a re-write.

There are several features of the run-tests.php code which should be preserved: It is very easy to run from the command line. It does not require special installation - the script is just there in the PHP distribution Although the code has become tangled the central idea is simple and easy for people to understand. The format of PHP test cases *must not* change as part of any re-write. run-tests.php should not require PHP to be built with any non-core extension with the exception of pcntl. We have to make this one exception because it is the only way to address the problem of running tests in parallel.

Any rewrite *must* be able to run all the existing tests as they are without modification. Since we know that tests within the same directory sometimes depend on each other, for instance they may modify the same file), parallelism can currently only be achieved at the level of test directories, not at individual tests.

Any new code must be easy to maintain and easy to extend. This means that we have to be clear about and agree on things like formatting, naming conventions and so on.

• Be able to skip whole directories with one check
• Provide the same functionality as REDEIRECTTEST, not necessarily using the current design
• Provide an API so thatthe code can be used by other applications (from Greg Beaver).
• An option to specify arguments to valgrind