qa:runtests:documentation

Developer documentation

A list of the options available in the old version of run-tests and information about whether that are (or will be) re-implemented.

Introduction

runtests is a test environment written in PHP. It is used to test PHP itself. It is not a replacement for unit test frameworks like PHPUnit.

The basic principle of automated tests is very simple. Compare a computed result with some pre-calculated “known good”, expected value. If both match, the test has passed. If they do not match, the tests fails, and there is probably a bug in PHP, or some precondition for the test was not met.

Testing PHP is a far more complex issue than it may seem at first glance, because each test may require an individual PHP configuration, certain settings at operating system level (for example environment variables), or even external services (like databases, LDAP servers or IMAP servers).

A test that requires external services can be complex to set up, but there is no way around that. How could you make sure that running a SQL statement against a database works without actually running it against a database?

Since a failing test will probably leave its environment in an unpredictable state, a high level of test isolation is required. In other words, this means that we need to set up a new preconfigured PHP process for each test. This is the only way to ensure that every test runs in a clean environment.

So, to execute each test, runtests must set up a PHP process, have it run the test, collect the output, and compare it to the pre-calculated “known good” expected output. That spawned PHP process (hopefully) terminates, so at the process level, we do not have to worry about the potential mess that a test has left behind. Still, if the test has created files, databases, or modified the global system environment, additional work may be required to clean up after the test.

Since PHP runs on most of the available platforms, runtests must also run on all these platforms. From an implementation point of view, this means that runtests must work on a minimal, stock PHP installation, and should make as few as possible assumptions on the system environment. All the more or less known cross-platform issues like different line endings, directory separators, case handling, as well as limits of different operating system families (file name path length, include path length) have to be taken into account and being dealt with in runtests.

Since runtests spawns off a separate PHP process to run a test, it is not necessary to run each test in the same PHP version that runtests runs on. In other words, that means that you can test a different PHP version than you are actually running - and in fact, it may be a good idea to use a “known good” PHP version to test a new, less tested PHP version. After all, bugs in the PHP version runtests itself is running on might affect runtests itself, and thus make the test results less reliable.

We have already mentioned that tests need an individual environment. Not all tests can be run at the command line, for example. If a test needs to make sure that PHP returns correct HTTP header, or processes GET or POST input, the test probably requires the CGI SAPI to run.

The implementation of runtests that is described in these pages is specific to PHP5.3 and beyond. Many of the implementation details are similar to the previous version of runtests but the overall code structure is very different.

Overall code structure

The most important classes in classes in the code are shown in the figure below and described in the following paragraphs.

Test Run

The main class is rtPhpTestRun. This class is responsible for the overall running of one or more test cases.

rtPhpTestRun instantiates the run configuration class (rtRuntestsConfiguration, see Configuration) and then executes a single test or a group of tests.

Test Group

A test group (rtPhpTestGroup) is currently defined as all of the tests within a single directory. It seems likely that groups of tests can be run in parallel, teh prototype code is designed to enable that.

There is currently no group configuration class, however, it is possible that one may be required. For example there might be groups of tests that cannot be run at the same time as other groups (tests with REDIRECT?).

Test Case

Each test case is executed in it's own process. Many classes are associated with the execution of a single test case. The following subsections give a brief description of the responsibilities of each class.

Run configuration

This group of classes is responsible for setting the configuration for the whole test run. So, for example the name of the name of the PHP executable under test is set in this group as is the name of the PHP CGI exectuable.

Test case configuration

Some settings are specific to each test case and need to be set for a single test. These classes are responsible for settings at the level of individual tests.

Run configuration classes

This sections contains a brief description of each class involved in setting the configuration for the whole test run.

rtRunTestsConfiguration

The main class in this section - all of the other classes are related to setting options in the run configuration object. There should be OS dependent instances for unix, windows etc.

rtCommandLineOptions

Instantiated by rtRuntestsConfiguration. Parses the command line options (argv[]) for runt-tests.php

rtEnvironmentVariables

Get a list of all of the Environment variable and maintains as an array. This is required both establishing configuration settings and to pass to to proc_open() when PHP code is run as part of a test (see rtPhpRunner and testcase/sections/executablesections).

rtAddToCommandLine

Somewhat bizarrely (to my mind) argv[] for run-tests can be added to using an Environment variable (TEST_PHP_ARGS). This class is instantiated after rtEnviromentVariables and may (if TEST_PHP_ARGS is used) add other options to an instance of rtCommandLineOptions. This is here for compatibility with the old version of run-tests, if no good reason for it comes to light it should go.

rtIniAsCommandLineArgs

The run-tests code pre-sets a number of ini settings automatically, for example error_reporting. This helps prevent users accidentally picking up different php.ini files which would cause tests to fail. This class maintains a list of ini settings and converts them to -d flags used in teh PHP command line when a test case is run, eg

php_exectuable -d flag1 -d flag2… test.php test_arguments

Note that these can be added to using the test case section INI. This class is instantiated from the setting class that builds the php command line, it is also instantiated from the rtTestConfiguration class when a test case contains an INI section. rtPreCondition list Maintains a list of all of the pre-conditions that must be satisfied before a test run is attempted. For example - check that a PHP executable has been specified.

rtPrecondition

Parent class for the classes that contain code to check for each pre-condition. See subdirectory 'precoditions' for the sub-classes that perform that checks.

rtSetting

Parent class for classes that set run configuration settings. See 'setting' directory for sub-classes. Note that run settings can be derived from eith command line options or Environment variables, the functions of teh settings classes is mainly to determine which of these has been used and to set the run configuration accordingly.

Test case classes

rtPhpTestFile

Reads the contents of a test file, checks that it's a valid test (see precondtions) and removes windows-style line endings. rtPhpTest The main test case class. Takes the contents of the file read by rtPhpTestFile and creates section objects for each section (see Test Case Sections). The run() method runs any executable sections and then compares teh test out put with the expected output.

rtTestPreCondtion

There are a number of preconditions that must be met before a test can be run. This is the parent class for preconditions. Preconditions are checked as the test file is read (see section on test precondtions)

rtTestConfiguration

The run configuration can be modified by test case sections. The test configuration class implements the modifications and is used by rtFileSection when running the test.

rtPhpRunner

As the name suggests, this runs PHP code. It's an almost exact copy of the original run-tests code (system_with_timeout()). No reason to change something that works fine as it is.

rtPhpTestResults

Collates the results of running (or attempting to run) a test. If the test has failed ensures that the difference between the expected and actual output is calculated and saved. Decides which files to keep and which to delete. Keeps a record of any saved files and of teh overall status (pass, fail, skip etc) of the test.

rtTestDifference

Exact copy of the differencing code in the original run-tests. This code is complex, it might be possible to do better but for now it seems best not to try and modify it.

rtOutputWriter

Takes the list of test results and writes in the desired format. Is a parent class for various different formats out output class. Currently only the 'list' format is implemented.

Test Section Classes

Each test is made up from a number of sections, for example, –TEST– a title –FILE– <?php

some php code

?> –EXPECT– the expected result of running the PHP code.

All sections have a name and some contents. Sometimes the contents just have information - for example the test title, sometimes the contents are executable, or used to compare against the output.

The base class for all sections is rtSection. This class has a method which initialises the contents to an array of strings.

There are four subclasses which extend rtSection, these are rtInformationSection, rtExecutableSection, rtoutputSection and rtConfigurationSection. Each of these are extended again by classes which represent specific sections. The class hierarchy is shown in the attached ODP charts.

A brief description of the responsibilities of each class is given below.

Information Sections

rtTestSection

This class just contains the test case title.

rtCreditsSection

This class contains information about who wrote the test

rtXfailSection

The presence of this section in the test indicates that the test is expected to fail. The contents contain information about the reason for an expected failure.

Executable Sections

rtFileSection

This is the most complex of the sections. The contents represent the PHP code that needs to be run. The section is responsible for assembling the command line that will run the code. The format of the command line is:

php_executable php_arguments test.php test_arguments

Note that both php_arguments aand test_arguments can be modified by other test sections and are therefore supplied from a TestConfiguration object and not from the RunConfiguration. The executable code is supplied in a file called test.php, where 'test' is the name of the testcase. The file is constructed fron teh test case contents and written by this section.

rtSkipIfSection

Executes code to determine if the test should be run or not. For example: <?php if( substr(PHP_OS, 0, 3) == 'WIN') {

die('skip Not for Windows'); 

Note that the first word in the comment is always 'skip' or 'warn'. For example in ext/standard/tests/time/001.phpt:

<?php

      if (!function_exists('microtime'))  
			die('skip microtime() not available'); 
      die('warn system dependent'); 

?> Note: the warning is a pretty strange thing to do as part of a section called SKIPIF. I've implemented it here for compatibility but it would be better to have a separate WARN section

rtCleanSection

This section runs code that should clean up after a test. Many people forget that this is a separate piece of executable code - therefore you can't define $myFile in the body of the test and expect to be able to unling $myFile in the CLEAN section. There are currently 71 tests that need to be fixed because the CLEAN sections are a mess.

The old run-tests code just chucks away any results from running a CLAEN section. In this code if the results of running the clean section are not an empty string then the results are printed out with a WARN flag.

OutputSections

These sections deal with the expected output of the test and compare it with the actual output. rtExpectSection

rtExpectFSection

rtExectRegexSection

Configuration sections

These sections have additional configuration information that changes the way that a single test is run

rtArgsSection

This section contains command line arguments for the test case. For an example of the usage see ext/standard/tests/general_functions/getopt.phpt

Test test_arguments are appended after the testcase as a string preceeded by '–'

php_executable php_arguments test.php test_arguments

rtEnvSection

This section adds any environmental variables that are required by a single testcase. For an example of usage see: ext/standard/tests/general_functions/parse_ini_basic.phpt The environmental variables specified in the test case are appended to the array of environment variables which is constructed in the run configuration and set in teh testsConfiguration

rtIniSection

This section adds additional commanline arguments to the PHP command line: php_executable php_arguments test.php test_arguments

These are added in the form of -d flags and are appended to those already set in the runConfiguration

Parallel Execution

Introduction

The parallel execution of the test cases is one of the main features of the new phpruntests. Compared to the previous run-tests script it allows to run the tests cases in about half the time. Here are a few data about the performance:

Before digging into the details there are a few issues to know:

Preconditions

Of course we need a few preconditions to perform the parallel test-execution:

  • *nix only – Currently we are not supporting windows platforms, because windows does not provide the same native possibilities for multiprocessing as unix does (mainly the fork routine). Therefore, our first goal was to focus on our main target audience and provide the parallel executions for unix only. The windows implementation is planned, however, as a subsequent step and will become a default feature of later phpruntests.

If these preconditions are not fulfilled, the tests are automatically executed sequentially.

Test-Groups

The test cases are stored across the whole php-folder, e.g.:

  • ext/xml/tests/xml_set_external_entity_ref_handler_error.phpt
  • Zend/tests/double_to_string.phpt
  • tests/lang/script_tag.phpt

A major issue was how to deal with dependencies between the single test-cases. For example, the MySQL tests: One test creates a database table which is also used by a couple of other test cases. So we had to ensure to keep the order of the test cases while distributing them to different processes. For that reason we designed the test-group which are representing a directory that contains phpt-testfiles. With these groups, we are able to map the structure of the test directories to our program in order to distribute only complete test groups to the different processes, not just the single test-cases. The test-cases themselves are executed in alphabetical order, exactly as they are stored in the directory.

With this design, we are able to avoid test dependencies in a very simple, but also efficient manner. Of course that also means that there must not be any directory-comprehensive dependencies. In other words, if you are writing depending test cases, make sure that they are stored in the same directory, and that the alphabetical order meets your requirements.

TaskScheduler

This is place were the magic happens, this package handles the parallel execution in phpruntests. The required source files are stored in

http://svn.php.net/viewvc/php/phpruntests/trunk/src/taskScheduler/

First of all you have to know that the taskScheduler was implemented from a generic point of view with the goal to execute any various task an distribute it to a various number of processes. Here is the basic concept: A task, however it looks like, has a procedure which is executed and delivers a result afterwards. A list of such tasks is passed to the taskScheduler which executes them and provides the results.

You might be interested in what else is possible with the taskScheduler. If so, please have a look a the prototype code, including some examples:

http://svn.php.net/viewvc/php/phpruntests/trunk/code-examples/taskScheduler/


rtTask

In our case, this task procedure is implemented in rtTaskTestGroup. which represents a kind of a nutshell of a test-group. This class also extends the superclass “rtTask” and implements the “rtTaskInterface” to ensure that the required methods are available.

rtTaskScheduler

This is the main class of the scheduler-package which is responsible for handling the processes and the communication between them. Like in other phpuntests classes this class provides a factory-method (rtTaskScheduler::getInstance) for instancing the right subclass. This method implements a check if PCNTL is available. If so, an instance of rtTaskSchedulerFile is returned. Otherwise the basic is returned which provides sequential execution only.

The simple, sequential variation is also implemented in this superclass (rtTaskScheduler::run) and does nothing more than passing through the task-list in a simple loop, executing the test-groups and storing the results.

  • taskList - array - a list of task-objects (extending rtTask, implementing rtTaskInterface)
  • resultList - array - the result list in the same order as the taskList
  • processCount - int - the number of processes (default=0). If the value is 0, the tasks are automatically executed sequentially.
  • reportStatus - int - defines the detail-level of the output during the test execution (see verbose-mode)

rtTaskSchedulerFile

This class implements the common variation of parallel execution. The suffix “File” describes the way the IPC is handled. In this case that means the the processes are communicate only via temporary files which are stored on the disk.

The run-method overrides the superclass and forks the defined number of child processes and assigns a unique ID to them. The tasks (containing a group of test- ases) are distributed sequentially to those child-processes via a task-file. e.g. 5 tasks, 3 processes:

task A → process 1

task B → process 2

task C → process 3

task D → process 1

task E → process 2

In detail, the whole task object is serialized and written to a temporary file. The filename is suffixed by the unique ID of the corresponding child process (e.g.: “TaskFile3”). The child reads the whole task file, splits off the single tasks and executes them. Afterwards it gets the result, which is an array of rtTestResult-Objects, serializes it and writes it back to its task-file. After executing all corresponding tasks, the child-process is terminating itself. In the meantime, the parent process is waiting until all children are finished and starts the receiver afterwards. The receiver is collecting the results from the task files and storing them to the result list in exactly the same order as the task list.

A brief look at the code shows that the tasks as well as the results are written to the file immediately (using FILE_APPEND) instead of collecting them first and write it only once. This maybe looks like a performance-leak, but it's absolutely necessary to avoid out-of-memory errors. The string of the collected serialized objects would become too large from a certain number of tasks.

As you can see, there are also two signal-handlers (SIGQUIT, SIGINT) registered. The associated method does only terminate the process with exit(0), but this is important to remove the temporary files in a case of interruption.

rtTaskSchedulerMsgQ

As above, the suffix “MsgQ” describes the implemented IPC-variation. All in all, this class does the same job as “rtTaskSchedulerFile” but uses message-queues instead of files to distribute the tasks and collect the results.

The run-method forks a sender, a receiver and the defined number of child processes which are responsible for executing the tasks. The sender passes through the task-list and writes the tasks to the the sender queue. The child-ID is set as the message-type, so every child process reads only its corresponding tasks and executes them. Afterwards, the child writes the results to the receiver-queue from where the receiver-process is collecting them.

The sender terminates if all tasks are distributed. When the receiver has collected all results, it sends a kill-signal to all child-processes via the input-queue and terminates itself. The parent-process is waiting for all children to finish and exits afterwards.

File or MsgQ?

Both implementations provide exactly the same functionality. The message-queue is maybe the more elegant one, but in fact it's another barrier on the way to the aimed platform-independence.

We choose the file variation as the common because it's a more flexible strategy when we start to think about platform-independence. Of course the usage of PCNTL makes it impossible to perform parallel-execution on non-unix platforms (see above), but this implementation could be a base to go about further improvements.

Of course we also compared the performance of these two implementations, the differences are absolutely disregarding. In some cases the file-IPC is even faster than the message-queue.

Verbose-mode (output)

As described above (see reportStatus), the taskScheduler is also responsible for the output during the test-execution. After executing a task, the child process passes the result to a static method of the outputWriter (rtTestOutputWriter::flushResult) which evaluates and flushes it to the console.

There are 3 levels of the verbose mode:

  • -v test status and test file
  • -vv same as above plus DESC and MSG about not-passed tests
  • -vvv all available information:
Example
(1)	XFAIL	/phpruntests/trunk/QA/QATESTS/tests/output/ob_011
(2)	DESC:	output buffering – fatalism
(3)	MSG:	This test will fail until the fix in version 1.178 of ext/main/output.c
(4)	CID:	5
(5)	MEM:	1727.43 kB

(6)	FILES:
	out:	/phpruntests/QA/QATESTS/tests/output/ob_011.out
	exp:	/phpruntests/QA/QATESTS/tests/output/ob_011.exp
	diff:	/phpruntests/QA/QATESTS/tests/output/ob_011.diff

(7)	PHP-COMMAND: /php/php53/sapi/cli/php  -d "output_handler=" -d "open_basedir=" -d "safe_mode=0" -d "disable_functions=" -d "output_buffering=Off" -d "error_reporting=32767" -d "display_errors=1" -d "display_startup_errors=1" -d "log_errors=0" -d "html_errors=0" -d "track_errors=1" -d "report_memleaks=1" -d "report_zend_debug=0" -d "docref_root=" -d "docref_ext=.html" -d "error_prepend_string=" -d "error_append_string=" -d "auto_prepend_file=" -d "auto_append_file=" -d "magic_quotes_runtime=0" -d "ignore_repeated_errors=0" -d "unicode.runtime_encoding=ISO-8859-1" -d "unicode.script_encoding=UTF-8" -d "unicode.output_encoding=UTF-8" -d "unicode.from_error_mode=U_INVALID_SUBSTITUTE" -f /phpruntests/QA/QATESTS/tests/output/ob_011.php  2>&1
  1. teststatus and testcase
  2. description (–TEST–)
  3. message (optional)
  4. child-ID – the id of the execution child-process (see child)
  5. current memory-usage
  6. saved files
  7. executed php-command

Integration in phpruntests

The taskScheduler is connected to the phpruntests-application at one point, in rtPhpTestRun. After preparing the command line arguments and environment parameter,s this class is responsible for collecting and preparing the test-cases and of course for kicking off the test-execution itself. And this is how it looks like:

       // create the task-list
       $taskList = array();
       foreach ($subDirectories as $subDirectory) {
       		$taskList[] = new rtTaskTestGroup($runConfiguration, $subDirectory);
       }

Initially the list of the task-objects is created. Every task gets passed the configuration-object, which is needed to execute the test-cases and the directory where the test-cases (phpt-files) are stored.

	// run the task-scheduler
	$scheduler = rtTaskScheduler::getInstance();
	$scheduler->setTaskList($taskList);
	$scheduler->setProcessCount($processCount);
	$scheduler->setReportStatus($reportStatus);
	$scheduler->run();
	$resultList = $scheduler->getResultList();

Subsequently a new instance of the taskScheduler is created to which the task-list is passed. Optionally the number of processes (processCount) and the verbose-level (reportStatus) can be defined. After running the taskScheduler the results are received and can be evaluated (this is the job of the outputWriter).

qa/runtests/documentation.txt · Last modified: 2011/04/06 12:59 (external edit)