Windows Build System v2

• Author: Kalle Sommer Nielsen kalle@php.net
• Status: Draft
• Target: PHP 8

This document tries to define a new build system for PHP on Windows.

The Windows build system for PHP have undergone quite some changes during PHP's life span, from a project based system with VC6 in PHP4 to a JScript based one in PHP5 and PHP7. Over the time of PHP5's life span, this system has greatly been improved, notably in PHP5.3 and PHP7 there have been more and more additions to this, but we are reaching a state where it is time to look at improving this system heavily.

Originally the PHP5+ build system, was designed for MSVC only, but have been opened up to allow more compilers, although these are only semi working, such as Intel. Due to this, it makes it hard to open up the system for more compilers and adding new features to this system more or less feel like a hack because there is no proper consistency in the system.

This document tries to look at what we can do to improve upon the system, and at the same time make it more attractive for extension developers to write config files.

There have also been an effort to make a CMake based build system during the GSoC, but this was never completed.

Since this is a whole new API that does not retrain old functions, the config files for this build system is .win. This allows extensions to be built on both PHP5-7 and PHP 8. On PHP 8 the .w32 files are simply ignored and so are .win files on PHP5-7, allowing multi versions to co-exists.

The current implementation utilizes Windows Script Host (“WSH”), using the cscript command that is implemented in JScript, which is Microsoft's dialect of JavaScript. We will continue to use WSH as backend for the Windows build system.

The location of the build system will also remain in the php-src/win32/build directory, along with other build related sources.

A rough layout structure would be as follows:

win32/build/		- Core build system files
 +- configure.js
 +- Makefile
 +- ...
win32/build/features/	- Features for compilers
 +- MultiProcessBuild.js
 +- Debug.js
 +- ...
win32/build/compilers/	- Supported compilers
 +- MSVC.js
 +- Intel.js
 +- ...

On the Unix build system, the config.m4 files allows a numeric order to be loaded in, this is done by adding a suffix to “config” with a value of 0-9 followed by the file extension, such as “config9.m4”. The Windows build system does not offer such, and there are no plans to implement such in this proposal.

Since the current build system is designed for MSVC, this creates a barrier for other compilers to be implemented. Therefore all basic functionality that is required to build PHP, such as CFLAGS, LFLAGS etc must be written in an abstract way that allows other compilers to tab into that functionality and do their own implementation if needed by using function hooks.

A compiler would then be implemented sort of like a SAPI/extension is in php-src by using an object/structure:

class MSVC implements Compiler
{
	public const name : string 		= 'Microsoft Visual C++ Compiler';
	public const short_name : string	= 'MSVC';
 
	public const compiler : string;
	public const linker : string;
 
	public arch : int;
	public archs : int			= ArchType.x86 | ArchType.x64;
 
	/* ... */
 
	public function MSVC()
	{
		this.compiler 	= PHPBuild.FindBinary('cl.exe');
		this.linker	= PHPBuild.FindBinary('link.exe');
 
		if(!this.isSupported())
		{
			throw new PHPBuildException('This version of the compiler is not supported');
		}
 
		/* ... */
	};
 
	private function isSupported() : bool
	{
		/* ... */
	};
 
	public function ...() : ...
	{
		/* ... */
	};
}
 
PHPBuild.AddCompiler(new MSVC);

This object would be called every time an operation related to the compiler is needed, such as checking if a feature is supported or to find its binary locations.

Given the proposed abstraction, this also allows us to make the build system support more architectures, as well as adding more in the future. By default we support x86 and x64 architectures, but we could potentially add support relatively simple by abstracting this in process if revamping the build system. This could potentially mean that we could build PHP for ARM on Windows in the future.

In the above example, the compiler.archs property is a bitfield for the ArchType enum that tells what supported architectures that currently are supported by this compiler.

By default the ArchType will contain two architectures, with expansion for more later on:

• x86
• x64

Currently PHP have tons of flags for different options sent to the compiler and linker, these are currently only controllable by manually adding a configure argument that then translates it into flag for specific compilers and does some work behind the scene.

Since features for compilers are sometimes specific, but often abstract, such as generating assembly output for source files. A list of features, that each compiler can hook into and translate that feature into flags sent to the compiler would greatly help improve over all support and consistency across such.

Features are represented as simple objects, a canonical feature name that compilers must understand, their feature name which can be used for verbosity and a few enabling hooks:

class MultiProcessBuild implements Feature
{
	public const name : string 		= 'mp';
	public const verbose_name : string	= 'Multi Process Build';
 
	public const arg_type : FeatureArgType	= FeatureArgType.WITH;
 
	public level : int			= 0;
 
	public function configure(arg)
	{
		// We know that if the value is numeric, this is number of processes 
		// that can be passed, we allow a maximum of 65356 if this is the case
 
		if(arg.isNumeric())
		{
			this.level = Math.min(arg.value, 65356);
		}
	};
 
	public function enable(compiler) : bool
	{
		switch(compiler.short_name)
		{
			case 'MSVC':
				return compiler.addCFlag('/MP' + (this.level > 0 ? this.level : ''));
			break;
			case '...':
				/* ... */
			break;
		}
 
		return false;
	};
}
 
PHPBuild.AddFeature(new MultiProcessBuild);

At first glance this seems like an over kill for a simple feature, such as adding a simple compiler flag, but it does provide flexibility to enable handling of how this feature will handled across different compilers. If we did not care about allowing to specify a custom level and if the compiler flag was the same across the board, we could trim the code down a bit:

class MultiProcessBuild implements Feature
{
	public const name : string 		= 'mp';
	public const verbose_name : string	= 'Multi Process Build';
 
	public const arg_type : FeatureArgType	= FeatureArgType.WITH;
 
 
	public function enable(compiler) : bool
	{
		return compiler.addCFlag('/MP');
	};
}
 
PHPBuild.AddFeature(new MultiProcessBuild);

Some features may depend on other features, if we stick to the Multi Process Build feature example, then this feature may not actually be enabled for debug builds. This means that it should explicitly be disallowed when it is configured, the configure method would then look like this:

// Notice that unlike the initial configure() method, we actually care about the 
// second parameter, "compiler", sent to this method now
public function configure(arg, compiler)
{
	// Multi Process Build is not supported in debug builds
	if(compiler.build_type == BuildType.Debug)
	{
		throw new PHPBuildException('Multi process builds and debug builds is not supported at the same time');
	}
 
	// We know that if the value is numeric, this is number of processes 
	// that can be passed, we allow a maximum of 65356 if this is the case
 
	if(arg.isNumeric())
	{
		this.level = Math.min(arg.value, 65356);
	}
};

Extensions and SAPIs currently have need to check for:

• Checking for libraries
• Setting compiler and linker flags
• Declaring themselves

While the last two points is not extremely difficult, checking for libraries can get messy, lets use ext/curl as an example for now. Consider the config.w32 file for ext/curl in PHP 7.2:

config.w32

// $Id$
// vim:ft=javascript
 
ARG_WITH("curl", "cURL support", "no");
 
if (PHP_CURL != "no") {
	if (CHECK_LIB("libcurl_a.lib;libcurl.lib", "curl", PHP_CURL) &&
			CHECK_HEADER_ADD_INCLUDE("curl/easy.h", "CFLAGS_CURL") &&
			CHECK_LIB("ssleay32.lib", "curl", PHP_CURL) &&
			CHECK_LIB("libeay32.lib", "curl", PHP_CURL) 
		&& CHECK_LIB("winmm.lib", "curl", PHP_CURL)
		&& CHECK_LIB("wldap32.lib", "curl", PHP_CURL)
		&& (((PHP_ZLIB=="no") && (CHECK_LIB("zlib_a.lib;zlib.lib", "curl", PHP_CURL))) || 
			(PHP_ZLIB_SHARED && CHECK_LIB("zlib.lib", "curl", PHP_CURL)) || (PHP_ZLIB == "yes" && (!PHP_ZLIB_SHARED)))
		) {
		EXTENSION("curl", "interface.c multi.c share.c curl_file.c");
		AC_DEFINE('HAVE_CURL', 1, 'Have cURL library');
		AC_DEFINE('HAVE_CURL_SSL', 1, 'Have SSL suppurt in cURL');
		AC_DEFINE('HAVE_CURL_EASY_STRERROR', 1, 'Have curl_easy_strerror in cURL');
		AC_DEFINE('HAVE_CURL_MULTI_STRERROR', 1, 'Have curl_multi_strerror in cURL');
		ADD_FLAG("CFLAGS_CURL", "/D CURL_STATICLIB");
		// TODO: check for curl_version_info
	} else {
		WARNING("curl not enabled; libraries and headers not found");
	}
}

This does a whole lot of things:

• Checking for libcurl_a.lib (static) or libcurl.lib
• Checking for curl/easy.h
• Checking for ssleay32.lib (openssl)
• Checking for libeay32.lib (openssl)
• Checking for winmm.lib (Windows Multi Media library)
• Checking for wldap32.lib (Windows implementation of LDAP)
• Checking for ext/zlib
  • If ext/zlib is disabled:
    • Check for zlib_a.lib (static) or zlib.lib
  • If ext/zlib is shared
    • Check for zlib.lib
  • Check for ext/zlib and that it is not shared
• Define extension information with files to compile
• Define HAVE_CURL
• Define HAVE_CURL_SSL
• Define HAVE_CURL_EASY_STRERROR
• Define HAVE_CURL_MULTI_STRERROR
• Set compiler flag /D CURL_STATICLIB

This is a lot of things to manually check for an extension, and as extensions depend on others which may be shared or statically built into PHP it becomes harder and harder to maintain. In fact, everything before declaring what files to compiler is in one big if conditional, this could be trimmed down of course to make it easier to see what is going on. One of the aims for the new build system is to provide flexibility for extensions in a reasonable manner and let the build system work its magic to hide manually having to remember and update all these checks and their static/shared counterparts.

A more elegant and rough concept can be illustrated like this:

class curl extends Extension
{
	public const name : string		= 'cURL';
	public const short_name : string	= 'curl';
 
	public const arg_type : ArgType		= ArgType.WITH;
 
	public shared : bool;
 
	public const files : array		= ['interface.c', 'multi.c', 'share.c', 'curl_file.c'];
 
 
	public curl()
	{
		// Required libraries, notice no _a here
		this.Libs(['libcurl', 'winmm', 'ssleay32', 'libeay32', 'wldap32']);
 
		// ZLib is special, as we do not want to make the php binary larger if we already
		// have Zlib enabled. The second argument to the Libs() method will check 
		// for static library versions too, this is disabled by setting this to false/0.
		this.Libs(['zlib'], (zlib = this.getExtension('zlib') ? !zlib.shared : true));
 
		// Required header
		this.Header('curl/easy.h');
 
		// Defines, the build system itself will register HAVE_<extension.short_name.toUpperCase()>
		this.Define('HAVE_CURL_SSL', 1);
		this.Define('HAVE_CURL_EASY_STRERROR', 1);
		this.Define('HAVE_CURL_MULTI_STRERROR', 1);
 
		// Compiler flag
		this.CFlag('/D CURL_STATICLIB');
}

This makes the libraries much easier to handle and check and it provides the same flexibility as before in a more friendly interface. Above example is just a small illustration of the API that can be used, the full API is described later in this document.

It is common for extensions to depend on others, these can either be optional, required or even semi depended on others, like the ext/curl extension that can depend on ZLib depending on how it was configured.

In the Extension constructor, the following methods are available for dependencies, the following example will use ext/pdo_sqlite:

public function pdo_sqlite()
{
	this.Dependencies(['pdo']);
};

If the 'pdo' extension was not enabled prior to this, then an exception is thrown from the Dependencies() method, stopping the configure command.

An extension can also optionally depend on another, and if it is available, then an extension can take advantage of its utility:

public function test_ext()
{
	// Works for both static and shared builds
	if(this.getExtension('json'))
	{
		this.Define('HAVE_JSON_SUPPORT', 1);
	}
 
	// Or if static json is required
	if(this.getExtension('json', this.build_type.StaticBuild))
	{
		this.Define('HAVE_STATIC_JSON_SUPPORT', 1);
	}
};

Some extensions, or SAPIs run external programs before compiling for generating parsers or similar, such is also possible in the new build system:

public function json()
{
	if(!this.FSO.FileExists('ext/json/json_scanner.c'))
	{
		PHPBuild.ExecuteBinary('re2c', '-t ext/json/php_json_scanner_defs.h --no-generation-date -bci -o ext/json/json_scanner.c ext/json/json_scanner.re');
	}
 
	/* ... */
};

It is very common for extensions to provide additional configurations from arguments, by default each declared extension will be supported by either:

• --with-<ext>
• --enable-<ext>

This type is defined in the Extension.arg_type property to an enum value of either ArgType.WITH or ArgType.ENABLE. However extensions may also add their own, this is done in the property called 'config_args':

class test_ext extends Extension
{
	public const name : string		= 'Test Extension';
	public const short_name : string	= 'test_ext';
 
	public const arg_type : ArgType		= ArgType.WITH;
	public const config_args : object	= {
							'test-ext-debug': ArgType.ENABLE
							};
 
	public shared : bool;
 
	public const files : array		= ['php_test_ext.c'];
 
 
	public function test_ext()
	{
		if(this.GetArg('test-ext-debug').isEnabled())
		{
			this.CFlag('/D HAVE_TEST_EXT_DEBUG');
		}
	}
}
 
PHPBuild.AddExtension(new test_ext);

The GetArg() method returns an 'BuildArgument' object, that contains helper methods, such as isEnabled() to quickly determine the value was turned by a manual --enable-test-ext-debug

There are some extensions that can be built in multiple ways, one such case is ext/pdo_dblib

TODO

• sapis
• consider changing 'new xxx' in calls, as they are executed directly
• improve arg api for exts
• multi build args for an ext
• multi extensions per config, like pdo_dblib
• talk about task size and target
• talk about design reasoning
• API reference
• implementation stages(?)