Request for Comments: Alternative implementation of mbstring using ICU

• Version: 1.3
• Date: 2014-02-05
• Author: Yasuo Ohgaki yohgaki@ohgaki.net Moriyoshi Koizumi moriyoshi@php.net
• Status: Declined
• First Published at: http://wiki.php.net/rfc/altmbstring

This RFC discusses the alternative implementation of mbstring extension that in turn uses ICU instead of libmbfl.

Note: This RFC is related to https://wiki.php.net/rfc/multibyte_char_handling This RFC is for long term resolution for multibyte character encoding related issues.

Note: This RFC is also address LGPL license issue used by current mbstring module. It is preferred to have alternative to mbstring that does not have license issue.

• LGPL license - libmbfl(multibyte filter) and Oniguruma(Multibyte regular expression) is licensed by LGPL. Users that complie PHP statically may have license problem.
• Lack of understanding -- It took long for those who don't use Unicode or other non-single-byte codesets to figure out how essential the functionality this extension covers, just until recently.
• Huge bundled libraries -- One of the bundled libraries, libmbfl, consists of a large set of Unicode-to-legacy charset mapping tables and vice versa. This may look redundant to those who aren't interested in manipulating multibyte strings.
• Limited support for locales -- libmbfl has a setting called “NLS” that determines the defaults for several functions, but only a random list of locales are supported; Armenian, Chinese (simplified and traditional), English, German, Japanese, Korean, Russian and Turkish (yes, French is not there...).
• Incompliancy with the standards -- Character cases are not well handled in a case-insensitive matches performed by stripos(), strripos() and so on because libmbfl doesn't implement Unicode collations.

To overcome these issues, a complete rewrite of the extension has long been wanted. But it didn't come into reality because there was no good Unicode library. Now that ICU is stable and we already relies on it (intl in 5.3), why not make it happen?

It is currently hosted by GitHub.

http://github.com/moriyoshi/mbstring-ng/

• mb_convert_encoding()
• mb_detect_encoding()
• mb_ereg()
• mb_ereg_replace()
• mb_internal_encoding()
• mb_list_encodings()
• mb_output_handler()
• mb_parse_str()
• mb_preferred_mime_name()
• mb_regex_set_options()
• mb_split()
• mb_strcut()
• mb_strimwidth()
• mb_stripos()
• mb_stristr()
• mb_strlen()
• mb_strpos()
• mb_strripos()
• mb_strrpos()
• mb_strstr()
• mb_strtolower()
• mb_strtotitle()
• mb_strtoupper()
• mb_strwidth()
• mb_substr()
• mb_substr_count()

• All features that exist in mbstring will be ported to mbstring-ng unless there are technical difficulties.

• mb_detect_encoding() doesn't work well anymore due to the inaccuracy of ICU's encoding detection.
• Request encoding translator now takes advantage of SAPI filter, therefore the name parts of the query components are not to be converted anymore.
• The features supported by ICU's regular expression engine is not as rich as of Oniguruma, which resulted in the reduced set f options for mb_regex_set_options(). With respect to this, I also extracted the regex functions from the former mbstring and repackaged it to be oniguruma extension
• The group reference placeholders for mb_ereg_replace() is now $0, $1, $2... instead of \0, \1, \2. This can be avoided if we don't use uregex_replaceAll() and implement our own.
• ILP64

Introduce mbsgring-ng as EXPERIMENTAL module for testing compatibility against existing applications.

Compiling multibyte aware module by default is important for eliminating vulnerabilities related to character encoding. Compile mbstring-ng by default when it is ready. Replace mbstring by mstring-ng if it is possible.

There will be a RFC for replacing mbstring by mbstring-ng, how it will be replaced, what to do with legacy mbstring, etc. Replacing module would be PHP 6 matter as it would break some applications.

It is better to remove LGPLed code from 'must have' module. mbstring-ng shall remove this issue.

Note: Even when PHP supports Unicode internally, multibyte aware features/functions are needed to handle char encoding properly. Unicode does not solve all issues. There should be some module to handle it. Otherwise, default string functions must have encoding parameters and it would be copy of mb_*() functions.

PHP 5.6 and up

VOTE: 2014/02/10 - 2014/02/17

Include mbstring-ng for PHP-5.6 as EXPERIMENTAL module
Real name	Yes	No
aharvey (aharvey)
bwoebi (bwoebi)
chobieeee (chobieeee)
derick (derick)
krakjoe (krakjoe)
mbeccati (mbeccati)
nikic (nikic)
pollita (pollita)
treffynnon (treffynnon)
tyrael (tyrael)
yohgaki (yohgaki)
Final result:	1	10
This poll has been closed.

Thank you for voting!

• https://wiki.php.net/rfc/multibyte_char_handling

1. 2014-01-27 Yasuo Ohgaki: Updated to replace existing mbstring
2. 2009-07-27 Moriyoshi Koizumi: Initial