rfc:context_sensitive_lexer

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
rfc:context_sensitive_lexer [2015/02/20 07:09] – marciorfc:context_sensitive_lexer [2017/09/22 13:28] (current) – external edit 127.0.0.1
Line 1: Line 1:
 ====== PHP RFC: Context Sensitive Lexer ====== ====== PHP RFC: Context Sensitive Lexer ======
-  * Version: 0.3+  * Version: 0.4.1
   * Date: 2015-02-15   * Date: 2015-02-15
   * Author: Márcio Almada   * Author: Márcio Almada
-  * Status: Draft+  * Status: Implemented (in PHP 7.0)
   * First Published at: http://wiki.php.net/rfc/context_sensitive_lexer   * First Published at: http://wiki.php.net/rfc/context_sensitive_lexer
  
Line 18: Line 18:
 class Collection { class Collection {
     public function forEach(callable $callback) { /* */ }     public function forEach(callable $callback) { /* */ }
 +    public function list() { /* */ }
 } }
  
-class List { 
-    public function append(List $list) { /* */ } 
-} 
 </code> </code>
  
-Notice that it's currently **not** possible to have the ''foreach'' method and ''List'' class delcared without having a syntax error:+Notice that it's currently **not** possible to have the ''foreach'' and ''list'' method declared without having a syntax error:
  
   PHP Parse error: Syntax error, unexpected T_FOREACH, expecting T_STRING on line 2   PHP Parse error: Syntax error, unexpected T_FOREACH, expecting T_STRING on line 2
-  PHP Parse error: Syntax error, unexpected T_LIST, expecting T_STRING on line 5+  PHP Parse error: Syntax error, unexpected T_LIST, expecting T_STRING on line 3
  
 ===== Proposal ===== ===== Proposal =====
  
 This RFC revisits the topic of [[https://wiki.php.net/rfc/keywords_as_identifiers|Keywords as Identifiers]] RFC. But this time This RFC revisits the topic of [[https://wiki.php.net/rfc/keywords_as_identifiers|Keywords as Identifiers]] RFC. But this time
-presenting a minimal and maintainable [[https://github.com/marcioAlmada/php-src/commit/d9d6f0c7e325dcd0d0ff3c3f2dc73c2364c3ad5f|patch]], +presenting a minimal and maintainable [[https://github.com/php/php-src/pull/1221|patch]], restricted to OO scope only, consistently comprehending:
-restricted to OO scope only, consistently comprehending:+
  
-  * Namespace, class, trait and interface names 
   * Properties, constants and methods defined on classes, interfaces and traits   * Properties, constants and methods defined on classes, interfaces and traits
   * Access of properties, constants and methods from objects and classes   * Access of properties, constants and methods from objects and classes
Line 43: Line 39:
  
   - Reduce the surface of BC breaks whenever new keywords are introduced   - Reduce the surface of BC breaks whenever new keywords are introduced
-  - Avoid restricting userland APIs. Dispensing the need for hacks like unecessary magic method calls, prefixed identifiers or the usage of a [[http://en.wikipedia.org/wiki/Thesaurus|thesaurus]] to avoid naming conflicts.+  - Avoid restricting userland APIs. Dispensing the need for hacks like unnecessary magic method calls, prefixed identifiers or the usage of a [[http://en.wikipedia.org/wiki/Thesaurus|thesaurus]] to avoid naming conflicts.
  
 This is a list of currently **globally** reserved words that will become **semi-reserved** in case proposed change gets approved: This is a list of currently **globally** reserved words that will become **semi-reserved** in case proposed change gets approved:
Line 51: Line 47:
   namespace  new  or  xor  try  use  var  exit  list  clone  include  include_once  throw  array   namespace  new  or  xor  try  use  var  exit  list  clone  include  include_once  throw  array
   print  echo  require  require_once  return  else  elseif  default  break  continue  switch  yield   print  echo  require  require_once  return  else  elseif  default  break  continue  switch  yield
-  function  if  endswitch  finally  for  foreach  declare  case  do  while  as  catch  die  self+  function  if  endswitch  finally  for  foreach  declare  case  do  while  as  catch  die  self parent
  
 ==== Limitations ==== ==== Limitations ====
  
-On purporse, it's still forbidden to define a **namespace**, **class**, **interface** or **trait** named as: +On purpose, it's still forbidden to define a **class constant** named as ''class'' because of the class name resolution ''::class'':
- +
-  * ''namespace'' +
-  * ''self'' +
-  * ''static'' +
-  * ''parent'' +
-  * ''array'' +
-  * ''callable''+
  
 <code php> <code php>
-namespace|class|interface|trait Namespace {}  // Fatal error +class Foo { 
-namespace|class|interface|trait Self {}       // Fatal error +  const class = 'Foo'; // Fatal error 
-namespace|class|interface|trait Static {}     // Fatal error +}
-namespace|class|interface|trait Parent {}     // Fatal error +
-namespace|class|interface|trait Array {}      // Fatal error +
-namespace|class|interface|trait Callable {}   // Fatal error+
  
-// Fatal error: Cannot use %s as %s name as it is reserved in %s on line %d+// Fatal error: Cannot redefine class constant Foo::CLASS as it is reserved in %s on line %d
 </code> </code>
  
-On purporse, it's still forbidden to define a **class constant** named as ''class'' because of the class name resolution operator ''::class'':+In practice, it means that we would drop from **64** to only **1** reserved word that affects only class constant names. 
 + 
 +''class|object'' properties **can** have any name because PHP has sigils and code like the following has always been allowed:
  
 <code php> <code php>
 class Foo { class Foo {
-  const class = 'Foo'; // Fatal error+  public $list = 'list';
 } }
  
-// Fatal error: Cannot redefine Foo::class, ::class is reserved in %s on line 2+(new Foo)->list;
 </code> </code>
  
Line 154: Line 142:
 } }
 </code> </code>
 +
 +===== Impact On Other RFCs =====
 +
 +Some RFCs are proposing to reserve new keywords in order to add features or reserve typehints names:
 +
 +  * https://wiki.php.net/rfc/in_operator
 +  * https://wiki.php.net/rfc/reserve_more_types_in_php_7
 +  * https://wiki.php.net/rfc/reserve_even_more_types_in_php_7
 +
 +With the approval of the current RFC, BC breaks surface would be much smaller in such cases.
 +
 +One notable example is the **in** operator RFC. Without a context sensitive lexer, proposed here, the new operator would create a BC break on **Doctrine** library and pretty much many other SQL writers or ORMs out there:
 +
 +https://github.com/doctrine/doctrine2/blob/master/lib/Doctrine/ORM/Query/Expr.php#L443
  
 ===== Implementation Details ===== ===== Implementation Details =====
 +
 +==== Patch 1 - Discarded ====
  
 The lexer now keeps track of the context needed to have unreserved words on OO scope and makes use of a minimal amount of RE2C lookahead capabilities when disambiguation becomes inevitable. The lexer now keeps track of the context needed to have unreserved words on OO scope and makes use of a minimal amount of RE2C lookahead capabilities when disambiguation becomes inevitable.
  
-For instance, the lexing rules to disambiguate ''::class'' (class name resolution operator) from a ''class constant'', ''static variable'' or ''static method'' access is:+For instance, the lexing rules to disambiguate ''::class'' (class name resolution operator) from a ''class constant'' or ''static method'' access is:
  
 <code c++> <code c++>
Line 172: Line 176:
 </code> </code>
  
-One additional compile time check was created:+A few additional compile time check were created:
  
 <code c> <code c>
-if (zend_string_equals_literal_ci(name, "class")) { +if(ZEND_NOT_RESERVED != zend_check_reserved_method_name(decl->name)) { 
-  zend_error_noreturn(E_COMPILE_ERROR, "Cannot redefine %s::%s as ::%s is reserved", +  zend_error_noreturn(E_COMPILE_ERROR, 
-    ce->name->val, name->val, name->val);+    "Cannot use '%s' as class method name as it is reserved", decl->name->val);
 } }
 </code> </code>
  
-Others were just adapted because, surprisingly, most of the necessary compile time checks were already in place and just needed +==== Patch 2 ==== 
-adjustments to restrict ''namespace'', ''array'' and ''callable'' as names. For instance the trait name validation:+ 
 +A new patch has been added during the voting phase. It's a different approach that proved to have many advantages over the first patch and therefore it is intended to supersede it. 
 + 
 +The new patch just requires the maintenance of a single inclusive parser rule listing all tokens that should be matched as a ''T_STRING'' on specific places: 
 + 
 +  - It offers no regression | forward compatibility risks and is highly predictable 
 +  - It has a very small footprint when compared to the previous attempt involving a pure lexical approach 
 +  - Requires no compile time checks 
 +  - Is highly configurable, to make a word semi-reserved you only have to edit an inclusive parser rule. 
 + 
 +In order to send information to the lexer about the context change, we just have to use ''identifier'' instead of ''T_STRING'' when applicable. For instance this is the needed changes on the parser grammar to allow semi reserved words on method names:
  
 <code c> <code c>
 // before // before
-if(ZEND_FETCH_CLASS_DEFAULT != zend_get_class_fetch_type(name)) { +method_modifiers function returns_ref T_STRING '(' parameter_list ')' //... 
-  zend_error_noreturn(E_COMPILE_ERROR, "Cannot use '%s' as trait name as it is reserved", name->val); +
-}+
 // after // after
-if(ZEND_FETCH_CLASS_DEFAULT != zend_check_reserved_name(name)) { +method_modifiers function returns_ref identifier '(' parameter_list ')' //...
-  zend_error_noreturn(E_COMPILE_ERROR, "Cannot use '%s' as trait name as it is reserved", name->val); +
-}+
 </code> </code>
  
-Current proposed patch:+===== Future Work And Maintenance =====
  
-  * Doesn't require ''lexical feedback'' (passing information from parser to lexer) +  * All php-src tests are passing with the new patch, some work still has to be done. There is a better possibility to expand semi reserved words support to namespaces and class names with the new patch, but this more ambitious proposal will be tailored only for PHP 7.1 by the RFC author.
-  * Keeps ext tokenizer functional +
-  * Introduces no maintenance issues +
-  * Has no performance impact +
-  * Introduces a minimal amount of changes on lexer+
  
-=> Many experiments with parsing were done before the current proposed patch which involves only lexing changes. But turns out the patches involving parsing had too many disadvantages and maintence issues.\\ +=> The first patch has been discarded during discussion on voting phase. It was considered too "ad-hoc" and could cause issues for PHP 7.1 and ahead.
-=> No performance loss was noticed but maybe the patch requires a better benchmark. +
- +
-===== Impact on performance ===== +
- +
-No loss noticed. +
- +
--- Add benchmark here if asked on discussion phase. --+
  
 ===== Proposed PHP Version(s) ===== ===== Proposed PHP Version(s) =====
Line 216: Line 216:
 This is proposed for the next PHP x, which at the time of this writing would be PHP 7. This is proposed for the next PHP x, which at the time of this writing would be PHP 7.
  
-===== Open Issues =====+===== Votes =====
  
-The patch may still contain small bugs related to the topics below, but this can be addressed during discussion phase:+This voting requires a 2/3 majority. The implementation will be evaluated on internals mailing list and will only be merged if it's 
 +considered good enough, independently of the voting results. The RCF author encourages voting for the feature.
  
-  * I still have to add more tests involving traits and trait conflict resolution syntax +<doodle title="Should PHP7 have a context sensitive lexer?" auth="marcio" voteType="single" closed="true"> 
-  * I still have to add more tests involving ''use X as Y'' syntax and entities with semi-reserved names+   * Yes 
 +   * No 
 +</doodle>
  
-The patch is 98% implemented and complexity around it will not grow. We could even vote the RFC before +Voting started on 2015-02-28 and ends on 2015-03-14.
-finishing these small details without impact on end quality.+
  
 ===== Patch ===== ===== Patch =====
  
-  - Most relevant commit is [[https://github.com/marcioAlmada/php-src/commit/c01014f9cca2eaf148ccc2e2e05f2b41d571b424|c01014f]], in case you would like to focus only on the important changes and skip the long tests. +==== Patch 1 - Discarded ==== 
-  - Pull request with all the tests and regenerated ext tokenizer is at [[https://github.com/php/php-src/pull/1054/files]]+ 
 +  - Pull request with all the tests and regenerated ext tokenizer is at [[https://github.com/php/php-src/pull/1054]] 
 + 
 +==== Patch 2 ==== 
 + 
 +  - Pull request with all the tests is at [[https://github.com/php/php-src/pull/1221/]] 
 + 
 +==== Later Changes === 
 + 
 +The *Patch 2* was merged and, later, method modifiers were allowed as class member names. This was a limitation from the older implementation candidate - Patch 1 - and there was no reason to keep it. The **Limitations** section was updated accordingly. Only the keyword **class** for class constants is reserved now.
  
 ===== References ===== ===== References =====
Line 237: Line 248:
 ===== Rejected Features ===== ===== Rejected Features =====
  
-None so far.+ * Prior to voting, the support for ''namespaces|classes|traits|interfaces'' names has been removed from the first patch as it could create some possible issues. 
 + 
 +=> The RFC author will try to solve the wider problem on PHP 7.1
  
 ===== Changelog ===== ===== Changelog =====
Line 243: Line 256:
   * 0.2: Additional support to namespaces, classes, interafces and traits names   * 0.2: Additional support to namespaces, classes, interafces and traits names
   * 0.3: Oops. Add forgotten support for typehints   * 0.3: Oops. Add forgotten support for typehints
 +  * 0.4: Reverts to 0.1 feature set because class name support created undesired situations regarding the future addition of a future short lambda syntax and possibly block other language changes.
 +  * 0.4.1: A new compatible implementation has been introduced
  
 ===== Acknowledgements ===== ===== Acknowledgements =====
rfc/context_sensitive_lexer.1424416176.txt.gz · Last modified: 2017/09/22 13:28 (external edit)

Page Tools