rfc:is_literal

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
rfc:is_literal [2021/06/21 19:27] craigfrancisrfc:is_literal [2022/02/14 00:36] (current) – Add some more examples from other languages craigfrancis
Line 1: Line 1:
-====== PHP RFC: Is_Trusted ======+====== PHP RFC: Is_Literal ======
  
-  * Version: 0.9 +  * Version: 1.1 
-  * Date: 2020-03-21 +  * Voting Start: 2021-07-05 19:30 BST / 18:30 UTC 
-  * Updated: 2021-06-06+  * Voting End: 2021-07-19 19:30 BST / 18:30 UTC 
 +  * RFC Started: 2020-03-21 
 +  * RFC Updated: 2021-07-04
   * Author: Craig Francis, craig#at#craigfrancis.co.uk   * Author: Craig Francis, craig#at#craigfrancis.co.uk
-  * Contributors: Joe Watkins, Máté Kocsis, Dan Ackroyd +  * Contributors: Joe Watkins, Máté Kocsis 
-  * Status: Under Discussion+  * Status: Voting
   * First Published at: https://wiki.php.net/rfc/is_literal   * First Published at: https://wiki.php.net/rfc/is_literal
   * GitHub Repo: https://github.com/craigfrancis/php-is-literal-rfc   * GitHub Repo: https://github.com/craigfrancis/php-is-literal-rfc
 +  * Implementation: https://github.com/php/php-src/compare/master...krakjoe:literals
  
 ===== Introduction ===== ===== Introduction =====
  
-Add the function //is_trusted()//, so developers can check if a variable can be trusted to not contain an Injection Vulnerability.+Add the function //is_literal()//, a lightweight and effective way to identify if a string was written by a developer, removing the risk of a variable containing an Injection Vulnerability.
  
-By trusting integers and literals (strings defined by the programmer), we can provide a lightweight, simple, and very effective way to prevent Injection Vulnerabilities.+It's a simple process where a flag is set internally on strings that have been written by a developer (as opposed to a user), where the flag persists through concatenation with other 'literal' strings. The function checks the flag is present and thus no user data is included.
  
 It avoids the "false sense of security" that comes with the flawed "Taint Checking" approach, [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/escaping.php?ts=4|because escaping is very difficult to get right]]. It's much safer for developers to use parameterised queries, and well-tested libraries. It avoids the "false sense of security" that comes with the flawed "Taint Checking" approach, [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/escaping.php?ts=4|because escaping is very difficult to get right]]. It's much safer for developers to use parameterised queries, and well-tested libraries.
  
-These libraries require certain sensitive values to only come from the developer; but because it's [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/mistakes.php?ts=4|easy to incorrectly include user values]], Injection Vulnerabilities are still introduced by the thousands of developers using these libraries incorrectly. You will notice the linked examples are based on examples found in the Libraries' official documentation, they still "work", and are typically shorter/easier than doing it correctly (I've found many of them on live websites, and it's why I'm here). A simple Query Builder example being:+//is_literal()// can be used by libraries to deal with a difficult problem - developers using them incorrectly. Libraries expect certain sensitive values to only come from the developer; but because it's [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/mistakes.php?ts=4|easy to incorrectly include user values]], Injection Vulnerabilities are still introduced by the thousands of developers using these libraries incorrectly. You will notice the linked examples are based on examples found in the Libraries' official documentation, they still "work", and are typically shorter/easier than doing it correctly (I've found many of them on live websites, and it's why I'm here). A simple Query Builder example being:
  
 <code php> <code php>
Line 26: Line 29:
 </code> </code>
  
-The "Future Scope" section explains why a dedicated type should come later, and how native functions could use the //trusted// flag as well.+(The "Future Scope" section explains why a dedicated type should come later, and how native functions could use the //is_literal// flag as well.)
  
 ===== Background ===== ===== Background =====
Line 34: Line 37:
 Injection and Cross-Site Scripting (XSS) vulnerabilities are **easy to make**, **hard to identify**, and **very common**. Injection and Cross-Site Scripting (XSS) vulnerabilities are **easy to make**, **hard to identify**, and **very common**.
  
-We like to think every developer reads the documentation, and would never directly include (inject) user values into their SQL/HTML/CLI - but we all know that's not the case.+With SQL Injection, it just takes 1 mistake, and the attacker can usually read everything in the database (SQL Map, Havij, jSQL, etc). 
 + 
 +When it comes to coding, we like to think every developer reads the documentation, and would never directly include (inject) user values into their SQL/HTML/CLI - but we all know that's not the case.
  
 It's why these two issues have **always** been on the [[https://owasp.org/www-project-top-ten/|OWASP Top 10]]; a list designed to raise awareness of common issues, ranked on their prevalence, exploitability, detectability, and impact: It's why these two issues have **always** been on the [[https://owasp.org/www-project-top-ten/|OWASP Top 10]]; a list designed to raise awareness of common issues, ranked on their prevalence, exploitability, detectability, and impact:
Line 56: Line 61:
 ==== Usage in PHP ==== ==== Usage in PHP ====
  
-Libraries would be able to use //is_trusted()// immediately, allowing them to warn developers about Injection Issues as soon as they receive any non-trusted values. Some already plan to implement this, for example:+Libraries would be able to use //is_literal()// immediately, allowing them to warn developers about Injection Issues as soon as they receive any non-literal values. Some already plan to implement this, for example:
  
 **Propel** (Mark Scherer): "given that this would help to more safely work with user input, I think this syntax would really help in Propel." **Propel** (Mark Scherer): "given that this would help to more safely work with user input, I think this syntax would really help in Propel."
Line 62: Line 67:
 **RedBean** (Gabor de Mooij): "You can list RedBeanPHP as a supporter, we will implement this into the core." **RedBean** (Gabor de Mooij): "You can list RedBeanPHP as a supporter, we will implement this into the core."
  
-**Psalm** (Matthew Brown): "I've just added support for a //literal-string// type to Psalm: https://psalm.dev/r/9440908f39"+**Psalm** (Matthew Brown): 13th June "I was skeptical about the first draft of this RFC when I saw it last month, but now I see the light (especially with the concat changes)". Then on the 14th June, "I've just added support for a //literal-string// type to Psalm: https://psalm.dev/r/9440908f39" ([[https://github.com/vimeo/psalm/releases/tag/4.8.0|4.8.0]]) 
 + 
 +**PHPStan** (Ondřej Mirtes): 1st September, has been implemented in [[https://github.com/phpstan/phpstan/releases/tag/0.12.97|0.12.97]].
  
 ===== Proposal ===== ===== Proposal =====
  
-Add the function //is_trusted()//, where "trusted" is defined as:+Add the function //is_literal()//.
  
-  - Strings defined by the programmer (literals), +A string shall pass the //is_literal// check if it was defined by the programmer in source codeor is the result of a function or instruction whose inputs would all pass the //is_literal// check.
-  - Strings defined by the engine (interned strings), +
-  - Any integer.+
  
-No input values are interned; instead an interned string includes:+Concatenation instructions and the following string functions are therefore able to produce literals:
  
-  - Strings defined by the programmer, +  - //str_repeat()// 
-  - Strings defined in the source code of php, +  - //str_pad()// 
-  - Strings defined by the engine (either at compile or runtime), with known values.+  - //implode()// 
 +  - //join()//
  
-Any function or instruction that is aware of trusted variables shall produce a trusted string if all input would pass //is_trusted()//This includes //sprintf()//, //str_repeat()//, //str_pad()//, //implode()//, //join()//, //array_pad()//, and //array_fill()//.+(Namespaces constructed for the programmer by the compiler will also be marked literal for convenience.)
  
 <code php> <code php>
-is_trusted('Example'); // true+is_literal('Example'); // true
  
 $a = 'Hello'; $a = 'Hello';
 $b = 'World'; $b = 'World';
  
-is_trusted($a); // true +is_literal($a); // true 
-is_trusted($a . $b); // true +is_literal($a . $b); // true 
-is_trusted("Hi $b"); // true+is_literal("Hi $b"); // true
  
-is_trusted($_GET['id']); // false +is_literal($_GET['id']); // false 
-is_trusted(sprintf('Hi %s', $_GET['name'])); // false +is_literal(sprintf('Hi %s', $_GET['name'])); // false 
-is_trusted('/bin/rm -rf ' . $_GET['path']); // false +is_literal('/bin/rm -rf ' . $_GET['path']); // false 
-is_trusted('<img src=' . htmlentities($_GET['src']) . ' />'); // false +is_literal('<img src=' . htmlentities($_GET['src']) . ' />'); // false 
-is_trusted('WHERE id = ' . $db->real_escape_string($_GET['id'])); // false+is_literal('WHERE id = ' . $db->real_escape_string($_GET['id'])); // false
  
 function example($input) { function example($input) {
-  if (!is_trusted($input)) { +  if (!is_literal($input)) { 
-    throw new Exception('Non-trusted value detected!');+    throw new Exception('Non-literal value detected!');
   }   }
   return $input;   return $input;
Line 104: Line 110:
  
 example($a); // OK example($a); // OK
-example(example($a)); // OK, still the same trusted value.+example(example($a)); // OK, still the same literal value.
 example(strtoupper($a)); // Exception thrown. example(strtoupper($a)); // Exception thrown.
 </code> </code>
  
-===== Try it =====+===== Try It =====
  
-[[https://3v4l.org/#focus=rfc.literals|Have a play with it on 3v4l.org]]+[[https://3v4l.org/#focus=rfc.literals|Test it out on 3v4l.org]]
  
-[[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4|How it can be used by libraries]] - Notice how this example library just raises a warning, to simply let the developer know about the issue, **without breaking anything**. And it provides an //"unsafe_value"// value-object to bypass the //is_trusted()// check, but none of the examples need to use it (can be useful as a temporary thing, but there are much safer/better solutions, which developers are/should already be using).+[[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4|How it can be used by libraries]] - Notice how this example library just raises a warning, to simply let the developer know about the issue, **without breaking anything**. And it provides an //"unsafe_value"// value-object to bypass the //is_literal()// check, but none of the examples need to use it (can be useful as a temporary thing, but there are much safer/better solutions, which developers are/should already be using).
  
 ===== FAQ's ===== ===== FAQ's =====
Line 130: Line 136:
 All three examples would be incorrectly considered "safe" (untainted). The first two need the values to be quoted. The third example, //htmlentities()// does not escape single quotes by default before PHP 8.1 ([[https://github.com/php/php-src/commit/50eca61f68815005f3b0f808578cc1ce3b4297f0|fixed]]), and it does not consider the issue of 'javascript:' URLs. All three examples would be incorrectly considered "safe" (untainted). The first two need the values to be quoted. The third example, //htmlentities()// does not escape single quotes by default before PHP 8.1 ([[https://github.com/php/php-src/commit/50eca61f68815005f3b0f808578cc1ce3b4297f0|fixed]]), and it does not consider the issue of 'javascript:' URLs.
  
-In comparison, //is_trusted()// doesn't have an equivalent of //untaint()//, or support escaping. Instead PHP will set the trusted flag, and as soon as the value has been manipulated or includes anything that is not trusted (e.g. user data), the trusted flag is lost.+In comparison, //is_literal()// doesn't have an equivalent of //untaint()//, or support escaping. Instead PHP will set the //is_literal// flag, and as soon as the value has been manipulated or includes anything that is not a literal (e.g. user data), the //is_literal// flag is removed.
  
-This allows libraries to use //is_trusted()// to check the sensitive values they receive from the developer. Then it's up to the library to handle the escaping (if it's even needed). The "Future Scope" section notes how native functions will be able to use the trusted flag as well.+This allows libraries to use //is_literal()// to check the sensitive values they receive from the developer. Then it's up to the library to handle the escaping (if it's even needed). The "Future Scope" section notes how native functions would be able to use the //is_literal// flag as well.
  
 ==== Education ==== ==== Education ====
  
-**Why not educate everyone?**+**Why not educate everyone instead?**
  
 You can't - developer training simply does not scale, and mistakes still happen. You can't - developer training simply does not scale, and mistakes still happen.
Line 146: Line 152:
 **Why not use static analysis?** **Why not use static analysis?**
  
-It will never be used by most developers.+Ultimately it will never be used by most developers.
  
 I still agree with [[https://news-web.php.net/php.internals/109192|Tyson Andre]], you should use Static Analysis, but it's an extra step that most programmers cannot be bothered to do, especially those who are new to programming (its usage tends to be higher among those writing well-tested libraries). I still agree with [[https://news-web.php.net/php.internals/109192|Tyson Andre]], you should use Static Analysis, but it's an extra step that most programmers cannot be bothered to do, especially those who are new to programming (its usage tends to be higher among those writing well-tested libraries).
Line 164: Line 170:
 **What about the performance impact?** **What about the performance impact?**
  
-Máté Kocsis has created a [[https://github.com/kocsismate/php-version-benchmarks/|php benchmark]] to replicate the old [[https://01.org/node/3774|Intel Tests]], and the [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/tests/results/with-concat/kocsismate.pdf|preliminary testing on this implementation]] has found a 0.124performance hit for the Laravel Demo app, and 0.161% for Symfony (rounds 4-6, which involved 5000 requests). These tests do not connect to a database, as the variability introduced makes it impossible to measure that low of a difference+Máté Kocsis has created a [[https://github.com/kocsismate/php-version-benchmarks/|php benchmark]] to replicate the old [[https://01.org/node/3774|Intel Tests]], the preliminary results found a 0.47impact with the Symfony demo app (it did not connect to a database, as the variability introduced would make it impossible to measure the difference).
- +
-The full stress-test is 3.719% when running this [[https://github.com/kocsismate/php-version-benchmarks/blob/main/app/zend/concat.php#L25|concat test]], but this is not representative of a typical PHP script (it's not normal to concatenate 4 strings, 5 million times, with no other actions). +
- +
-Joe Watkins has also noted that further optimisations are possible (the implementation has focused on making it work).+
  
 ==== String Concatenation ==== ==== String Concatenation ====
Line 174: Line 176:
 **Is string concatenation supported?** **Is string concatenation supported?**
  
-Yes. The trusted flag is preserved when two trusted values are concatenated; this makes it easier to use //is_trusted()//, especially by developers that use concatenation for their SQL/HTML/CLI/etc+Yes. The //is_literal// flag is preserved when two literal values are concatenated; this makes it easier to use //is_literal()//, especially by developers that use concatenation for their SQL/HTML/CLI/etc.
- +
-Previously we tried a version that only supported concatenation at compile-time (not run-time), to see if it would reduce the performance impact even further. The idea was to require everyone to use special //trusted_concat()// and //trusted_implode()// functions, which would raise exceptions to highlight where mistakes were made. These two functions can still be implemented by developers themselves (see [[#support_functions|Support Functions]] below), as they can be useful; but requiring everyone to use them would have required big changes to existing projects, and exceptions are not a graceful way of handling mistakes.+
  
-Performance wisemy [[https://github.com/craigfrancis/php-is-literal-rfc/tree/main/tests|simplistic testing]] found there was still [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/tests/results/with-concat/local.pdf|a small impact without run-time concat]]:+Previously we tried a version that only supported concatenation at compile-time (not run-time)to see if it would reduce the performance impact even furtherThe idea was to require everyone to use special //literal_concat()// and //literal_implode()// functions, which would raise exceptions to highlight where mistakes were madeThese two functions can still be implemented by developers themselves (see [[#support_functions|Support Functions]] below), as they can be useful; but requiring everyone to use them would have required big changes to existing projects, and exceptions are not a graceful way of handling mistakes.
  
-    Laravel Demo App: +0.30% withvs +0.18% without. +Performance wisemy [[https://github.com/craigfrancis/php-is-literal-rfc/tree/main/tests|simplistic testing]] found there was still [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/tests/results/with-concat/local.pdf|a small impact without run-time concat]].
-    Symfony Demo App+0.06% with, vs +0.06% without. +
-    My Concat Test  +4.36% with, vs +2.23% without+
-    - +
-    Website with 22 SQL queries: Inconclusive, too variable.+
  
-(This is because //concat_function()// in "zend_operators.c" uses //zend_string_extend()// which needs to remove the trusted flag. Also "zend_vm_def.h" does the same; and supports a quick concat with an empty string (x2), which would need its flag removed as well).+(Under The Hood: This is because //concat_function()// in "zend_operators.c" uses //zend_string_extend()// which needs to remove the //is_literal// flag. Also "zend_vm_def.h" does the same; and supports a quick concat with an empty string (x2), which would need its flag removed as well).
  
 And by supporting both forms of concatenation, it makes it easier for developers to understand (many are not aware of the difference). And by supporting both forms of concatenation, it makes it easier for developers to understand (many are not aware of the difference).
Line 192: Line 188:
 ==== String Splitting ==== ==== String Splitting ====
  
-**Why don't you support string splitting?**+**Why don't you support string splitting then?**
  
 In short, we can't find any real use cases (security features should try to keep the implementation as simple as possible). In short, we can't find any real use cases (security features should try to keep the implementation as simple as possible).
  
-Also, the security considerations are different. Concatenation joins known/fixed units together, whereas if you're starting with a trusted string, and the program allows the Evil-User to split the string (e.g. setting the length in substr), then they get considerable control over the result (it creates an untrusted modification).+Also, the security considerations are different. Concatenation joins known/fixed units together, whereas if you're starting with a literal string, and the program allows the Evil-User to split the string (e.g. setting the length in substr), then they get considerable control over the result (it creates an untrusted modification).
  
 These are unlikely to be written by a programmer, but consider these: These are unlikely to be written by a programmer, but consider these:
Line 208: Line 204:
 If that URL was used in a Content-Security-Policy, then it's necessary to remove the query string, but as more of the string is removed, the more resources can be included ("https:" basically allows resources from anywhere). With the HTML example, moving from the tag content to the attribute can be a problem (technically the HTML Templating Engine should be fine, but unfortunately libraries like Twig are not currently context aware, so you need to change from the default 'html' encoding to explicitly using 'html_attr' encoding). If that URL was used in a Content-Security-Policy, then it's necessary to remove the query string, but as more of the string is removed, the more resources can be included ("https:" basically allows resources from anywhere). With the HTML example, moving from the tag content to the attribute can be a problem (technically the HTML Templating Engine should be fine, but unfortunately libraries like Twig are not currently context aware, so you need to change from the default 'html' encoding to explicitly using 'html_attr' encoding).
  
-Or in other words; trying to determine if the //trusted// flag should be passed through functions like //substr()// is difficult. Having a security feature be difficult to reason about, gives a much higher chance of mistakes.+Or in other words; trying to determine if the //is_literal// flag should be passed through functions like //substr()// is complex. Having a security feature be difficult to reason about, gives a much higher chance of mistakes.
  
 Krzysztof Kotowicz has confirmed that, at Google, with "go-safe-html", splitting is explicitly not supported because it "can cause issues"; for example, "arbitrary split position of a HTML string can change the context". Krzysztof Kotowicz has confirmed that, at Google, with "go-safe-html", splitting is explicitly not supported because it "can cause issues"; for example, "arbitrary split position of a HTML string can change the context".
Line 216: Line 212:
 **What about an undefined number of parameters, e.g. //WHERE id IN (?, ?, ?)//?** **What about an undefined number of parameters, e.g. //WHERE id IN (?, ?, ?)//?**
  
-If the values are explicitly cast to integers, there is no need to make a change. +You can follow the advice from [[https://stackoverflow.com/a/23641033/538216|Levi Morrison]], [[https://www.php.net/manual/en/pdostatement.execute.php#example-1012|PDO Execute]], and [[https://www.drupal.org/docs/7/security/writing-secure-code/database-access#s-multiple-arguments|Drupal Multiple Arguments]], and implement as such:
- +
-That said, ideally you would still follow the advice from [[https://stackoverflow.com/a/23641033/538216|Levi Morrison]], [[https://www.php.net/manual/en/pdostatement.execute.php#example-1012|PDO Execute]], and [[https://www.drupal.org/docs/7/security/writing-secure-code/database-access#s-multiple-arguments|Drupal Multiple Arguments]], and implement as such:+
  
 <code php> <code php>
Line 224: Line 218:
 </code> </code>
  
-Or, if you prefer to use concatenation:+Or, you could use concatenation:
  
 <code php> <code php>
Line 232: Line 226:
 } }
 </code> </code>
 +
 +And libraries can easily abstract this for the developer.
  
 ==== Non-Parameterised Values ==== ==== Non-Parameterised Values ====
Line 253: Line 249:
 By using an allow-list, we ensure the user (attacker) cannot use anything unexpected. By using an allow-list, we ensure the user (attacker) cannot use anything unexpected.
  
-==== Non-Trusted Values ====+==== Non-Literal Values ====
  
-**How does this work in cases where you can't use trusted values?**+**How does this work in cases where you can't use literal values?**
  
 For example [[https://news-web.php.net/php.internals/87667|Dennis Birkholz]] noted that some Systems/Frameworks currently define some variables (e.g. table name prefixes) without the use of a literal (e.g. ini/json/yaml). And Larry Garfield noted that in Drupal's ORM "the table name itself is user-defined" (not in the PHP script). For example [[https://news-web.php.net/php.internals/87667|Dennis Birkholz]] noted that some Systems/Frameworks currently define some variables (e.g. table name prefixes) without the use of a literal (e.g. ini/json/yaml). And Larry Garfield noted that in Drupal's ORM "the table name itself is user-defined" (not in the PHP script).
  
-While most systems can use trusted values entirely, these special non-trusted values should still be handled separately (and carefully). This approach allows the library to ensure the majority of the input (SQL) is trusted, and then it can consistently check/escape those special values (e.g. does it match a valid table/field name, which can be included safely).+While most systems can use literal values entirely, these special non-literal values should still be handled separately (and carefully). This approach allows the library to ensure the majority of the input (SQL) is a literal, and then it can consistently check/escape those special values (e.g. does it match a valid table/field name, which can be included safely).
  
 [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4#L194|How this can be done with aliases]], or the [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4#L229|example Query Builder]]. [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4#L194|How this can be done with aliases]], or the [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4#L229|example Query Builder]].
  
-==== Usage by Libraries ====+==== Faking It ====
  
-**Could libraries use is_trusted() internally?**+**What if I really really need to mark a value as a literal?**
  
-Yes, they could.+This implementation does not provide a way for a developer to mark anything they want as a literal. This is on purpose. We do not want to recreate the biggest flaw of Taint Checking. It would be very easy for a naive developer to mark all escaped values as a literal (seeing it as a safe value, which is [[#taint_checking|wrong]]).
  
-It would be fantastic if they did use additional //is_trusted()// checks after receiving the values from developers (it ensures the library hasn't introduced a vulnerability either)but this isn't a prioritysimply because libraries are rarely the source of Injection Vulnerabilities.+That said, we do not pretend there aren't ways around this (e.g. using [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/is-literal-bypass.php|var_export]])but doing so is clearly the developer doing something wrong. We want to provide safety railsbut there is nothing stopping the developer from jumping over them if that's their choice.
  
-That said, consider the Drupalgeddon vulnerability; where //$db->expandArguments()// allowed unsafe/non-trusted values to be used as placeholders with //IN (:arg_0, :arg_1)//. By using something like the [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4#L229|example Query Builder]], //is_trusted()// could have been used to check the raw SQL, and then the non-trusted field/parameter names would be checked and applied separately.+==== Usage by Libraries ====
  
-Zend also had a couple of issues with ORDER BY, where it didn't check the inputs either ([[https://framework.zend.com/security/advisory/ZF2014-04|1]]/[[https://framework.zend.com/security/advisory/ZF2016-03|2]]).+**How can libraries use is_literal()?**
  
-==== Integer Values ====+The main focus is on values that developers provide to the library, this [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4|example library]] shows how certain sensitive values are checked as they are received, where it just uses basic warnings by default, could raise exceptions, or have the checks turned off on a per query basis (or entirely). Libraries could choose to only run these checks in development mode (and turned off in production), or do additional checks to see if the value is likely to be an issue (e.g. value matches a field name), or write to a log, or report via an API/email, etc.
  
-**Can you support Integer values?**+They could also use additional //is_literal()// checks later in the process (internally), to ensure the library hasn't introduced a vulnerability either; but this isn't a priority, simply because libraries are rarely the source of Injection Vulnerabilities.
  
-Yes, support for integers is now included. +==== Integer Values ====
- +
-It was noted by Matthew Brown (and others) that a lot of existing code and tutorials uses integers directly, and they do not cause a security issue. +
- +
-We tried to flag integers defined in the source code, in the same way we are doing with strings. Unfortunately [[https://news-web.php.net/php.internals/114964|it would require a big change to add a trusted flag on integers]]. Changing how integers work internally would have made a big performance impact, and potentially affected every part of PHP (including extensions).+
  
-That said, while it's not as philosophically pure, we can still trust all integers in regards to Injection Vulnerabilitiesno matter where they came from.+We wanted to flag integers defined in the source code, in the same way we are doing with strings. Unfortunately [[https://news-web.php.net/php.internals/114964|it would require a big change to add a literal flag on integers]]. Changing how integers work internally would have made a big performance impactand potentially affected every part of PHP (including extensions).
  
-We have not found any issues with allowing integers in SQL, HTML, CLI; and other contexts as well (e.g. preg, mail additional_params, XPath query, and even eval).+Due to this limitation, we considered an approach to trust all integers. It was noted that existing code and tutorials already use integers directly. While this is not as philosophically pure, we continued to explore this possibility because we could not find any way that an Injection Vulnerability could be introduced with integers in SQL, HTML, CLI; and other contexts as well (e.g. preg, mail additional_params, XPath query, and even eval).
  
-We could not find any character encoding issues (The closest we could find was EBCDIC, an old IBM character encoding, which encodes the 0-9 characters differently; which anyone using it would need to re-encode either way, and [[https://www.php.net/manual/en/migration80.other-changes.php#migration80.other-changes.ebcdic|EBCDIC is not supported by PHP]]).+We could not find any character encoding issues either (The closest we could find was EBCDIC, an old IBM character encoding, which encodes the 0-9 characters differently; which anyone using it would need to re-encode either way, and [[https://www.php.net/manual/en/migration80.other-changes.php#migration80.other-changes.ebcdic|EBCDIC is not supported by PHP]]). And we could not find any issue with a 64bit PHP server sending a large number to a 32bit database, because the number is being encoded as characters in a string, so that's also fine.
  
-We also checked what would happen if a 64bit PHP server sent a large number to a 32bit databasebut that's not an issue eitherbecause the number is being encoded as characters in a stringso that's also fine.+Howeverthe feedback received on the Internals mailing list was that while safe from Injection Vulnerabilities it might cause developers to assume them to be safe from developer/logic errorsand ultimately the preference was the simpler approach, that did not allow integers from any source.
  
 ==== Other Values ==== ==== Other Values ====
Line 297: Line 289:
 **Why don't you support Boolean/Float values?** **Why don't you support Boolean/Float values?**
  
-It's a very low value feature, and we cannot be sure of the security implications.+It's a very low-value feature, and we cannot be sure of the security implications.
  
-For example, the value you put in often is not always the same as what you get out:+For example, the value you put in is not always the same as what you get out:
  
 <code php> <code php>
Line 314: Line 306:
 ==== Naming ==== ==== Naming ====
  
-**Why is it called is_trusted()?**+**Why is it called is_literal()?**
  
-First, there is no perfect name.+A "Literal String" is the standard name for strings in source code. See [[https://www.google.com/search?q=what+is+literal+string+in+php|Google]].
  
-We did start with //is_literal()// as placeholder name (at a time we only trusted literals). This name wasn't perfect, but it would have allowed developers to search and get an idea of what literal wasWhen [[#integer_values|integer values]] were deemed necessary to help adoption, the name became more of a problem. We also need to keep to a single word name (to support a dedicated type in the future). This is where //is_trusted()// and //is_known()// was proposed. We had a [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/name/2021-07-20.png|vote on the name]], which gave us a 18 to 3 result in favour of //is_trusted()//.+> A string literal is the notation for representing string value within the text of a computer programIn PHPstrings can be created with single quotes, double quotes or using the heredoc or the nowdoc syntax. 
 + 
 +We also need to keep to a single word name (to support a dedicated type in the future).
  
 ==== Support Functions ==== ==== Support Functions ====
Line 324: Line 318:
 **What about other support functions?** **What about other support functions?**
  
-We did consider //trusted_concat()// and //trusted_implode()// functions (see [[#string_concatenation|String Concatenation]] above), but these can be userland functions:+We did consider //literal_concat()// and //literal_implode()// functions (see [[#string_concatenation|String Concatenation]] above), but these can be userland functions:
  
 <code php> <code php>
-function trusted_implode($separator, $array) {+function literal_implode($separator, $array) {
   $return = implode($separator, $array);   $return = implode($separator, $array);
-  if (!is_trusted($return)) {+  if (!is_literal($return)) {
       // You will probably only want to raise       // You will probably only want to raise
       // an exception on your development server.       // an exception on your development server.
-    throw new Exception('Non-trusted value detected!');+    throw new Exception('Non-literal value detected!');
   }   }
   return $return;   return $return;
 } }
  
-function trusted_concat(...$a) { +function literal_concat(...$a) { 
-  return trusted_implode('', $a);+  return literal_implode('', $a);
 } }
 </code> </code>
Line 356: Line 350:
 </code> </code>
  
-If a developer changed the literal //'ASC'// to //$_GET['order']//, the error would be noticed by //$db->query()//, but it's not clear where the non-trusted value was introduced. Whereas, if they used //trusted_concat()//, that would raise an exception much earlier, and highlight exactly where the mistake happened:+If a developer changed the literal //'ASC'// to //$_GET['order']//, the error would be noticed by //$db->query()//, but it's not clear where the non-literal value was introduced. Whereas, if they used //literal_concat()//, that would raise an exception much earlier, stopping script execution, and highlight exactly where the mistake happened:
  
 <code php> <code php>
-$sql = trusted_concat($sql, ' ORDER BY name ', $sortOrder);+$sql = literal_concat($sql, ' ORDER BY name ', $sortOrder);
 </code> </code>
  
Line 366: Line 360:
 **Why not support other string functions?** **Why not support other string functions?**
  
-Like [[#string_splitting|String Splitting]], we can't find any real use cases, and don't want to make this complicated. For example //strtoupper()// might be reasonable, but we will need to consider how it would be used (good and bad), and check for any oddities (e.g. output varying based on the current locale). Also, functions like //str_shuffle()// create unpredictable results.+Like [[#string_splitting|String Splitting]], we can't find any real use cases, and don't want to make this complicated. For example //strtoupper()// might be reasonable, but we would need to consider how it would be used, and check for any oddities (e.g. output varying based on the current locale). Also, functions like //str_shuffle()// create unpredictable results.
  
 ==== Limitations ==== ==== Limitations ====
Line 372: Line 366:
 **Does this mean the value is completely safe?** **Does this mean the value is completely safe?**
  
-While these values can be trusted to not contain an Injection Vulnerability, they cannot be completely safe from every kind of issue, For example:+While these values are not at risk of containing an Injection Vulnerability, obviously they cannot be completely safe from every kind of developer/logic issue, For example:
  
 <code php> <code php>
Line 383: Line 377:
 There's no single RFC that can completely solve all developer errors, but this takes one of the biggest ones off the table. There's no single RFC that can completely solve all developer errors, but this takes one of the biggest ones off the table.
  
-==== Faking it ====+==== Compiler Optimisations ====
  
-**What if I really want to mark a value as trusted?**+The implementation has been updated to avoid situations that could have confused the developer:
  
-This implementation does not provide way for a developer to mark anything they want as trustedThis is on purposeWe do not want to recreate the biggest flaw of Taint CheckingIt would be very easy for naive developer to mark all escaped values as trusted ([[#taint_checking|wrong]]).+<code php> 
 +$one = 1; 
 +$= 'A' $one; // false, flag removed because it's being concatenated with an integer. 
 +$b = 'A' . 1; // Was true, as the compiler optimised this to the literal 'A1'. 
 + 
 +$= "Hello "; 
 +$b = $a . 2; // Was true, as the 2 was coerced to the string '2' (to optimise the concatenation). 
 + 
 +$a = implode("-", [1, 2, 3]); // Was true with OPcache, as it could optimise this to the literal '1-2-3' 
 + 
 +$a = chr(97); // Was true, due to the use of Interned Strings. 
 +</code>
  
-That said, we do not pretend there aren't ways around this (e.g. using [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/is-trusted-bypass.php|var_export]]), but doing so is clearly the developer doing something wrongWe want to provide safety rails, but there is nothing stopping the developer from jumping over them if that's their choice.+This has been achieved by using the Lexer to mark strings as a literal (i.eearlier in the process).
  
 ==== Extensions ==== ==== Extensions ====
Line 395: Line 400:
 **Extensions create and manipulate strings, won't this break the flag on strings?** **Extensions create and manipulate strings, won't this break the flag on strings?**
  
-Strings have multiple flags already that are off by default - this is the correct behaviour when extensions create their own strings (should not be flagged as trusted). If an extension is found to be already using the flag we're using for is_trusted (unlikely), that's the same as any new flag being introduced into PHP, and will need to be updated in the same way.+Strings have multiple flags already that are off by default - this is the correct behaviour when extensions create their own strings (should not be flagged as a literal). If an extension is found to be already using the flag we're using for is_literal (unlikely), that's the same as any new flag being introduced into PHP, and will need to be updated in the same way.
  
 ==== Reflection API ==== ==== Reflection API ====
Line 401: Line 406:
 **Why don't you use the Reflection API?** **Why don't you use the Reflection API?**
  
-This allows you to "introspect classes, interfaces, functions, methods and extensions"; it's not currently set up for object methods to inspect the code calling it. Even if that was to be added (unlikely), it could only check if the trusted value was defined there, it couldn't handle variables (tracking back to their source), nor could it provide any future scope for a dedicated type, nor could native functions work with this (see "Future Scope").+This allows you to "introspect classes, interfaces, functions, methods and extensions"; it's not currently set up for object methods to inspect the code calling it. Even if that was to be added (unlikely), it could only check if the literal value was defined there, it couldn't handle variables (tracking back to their source), nor could it provide any future scope for a dedicated type, nor could native functions work with this (see "Future Scope").
  
-==== Interned Strings ====+===== Previous Examples =====
  
-**Why does the output from //chr()// appear as trusted?**+**Go** can use an "[[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/others/go/index.go|un-exported string type]]", a technique which is used by [[https://blogtitle.github.io/go-safe-html/|go-safe-html]].
  
-This was noticed by Claude Pache, and on technical level is due to the [[https://news-web.php.net/php.internals/114877|use of Interned Strings]], an optimisation used by //RETURN_CHAR// that re-uses single character valuesIt's effectively the same as calling //sprintf('%c', $i)//, which is also not an issue, as the developer is choosing to do this.+**C++** can use "[[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/others/cpp/index.cpp|consteval annotation]]".
  
-===== Previous Work =====+**Rust** can use a "[[https://github.com/craigfrancis/php-is-literal-rfc/tree/main/others/rust|procedural macro]]", to check the provided value is a literal at compile time (a bit complicated).
  
-**Go** programs can use "ScriptFromConstant" to express the concept of a "compile time constant" ([[https://blogtitle.github.io/go-safe-html/|more details]]).+**Java** can use a "[[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/others/java/src/main/java/com/example/isliteral/index.java|@CompileTimeConstant annotation]]" from [[https://errorprone.info/bugpattern/CompileTimeConstant|Error Prone]] to ensure method parameters can only use "compile-time constant expressions".
  
-**Java** can use [[https://errorprone.info/|Error Prone]] with [[https://errorprone.info/bugpattern/CompileTimeConstant|@CompileTimeConstant]] to ensure method parameters can only use "compile-time constant expressions".+**Node** has the [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/others/npm/index.js|is-template-object polyfill]], which checks a tag function was provided a "tagged template literal" (this technique is used in [[https://www.npmjs.com/package/safesql|safesql]], via [[https://www.npmjs.com/package/template-tag-common|template-tag-common]]). Alternatively Node developers can use [[https://github.com/craigfrancis/php-is-literal-rfc/blob/main/others/npm-closure-library/index.js|goog.string.Const]] from Google's Closure Library.
  
 **JavaScript** is getting [[https://github.com/tc39/proposal-array-is-template-object|isTemplateObject]], for "Distinguishing strings from a trusted developer from strings that may be attacker controlled" (intended to be [[https://github.com/mikewest/tc39-proposal-literals|used with Trusted Types]]). **JavaScript** is getting [[https://github.com/tc39/proposal-array-is-template-object|isTemplateObject]], for "Distinguishing strings from a trusted developer from strings that may be attacker controlled" (intended to be [[https://github.com/mikewest/tc39-proposal-literals|used with Trusted Types]]).
  
-**Perl** has a [[https://perldoc.perl.org/perlsec#Taint-mode|Taint Mode]], via the -T flag, where all input is marked as "tainted", and cannot be used by some methods (like commands that modify files), unless you use a regular expression to match and return known-good values (where regular expressions are easy to get wrong).+**Perl** has a [[https://perldoc.perl.org/perlsec#Taint-mode|Taint Mode]], via the -T flag, where all input is marked as "tainted", and cannot be used by some methods (like commands that modify files), unless you use a regular expression to match and return known-good values (regular expressions are easy to get wrong).
  
 There is a [[https://github.com/laruence/taint|Taint extension for PHP]] by Xinchen Hui, and [[https://wiki.php.net/rfc/taint|a previous RFC proposing it be added to the language]] by Wietse Venema. There is a [[https://github.com/laruence/taint|Taint extension for PHP]] by Xinchen Hui, and [[https://wiki.php.net/rfc/taint|a previous RFC proposing it be added to the language]] by Wietse Venema.
Line 428: Line 433:
   * Each of those functions would need a bypass for cases where unsafe SQL was intentionally being used (e.g. phpMyAdmin taking SQL from POST data) because some applications intentionally "pass raw, user submitted, SQL" (Ronald Chmara [[https://news-web.php.net/php.internals/87406|1]]/[[https://news-web.php.net/php.internals/87446|2]]).   * Each of those functions would need a bypass for cases where unsafe SQL was intentionally being used (e.g. phpMyAdmin taking SQL from POST data) because some applications intentionally "pass raw, user submitted, SQL" (Ronald Chmara [[https://news-web.php.net/php.internals/87406|1]]/[[https://news-web.php.net/php.internals/87446|2]]).
  
-All of these concerns have been addressed by //is_trusted()//.+All of these concerns have been addressed by //is_literal()//.
  
-I also agree with [[https://news-web.php.net/php.internals/87400|Scott Arciszewski]], "SQL injection is almost a solved problem [by using] prepared statements", where //is_trusted()// is essential for identifying the mistakes developers are still making.+I also agree with [[https://news-web.php.net/php.internals/87400|Scott Arciszewski]], "SQL injection is almost a solved problem [by using] prepared statements", where //is_literal()// is essential for identifying the mistakes developers are still making.
  
 ===== Backward Incompatible Changes ===== ===== Backward Incompatible Changes =====
  
-No known BC breaks, except for code-bases that already contain the userland function //is_trusted()// which is unlikely.+No known BC breaks, except for code-bases that already contain the userland function //is_literal()// which is unlikely.
  
 ===== Proposed PHP Version(s) ===== ===== Proposed PHP Version(s) =====
Line 460: Line 465:
 ===== Future Scope ===== ===== Future Scope =====
  
-1) As noted by someniatko and Matthew Brown, having a dedicated type would be useful in the future, as "it would serve clearer intent", which can be used by IDEs, Static Analysis, etc. It was [[https://externals.io/message/114835#114847|agreed we would add this type later]], via a separate RFC, so this RFC can focus on the trusted flag, and provide libraries a simple backwards-compatible function, where they can decide how to handle non-trusted values.+1) As noted by someniatko and Matthew Brown, having a dedicated type would be useful in the future, as "it would serve clearer intent", which can be used by IDEs, Static Analysis, etc. It was [[https://externals.io/message/114835#114847|agreed we would add this type later]], via a separate RFC, so this RFC can focus on the //is_literal// flag, and provide libraries a simple backwards-compatible function, where they can decide how to handle non-literal values.
  
 2) As noted by MarkR, the biggest benefit will come when this flag can be used by PDO and similar functions (//mysqli_query//, //preg_match//, //exec//, etc). 2) As noted by MarkR, the biggest benefit will come when this flag can be used by PDO and similar functions (//mysqli_query//, //preg_match//, //exec//, etc).
  
-However, first we need libraries to start using //is_trusted()// to check their inputs. The library can then do their thing, and apply the appropriate escaping, which can result in a value that no longer has the trusted flag set, but should still be trusted.+However, first we need libraries to start using //is_literal()// to check their inputs. The library can then do their thing, and apply the appropriate escaping, which can result in a value that no longer has the //is_literal// flag set, but is perfectly safe for the native functions.
  
-With a future RFC, we could potentially introduce checks for the native functions. For example, if we use the [[https://web.dev/trusted-types/|Trusted Types]] concept from JavaScript (which protects [[https://www.youtube.com/watch?v=po6GumtHRmU&t=92s|60+ Injection Sinks]], like innerHTML), the libraries create a stringable object as their output. These objects can be added to a list of trusted objects for the relevant native functions. The native functions could then **warn** developers when they do not receive a value with the trusted flag, or one of the trusted objects. These warnings would **not break anything**, they just make developers aware of the mistakes they have made, and we will always need a way of switching them off entirely (e.g. phpMyAdmin).+With a future RFC, we could potentially introduce checks for the native functions. For example, if we use the [[https://web.dev/trusted-types/|Trusted Types]] concept from JavaScript (which protects [[https://www.youtube.com/watch?v=po6GumtHRmU&t=92s|60+ Injection Sinks]], like innerHTML), the libraries create a stringable object as their output. These objects can be added to a list of safe objects for the relevant native functions. The native functions could then **warn** developers when they do not receive a value with the //is_literal// flag, or one of the safe objects. These warnings would **not break anything**, they just make developers aware of the mistakes they have made, and we will always need a way of switching them off entirely (e.g. phpMyAdmin).
  
-===== Proposed Voting Choices =====+===== Voting =====
  
-Accept the RFCYes/No+Accept the RFC 
 + 
 +<doodle title="is_literal" auth="craigfrancis" voteType="single" closed="true"> 
 +   Yes 
 +   No 
 +</doodle>
  
 ===== Implementation ===== ===== Implementation =====
  
 [[https://github.com/php/php-src/compare/master...krakjoe:literals|Joe Watkin's implementation]] [[https://github.com/php/php-src/compare/master...krakjoe:literals|Joe Watkin's implementation]]
- 
-===== References ===== 
- 
-N/A 
  
 ===== Rejected Features ===== ===== Rejected Features =====
  
-N/A+  - [[#integer_values|Supporting Integers]]
  
 ===== Thanks ===== ===== Thanks =====
Line 488: Line 494:
   - **Joe Watkins**, krakjoe, for writing the full implementation, including support for concatenation and integers, and helping me though the RFC process.   - **Joe Watkins**, krakjoe, for writing the full implementation, including support for concatenation and integers, and helping me though the RFC process.
   - **Máté Kocsis**, mate-kocsis, for setting up and doing the performance testing.   - **Máté Kocsis**, mate-kocsis, for setting up and doing the performance testing.
-  - **Dan Ackroyd**, DanAck, for starting the [[https://github.com/php/php-src/compare/master...Danack:is_trusted_attempt_two|first implementation]], which made this a reality, providing //literal_concat()// and //literal_implode()//, and followup on how it should work.+  - **Scott Arciszewski**, CiPHPerCoder, for checking over the RFC, and provided text on how we could implement integer support under a //is_noble()// name. 
 +  - **Dan Ackroyd**, DanAck, for starting the [[https://github.com/php/php-src/compare/master...Danack:is_literal_attempt_two|first implementation]], which made this a reality, providing //literal_concat()// and //literal_implode()//, and followup on how it should work.
   - **Xinchen Hui**, who created the Taint Extension, allowing me to test the idea; and noting how Taint in PHP5 was complex, but "with PHP7's new zend_string, and string flags, the implementation will become easier" [[https://news-web.php.net/php.internals/87396|source]].   - **Xinchen Hui**, who created the Taint Extension, allowing me to test the idea; and noting how Taint in PHP5 was complex, but "with PHP7's new zend_string, and string flags, the implementation will become easier" [[https://news-web.php.net/php.internals/87396|source]].
   - **Rowan Francis**, for proof-reading, and helping me make an RFC that contains readable English.   - **Rowan Francis**, for proof-reading, and helping me make an RFC that contains readable English.
Line 494: Line 501:
   - **Nikita Popov**, NikiC, for suggesting where the flag could be stored. Initially this was going to be the "GC_PROTECTED flag for strings", which allowed Dan to start the first implementation.   - **Nikita Popov**, NikiC, for suggesting where the flag could be stored. Initially this was going to be the "GC_PROTECTED flag for strings", which allowed Dan to start the first implementation.
   - **Mark Randall**, MarkR, for suggestions, and noting that "interned strings in PHP have a flag", which started the conversation on how this could be implemented.   - **Mark Randall**, MarkR, for suggestions, and noting that "interned strings in PHP have a flag", which started the conversation on how this could be implemented.
-  - **Sara Golemon**, SaraMG, for noting how this RFC had to explain how //is_trusted()// is different to the flawed Taint Checking approach, so we don't get "a false sense of security or require far too much escape hatching".+  - **Sara Golemon**, SaraMG, for noting how this RFC had to explain how //is_literal()// is different to the flawed Taint Checking approach, so we don't get "a false sense of security or require far too much escape hatching".
  
rfc/is_literal.1624303641.txt.gz · Last modified: 2021/06/21 19:27 by craigfrancis