rfc:retry-keyword

PHP RFC: Retry functionality

  • Version: 1.0
  • Date: 2016-06-19
  • Author: Sammy Kaye Powers, me@sammyk.me
  • Status: Under Discussion

Introduction

The retry keyword will make it easier to re-execute code blocks that failed due to recoverable errors by jumping to the top of the try block.

PHP is primarily used for web apps, and many times these apps need to communicate with third-party services or data stores that can temporality fail with recoverable errors. Failures typically throw exceptions which can be captured in a try/catch/finally block. Once a recoverable error reaches a catch block, it's not always trivial in user-land to retry the try block.

This RFC proposes adding two implementations of the retry keyword to a try/catch/finally block with a separate vote for each implementation.

Block-level implementation example

The retry block allows a developer to:

  • Retry the try block n times or forever (in this example 3 times)
  • Execute arbitrary code before each retry (to sleep, log, check exception code, etc)
  • Access to the number of times the try block has been executed (with $attempt)
  • Use the break keyword to break out of retry attempts for reasons other than 1) the exception not being thrown or 2) running out of retry attempts; (more on this later)
try {
    somethingSketchy();
} retry 3 (RecoverableException $e, $attempt) {
    echo "Failed doing sketchy thing on try #{$attempt}. Retrying...";
    sleep(1);
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

Keyword implementation example

The retry keyword implementation offers a very light-weight syntax for retrying the try block.

try {
    somethingSketchy();
} catch (RecoverableException $e)
    retry; // Go to top of try block
}

Proposal

This new keyword makes what is currently somewhat of a chore into a quick and easy activity.

Currently in order to retry a block of code that failed with a recoverable error in user-land, the developer needs to write quite a bit of bootstrap code. It forces the developer to make extra methods or functions, which are not always necessary or even helpful, and are just written to avoid copy/paste.

Currently

function uploadImage($path) {
  $attempt = function() use ($path) {
    $obj = $this->s3->bucket('bucket-name')->object('key');
    $obj->upload_file($path);
  };
 
  try {
    $attempt();
  } catch (AWS\S3\UploadException $e)
    $attempt();
  }
}

Proposed

function uploadImage($path) {
    try {
        $obj = $this->s3->bucket('bucket-name')->object('key');
        $obj->upload_file($path);
    } catch (AWS\S3\UploadException $e)
        retry;
    }
}

There are currently a few ways to implement a feature that will retry a failed block of code n times.

Recursive Functions

function myRetryFunction($maxTries) {
    try {
        somethingSketchy();
    } catch (RecoverableException $e) {
        if ($maxTries === 0) {
            die('Tried a bunch but failed.');
        }
        myRetryFunction(--$maxTries);
    }
}
 
myRetryFunction(5);

Wrapping recoverable failures in functions/closures is less than ideal. Let's try another method.

For Loops

$maxTries = 5;
 
for ($x=0; $x<=$maxTries; $x++) {
    try {
        somethingSketchy();
        break;
    } catch (RecoverableException $e) {
        die('Tried a bunch but failed.');
    }
}

Wrapping the recoverable code in for loops is also less than ideal as it make the code less readable.

Use Goto

$maxTries = 5;
 
retryTheThing:
try {
    somethingSketchy();
} catch (RecoverableException $e) {
    if (--$maxTries > 0) {
        goto retryTheThing;
    }
    die('Tried a bunch but failed.');
}

While this is arguably the cleanest option, it still requires the developer to define and manage a label which over several refactors might gradually move further away from the top of the try line. This implementation also makes it easy to accidentally execute any code after the label & before the try for each retry which is not entirely obvious at first glance.

$maxTries = 5;
 
retryTheThing:
 
someCodeIDoNotWantToRetry();
 
try {
    somethingSketchy();
} catch (RecoverableException $e) {
    if (--$maxTries > 0) {
        goto retryTheThing;
    }
    die('Tried a bunch but failed.');
}

Use Retry

Using the retry keyword implementation allows a developer to use her own method of tracking the number of attempts as well as a way to execute any arbitrary code before trying again.

const MAX_TRIES = 5;
$attempt = 0;
try {
    somethingSketchy();
} catch (RecoverableException $e) {
    if (++$attempt < MAX_TRIES) {
        sleep(1);
        // And log stuff maybe
        retry;
    }
    die('Tried a bunch but failed.');
}

Alternatively using the block-level implementation of retry keeps the developer from having to write her own boilerplate to both 1) track the number of attempts and 2) execute arbitrary code before retrying.

try {
    somethingSketchy();
} retry 5 (RecoverableException $e, $attempt) {
    sleep(1);
    // And maybe log stuff when $attempt > 3 or something
} catch (RecoverableException $e) {
    die('Tried a bunch but failed.');
}

The code becomes quite a bit more readable because it removes the nested structures one's eyes has to jump through while scanning the code.

Both the retry keyword and block-level implementations really do simplify quite a few workflows, that many PHP developers have just learned to accept, and could be improved greatly.

Catching multiple exceptions

As the retry block is basically a fancy catch block, it also supports catching multiple recoverable exceptions.

try {
    throw new RecoverableException("FAILED");
} retry 3 (RecoverableException | AnotherRecoverableException $e, $attempt) {
    echo "Failed on try #{$attempt}. Retrying...";
    sleep(1);
} catch (RecoverableException $e) {
    echo $e->getMessage();
}
 
// AnotherRecoverableException bubbles up after 3 retries

Bubbling up exceptions

When a retry block exists on a try/catch/finally structure, the catch block is optional allowing the exception to bubble up for another part of the application to catch it.

try {
    throw new RecoverableException("FAILED");
} retry 10 (RecoverableException $e, $attempt) {
    echo "Retrying...";
}
 
// After 10 times RecoverableException is still thrown & uncaught

Implicit infinite retries

The number of retries can be defined with an int literal or a constant. If omitted, retry will assume infinite retries.

try {
    throw new RecoverableException("FAILED");
} retry (RecoverableException $e, $attempt) {
    echo "Retrying forever...";
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

Alternatively the INF math constant can be used to specify infine retries.

try {
    throw new RecoverableException("FAILED");
} retry INF (RecoverableException $e, $attempt) {
    echo "Retrying forever...";
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

Breaking out of retry

It is sometimes necessary to have some logic that would abort any more retry attempts, like in the case of retrying forever. That can be done using the break keyword.

try {
    throw new RecoverableException("FAILED");
} retry INF (RecoverableException $e, $attempt) {
    if (42 === $e->getCode()) {
        break;
    }
    echo "Retrying forever...";
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

A full example

Below is a full example that illustrates the full potential of the retry block and all of its features.

class RecoverableException extends Exception {}
class AnotherRecoverableException extends Exception {}
class NonRecoverableException extends Exception {}
 
$id = 42;
try {
    throw new RecoverableException("FAILED getting ID #{$id}");
} retry 3 (RecoverableException | AnotherRecoverableException $e, $attempt) {
    if (42 === $e->getCode()) {
        break;
    }
    echo "Failed getting ID #{$id} on try #{$attempt}. Retrying...";
    sleep(1);
} catch (RecoverableException | AnotherRecoverableException $e) {
    echo $e->getMessage();
} catch (NonRecoverableException $e) {
    echo $e->getMessage();
}

In order to illustrate just how much boilerplate the retry block removes, check out a full userland implementation that covers all the features of the example above with a fully-featured retry() function.

class RecoverableException extends Exception {}
class AnotherRecoverableException extends Exception {}
class NonRecoverableException extends Exception {}
 
function retry(int $retryCount, callable $tryThis, callable $beforeRetry = null, array $targetExceptions = ['Exception'])
{
    $attempts = 0;
    tryCode:
    try {
        return $tryThis();
    } catch (\Throwable $e) {
        $isTargetException = false;
        foreach ($targetExceptions as $targetException) {
            if ($e instanceof $targetException) {
                $isTargetException = true;
                break;
            }
        }
        if (!$retryCount || !$isTargetException) {
            throw $e;
        }
        $retryCount--;
        $shouldRetry = true;
        if ($beforeRetry) {
            $shouldRetry = $beforeRetry($e, ++$attempts);
        }
        if ($shouldRetry) {
            goto tryCode;
        }
        throw $e;
    }
}
 
$id = 42;
try {
    $result = retry(3, function () use ($id) {
        throw new AnotherRecoverableException("FAILED getting ID #{$id}");
    }, function ($e, $attempt) use ($id) {
        if (42 === $e->getCode()) {
            return false;
        }
        echo "Failed getting ID #{$id} on try #{$attempt}. Retrying...";
        sleep(1);
        return true;
    }, [RecoverableException::class, AnotherRecoverableException::class]);
} catch (RecoverableException | AnotherRecoverableException $e) {
    echo $e->getMessage();
} catch (NonRecoverableException $e) {
    echo $e->getMessage();
}

When comparing the readability of those two code snippets one can see the amount of cognitive overhead that the retry feature can reduce.

It's important not to confuse the use of this feature, with developers just not using enough methods. More and more and more methods is quite popular with the OOP mindset, but 1) more methods are not in fact always a superior alternative to retry, and 2) PHP is not just an OOP language.

Retry in the wild

A retry feature is also not unheard of in other languages.

Other languages seem to lack retry logic directly, but Google is full of people trying to work out how to do it with a whole range of complex approaches. We'd make a lot of lives easier if the keyword existed, instead of forcing people to loop and break and count and recurse and... goto.

Use Cases

There are myriad use cases in which retry could be useful.

A) A popular use case would be with temporary failed TCP/IP connections which are extremely common. If a server is not available, simply sleep a second and retry. Maybe do this five times until it works.

B) Attempting to make an OAuth 2.0-based API request, getting a 401 due to an expired token, refreshing that token then retrying to original request.

C) Temporarily locked I/O.

D) Find/Create logic. This is used in the Rails world, when their ActiveRecord “ORM” does SELECT...INSERT. There is a chance that a race condition will flair up between the SELECT returning 0 rows and the INSERT happening, leading to a unique index exception being thrown. Catch that specific exception and fire out a retry, there is no way it can keep happening unless you are deleting it super quickly and timing those race conditions perfectly.

Ideally apps would attempt to retry recoverable failures far more often than we see in the wild, but it's currently not trivial to do so. This scares off a lot of developers from adding the retry logic, and if it were easier, we may well see more stable apps as more developers adopt the easy-to-use syntax.

Backward Incompatible Changes

This RFC would not introduce any BC breaks.

Proposed PHP Version

Next PHP 7.3.

Proposed Voting Choices

Requires a 2/3 majority.

Add block-level retry

try {
    somethingSketchy();
} retry 3 (RecoverableException $e, $attempt) {
    sleep(1);
} catch (RecoverableException $e) {
    echo $e->getMessage();
}
Add block-level retry?
Real name Yes No
Final result: 0 0
This poll has been closed.

Add retry keyword

try {
    somethingSketchy();
} catch (RecoverableException $e)
    retry;
}
Add retry keyword?
Real name Yes No
Final result: 0 0
This poll has been closed.

Patches and Tests

There are a few slightly outdated WIP implementations of retry:

Credits

Phil Sturgeon put me up to it. Blame him.

rfc/retry-keyword.txt · Last modified: 2017/09/22 13:28 by 127.0.0.1