PHP RFC: Retry functionality

• Version: 1.0
• Date: 2016-06-19
• Author: Sammy Kaye Powers, me@sammyk.me
• Status: Under Discussion

The retry keyword will make it easier to re-execute code blocks that failed due to recoverable errors by jumping to the top of the try block.

PHP is primarily used for web apps, and many times these apps need to communicate with third-party services or data stores that can temporality fail with recoverable errors. Failures typically throw exceptions which can be captured in a try/catch/finally block. Once a recoverable error reaches a catch block, it's not always trivial in user-land to retry the try block.

This RFC proposes adding two implementations of the retry keyword to a try/catch/finally block with a separate vote for each implementation.

The retry block allows a developer to:

• Retry the try block n times or forever (in this example 3 times)
• Execute arbitrary code before each retry (to sleep, log, check exception code, etc)
• Access to the number of times the try block has been executed (with $attempt)
• Use the break keyword to break out of retry attempts for reasons other than 1) the exception not being thrown or 2) running out of retry attempts; (more on this later)

try {
    somethingSketchy();
} retry 3 (RecoverableException $e, $attempt) {
    echo "Failed doing sketchy thing on try #{$attempt}. Retrying...";
    sleep(1);
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

The retry keyword implementation offers a very light-weight syntax for retrying the try block.

try {
    somethingSketchy();
} catch (RecoverableException $e)
    retry; // Go to top of try block
}

This new keyword makes what is currently somewhat of a chore into a quick and easy activity.

Currently in order to retry a block of code that failed with a recoverable error in user-land, the developer needs to write quite a bit of bootstrap code. It forces the developer to make extra methods or functions, which are not always necessary or even helpful, and are just written to avoid copy/paste.

Currently

function uploadImage($path) {
  $attempt = function() use ($path) {
    $obj = $this->s3->bucket('bucket-name')->object('key');
    $obj->upload_file($path);
  };
 
  try {
    $attempt();
  } catch (AWS\S3\UploadException $e)
    $attempt();
  }
}

Proposed

function uploadImage($path) {
    try {
        $obj = $this->s3->bucket('bucket-name')->object('key');
        $obj->upload_file($path);
    } catch (AWS\S3\UploadException $e)
        retry;
    }
}

There are currently a few ways to implement a feature that will retry a failed block of code n times.

function myRetryFunction($maxTries) {
    try {
        somethingSketchy();
    } catch (RecoverableException $e) {
        if ($maxTries === 0) {
            die('Tried a bunch but failed.');
        }
        myRetryFunction(--$maxTries);
    }
}
 
myRetryFunction(5);

Wrapping recoverable failures in functions/closures is less than ideal. Let's try another method.

$maxTries = 5;
 
for ($x=0; $x<=$maxTries; $x++) {
    try {
        somethingSketchy();
        break;
    } catch (RecoverableException $e) {
        die('Tried a bunch but failed.');
    }
}

Wrapping the recoverable code in for loops is also less than ideal as it make the code less readable.

$maxTries = 5;
 
retryTheThing:
try {
    somethingSketchy();
} catch (RecoverableException $e) {
    if (--$maxTries > 0) {
        goto retryTheThing;
    }
    die('Tried a bunch but failed.');
}

While this is arguably the cleanest option, it still requires the developer to define and manage a label which over several refactors might gradually move further away from the top of the try line. This implementation also makes it easy to accidentally execute any code after the label & before the try for each retry which is not entirely obvious at first glance.

$maxTries = 5;
 
retryTheThing:
 
someCodeIDoNotWantToRetry();
 
try {
    somethingSketchy();
} catch (RecoverableException $e) {
    if (--$maxTries > 0) {
        goto retryTheThing;
    }
    die('Tried a bunch but failed.');
}

Using the retry keyword implementation allows a developer to use her own method of tracking the number of attempts as well as a way to execute any arbitrary code before trying again.

const MAX_TRIES = 5;
$attempt = 0;
try {
    somethingSketchy();
} catch (RecoverableException $e) {
    if (++$attempt < MAX_TRIES) {
        sleep(1);
        // And log stuff maybe
        retry;
    }
    die('Tried a bunch but failed.');
}

Alternatively using the block-level implementation of retry keeps the developer from having to write her own boilerplate to both 1) track the number of attempts and 2) execute arbitrary code before retrying.

try {
    somethingSketchy();
} retry 5 (RecoverableException $e, $attempt) {
    sleep(1);
    // And maybe log stuff when $attempt > 3 or something
} catch (RecoverableException $e) {
    die('Tried a bunch but failed.');
}

The code becomes quite a bit more readable because it removes the nested structures one's eyes has to jump through while scanning the code.

Both the retry keyword and block-level implementations really do simplify quite a few workflows, that many PHP developers have just learned to accept, and could be improved greatly.

As the retry block is basically a fancy catch block, it also supports catching multiple recoverable exceptions.

try {
    throw new RecoverableException("FAILED");
} retry 3 (RecoverableException | AnotherRecoverableException $e, $attempt) {
    echo "Failed on try #{$attempt}. Retrying...";
    sleep(1);
} catch (RecoverableException $e) {
    echo $e->getMessage();
}
 
// AnotherRecoverableException bubbles up after 3 retries

When a retry block exists on a try/catch/finally structure, the catch block is optional allowing the exception to bubble up for another part of the application to catch it.

try {
    throw new RecoverableException("FAILED");
} retry 10 (RecoverableException $e, $attempt) {
    echo "Retrying...";
}
 
// After 10 times RecoverableException is still thrown & uncaught

The number of retries can be defined with an int literal or a constant. If omitted, retry will assume infinite retries.

try {
    throw new RecoverableException("FAILED");
} retry (RecoverableException $e, $attempt) {
    echo "Retrying forever...";
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

Alternatively the INF math constant can be used to specify infine retries.

try {
    throw new RecoverableException("FAILED");
} retry INF (RecoverableException $e, $attempt) {
    echo "Retrying forever...";
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

It is sometimes necessary to have some logic that would abort any more retry attempts, like in the case of retrying forever. That can be done using the break keyword.

try {
    throw new RecoverableException("FAILED");
} retry INF (RecoverableException $e, $attempt) {
    if (42 === $e->getCode()) {
        break;
    }
    echo "Retrying forever...";
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

Below is a full example that illustrates the full potential of the retry block and all of its features.

class RecoverableException extends Exception {}
class AnotherRecoverableException extends Exception {}
class NonRecoverableException extends Exception {}
 
$id = 42;
try {
    throw new RecoverableException("FAILED getting ID #{$id}");
} retry 3 (RecoverableException | AnotherRecoverableException $e, $attempt) {
    if (42 === $e->getCode()) {
        break;
    }
    echo "Failed getting ID #{$id} on try #{$attempt}. Retrying...";
    sleep(1);
} catch (RecoverableException | AnotherRecoverableException $e) {
    echo $e->getMessage();
} catch (NonRecoverableException $e) {
    echo $e->getMessage();
}

In order to illustrate just how much boilerplate the retry block removes, check out a full userland implementation that covers all the features of the example above with a fully-featured retry() function.

class RecoverableException extends Exception {}
class AnotherRecoverableException extends Exception {}
class NonRecoverableException extends Exception {}
 
function retry(int $retryCount, callable $tryThis, callable $beforeRetry = null, array $targetExceptions = ['Exception'])
{
    $attempts = 0;
    tryCode:
    try {
        return $tryThis();
    } catch (\Throwable $e) {
        $isTargetException = false;
        foreach ($targetExceptions as $targetException) {
            if ($e instanceof $targetException) {
                $isTargetException = true;
                break;
            }
        }
        if (!$retryCount || !$isTargetException) {
            throw $e;
        }
        $retryCount--;
        $shouldRetry = true;
        if ($beforeRetry) {
            $shouldRetry = $beforeRetry($e, ++$attempts);
        }
        if ($shouldRetry) {
            goto tryCode;
        }
        throw $e;
    }
}
 
$id = 42;
try {
    $result = retry(3, function () use ($id) {
        throw new AnotherRecoverableException("FAILED getting ID #{$id}");
    }, function ($e, $attempt) use ($id) {
        if (42 === $e->getCode()) {
            return false;
        }
        echo "Failed getting ID #{$id} on try #{$attempt}. Retrying...";
        sleep(1);
        return true;
    }, [RecoverableException::class, AnotherRecoverableException::class]);
} catch (RecoverableException | AnotherRecoverableException $e) {
    echo $e->getMessage();
} catch (NonRecoverableException $e) {
    echo $e->getMessage();
}

When comparing the readability of those two code snippets one can see the amount of cognitive overhead that the retry feature can reduce.

It's important not to confuse the use of this feature, with developers just not using enough methods. More and more and more methods is quite popular with the OOP mindset, but 1) more methods are not in fact always a superior alternative to retry, and 2) PHP is not just an OOP language.

A retry feature is also not unheard of in other languages.

• Ruby has the retry keyword, highlighted nicely in Bastards Book of Ruby: Exception and Error Handling
• Spring does some really weird stuff to make retry possible
• Python third party library offers annotations to allow methods retry

Other languages seem to lack retry logic directly, but Google is full of people trying to work out how to do it with a whole range of complex approaches. We'd make a lot of lives easier if the keyword existed, instead of forcing people to loop and break and count and recurse and... goto.

There are myriad use cases in which retry could be useful.

A) A popular use case would be with temporary failed TCP/IP connections which are extremely common. If a server is not available, simply sleep a second and retry. Maybe do this five times until it works.

B) Attempting to make an OAuth 2.0-based API request, getting a 401 due to an expired token, refreshing that token then retrying to original request.

C) Temporarily locked I/O.

D) Find/Create logic. This is used in the Rails world, when their ActiveRecord “ORM” does SELECT...INSERT. There is a chance that a race condition will flair up between the SELECT returning 0 rows and the INSERT happening, leading to a unique index exception being thrown. Catch that specific exception and fire out a retry, there is no way it can keep happening unless you are deleting it super quickly and timing those race conditions perfectly.

Ideally apps would attempt to retry recoverable failures far more often than we see in the wild, but it's currently not trivial to do so. This scares off a lot of developers from adding the retry logic, and if it were easier, we may well see more stable apps as more developers adopt the easy-to-use syntax.

This RFC would not introduce any BC breaks.

Next PHP 7.3.

Requires a 2/3 majority.

try {
    somethingSketchy();
} retry 3 (RecoverableException $e, $attempt) {
    sleep(1);
} catch (RecoverableException $e) {
    echo $e->getMessage();
}

try {
    somethingSketchy();
} catch (RecoverableException $e)
    retry;
}

There are a few slightly outdated WIP implementations of retry:

• WIP patch: retry block
• WIP patch: retry keyword

Phil Sturgeon put me up to it. Blame him.