Timeouts, retries, and backoff with jitter

2 minute read | Last updated 2-Aug-2022 | An icon depicting a retail tag with a heart for 'favourite' AWS Cloud

The Amazon Builder’s Library is a great set of deep dive papers into the challenges with modern systems. This post highlights some of the challenges in dealing with failure at scale.

“Timeouts, retries, and backoff with jitter, looks at various types of failures and their potential impact on both your service and it’s consumers.

I call out a few more details in the Twitter thread below…

Tweet 1/6 Next tweet

last week, I looked at a number of @awscloud white papers. this week, I’ll be diving into the Amazon Builder’s LIbrary

first up: “Timeouts, retries, and backoff with jitter”, by @MarcJBrooker, https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/

☁️ #cloud #devops

marknca @marknca tweeted at 08-Nov-2021, 14:13

Tweet 2/6 Next tweet Start

this is available unrolled at https://t.co/dqEcUqffGB

Friday’s thread is up at https://markn.ca/2021/machine-learning-best-practices-for-public-sector-organizations/

☁️ #cloud #devops

marknca @marknca tweeted at 08-Nov-2021, 14:13

Tweet 3/6 Next tweet Start

tldr: happens. plan for it. make sure to keep your customers perspective in mind

☁️ #cloud #devops

marknca @marknca tweeted at 08-Nov-2021, 14:13

Tweet 4/6 Next tweet Start

. @MarcJBrooker calls out the 3 primary techniques that Amazon uses for handling failures:

1. timeouts
2. retries
3. backoff

the rest of the details the how, why, & when or each of these techniques

☁️ #cloud #devops

marknca @marknca tweeted at 08-Nov-2021, 14:13

Tweet 5/6 Next tweet Start

the discussion of timeouts is of particular note. looking at the impacts of server vs. client timeouts & how to manage both without causing a flood of retries is really interesting

☁️ #cloud #devops

marknca @marknca tweeted at 08-Nov-2021, 14:13

Tweet 6/6 Next tweet Start

retries is where the paper goes next. the author calls out that retries as “selfish” & cause failures to amplify. this is why Amazon combines them with the backoff technique to avoid a flood of requests

☁️ #cloud #devops

marknca @marknca tweeted at 08-Nov-2021, 14:13

Start