Fairness in multi-tenant systems

today we’re looking at “Fairness in multi-tenant systems” by @dyanacek from the Amazon Builder’s library

it’s available at https://aws.amazon.com/builders-library/fairness-in-multi-tenant-systems/

☁️ cloud devops @awscloud

@marknca tweeted at 10-Nov-2021, 14:00

you can find this thread unrolled at https://t.co/henuJwkAKm

…and yesterday’s thread on “Making retries safe with idempotent APIs” at https://markn.ca/2021/making-retries-safe-with-idempotent-apis/

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

I’ve helped a lot of teams build out multi-tenant systems. it’s a fascinating problem space. you’re trying to find the right balance between isolation & economy of scale

it’s not easy

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

before we dive into this , let me just add the “SaaS Lens” from the Well-Architected Framework to your reading list as well

this Lens explains a few different multi-tenancy models

it’s at https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/saas-lens.html

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

ok, back on track, this paper from @dyanacek looks at how Amazon manages APIs requests in order to avoid overload

what does that have to do with multi-tenancy? you ask (you did ask, right?)

well…

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

it’s economy of scale bit. you optimize resource usage (& thus spend) by making sure that you’re streamlining the use of your service

@dyanacek takes it a step further in this & shows how you can use this to reduce pressures on your systems as well

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

first up, is the case for multitenancy. there’s a few well structured arguments in the paper, but basically it’s all about resource optimization

idle = bad (generally)

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

one of the biggest downfalls is rightfully called out by @dyanacek as well; tenants impacting each other

if we’re each sharing a resource, what if I grab more than my fair share?

this is something we need to solve for…

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

which is why this is a great one, it transitions out of the example highlighting the +/- and into “fairness”

summed up as, “every client in a multi-tenant system is provided with a single-tenant experience” << or at least they SHOULD be

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

one area that gets more attention is the case when demand is outpacing supply (which should increasing as scaling catches up). what do you then?

the author introduces the concept of “load shedding”

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

…a/k/a saying quickly with little resource cost. this can relieve the pressure on the backend & clients can easily retry using the techniques we highlighted in previous threads

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

load shedding isn’t enough to solve the issue of fairness, that’s where “rate limiting” comes into play. you can use this technique to “shape unplanned increases in traffic”

the paper details how & what to look out for when adding this pattern to your design

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

quotas go hand-in-hand with rate limited & the author spends quite a bit of time of them as well (rightfully so!)

there’s a fine art to implementing a quota system & the does a good job of providing a sold overview

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

all of these techniques are known as “admission control systems”

the paper highlights how Amazon uses these (and why, and when), showing a few different models and patterns that could help you out

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

finally the paper dives into architecture design patterns that can help reduce your need for these techniques.

nothing is every perfect, so knowing all of the tools ⚒️ at your disposal is critical

☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

all-in-all, this is a FANTASTIC paper by @dyanacek highlight years of hard won learning.

it builds nicely on the other two papers I’ve mini-reviewed this week. I really think you should add this one to your reading list!

/ ☁️ cloud devops

@marknca tweeted at 10-Nov-2021, 14:00

Start