System design glossary

How long one request takes from start to finish

Measured here at the client, so it includes every hop, every queue and every retry along the way. It is the number your users actually feel.

See also: p50, p99, Queue time

A ranking, not an average

Sort every request by how long it took. The 50th percentile is the one in the middle, the 99th is near the slow end. Percentiles are used instead of averages because one very slow request barely moves an average but is a real person waiting.

See also: p50, p95, p99

The typical request: half were faster, half slower

Also called the median. It tells you what a normal experience looks like, but it deliberately ignores the worst half, so a healthy p50 can hide serious problems.

See also: p99, Percentile

The slowest 1 in 20 requests

Sits between the typical case and the worst case. Useful for spotting trouble building before it shows up in p99.

See also: p50, p99

The slowest 1 in 100 requests

The number to watch. It is where overload shows up first: p50 can look perfectly fine while p99 climbs, because queues punish the unlucky requests long before they slow down the typical one. On a busy page made of many requests, almost every user hits their p99 somewhere.

See also: p50, Utilisation, Queue time

Time spent waiting for a free slot, not being worked on

A request that takes 200 ms might have been served in 5 ms and queued for 195. Under load, most latency is waiting rather than working, which is why adding capacity helps more than making the work itself faster.

See also: Latency, Utilisation, Backlog

The extra delay when no warm instance is free

The platform must provision and boot an instance before your function even runs. Short keep-warm times and bursty traffic are the recipe for a high cold-start rate.

See also: Serverless function, Latency

Requests finished per second

Work actually completed, as opposed to work arriving. Once a system is saturated, throughput flattens out no matter how much more traffic you send: that flat line is the system telling you its real limit.

See also: RPS, Goodput, Offered

Traffic arriving, whether or not it succeeds

Compare it with goodput. The gap between the two lines is the traffic your system is failing to serve.

See also: Goodput, Dropped

Requests that actually succeeded, per second

The only throughput number that counts, because a fast error is still an error. A system can look busy while its goodput is zero, which is exactly what a retry storm looks like.

See also: Throughput, Offered, Dropped

How many deliveries one publish becomes

Total load downstream is the publish rate multiplied by the fan-out. Adding a subscriber silently adds a whole copy of the traffic, which is the easiest way to grow load without noticing.

See also: Pub/sub topic

A CDN miss travelling to your servers

The origin fetch rate is the load that actually reaches you; offered minus that is what the CDN absorbed. Small drops in hit rate multiply this number quickly.

See also: CDN, Hit rate

One output quality produced from one source file

An encode job produces the whole quality ladder: several output files per input. Storage behind the farm therefore sees the job rate multiplied by the ladder size, and that write amplification, not the upload rate, is what you size the store for.

See also: Transcoder, Object store

Requests that arrived but were never served successfully

The gap between offered and goodput. Some were shed on purpose to protect the system, some timed out, some hit errors. The failure breakdown tells you which.

See also: Shed, Timeout, Error rate

The share of requests that failed

Counts every kind of failure together. Watch it alongside latency: a system that sheds load keeps latency low and error rate high, while one that queues everything does the opposite.

See also: Dropped, Shed, Timeout

Turned away immediately because the queue was full

A deliberate defence, not a malfunction. Refusing a request instantly is far cheaper than accepting it, making it wait, and failing anyway, and it keeps the system responsive for everyone else.

See also: Queue limit, Rate limiter

The caller gave up waiting

The work usually carries on downstream even though nobody is waiting for the answer any more, so the capacity is spent for nothing. That is what makes retry storms so destructive.

See also: Retry, Retry storm

Automatically asking again after a failure

Helpful when failures are rare and random. Harmful when the system is already overloaded, because retries add load exactly when there is least to spare.

See also: Retry storm, Timeout, Circuit breaker

Retries multiplying the overload that caused them

A slow service causes timeouts, timeouts cause retries, retries add load, which makes it slower still. Systems can collapse to zero goodput this way while looking fully busy.

See also: Retry, Timeout, Circuit breaker

A message that failed every delivery attempt

It is parked on a shelf instead of vanishing, so failures stay countable and replayable. A growing dead letter count means something downstream is persistently broken, not just slow.

See also: Retry queue, Redelivery

A failed delivery being tried again

It rises before dead letters do, so it is your early warning. Redeliveries also add load exactly when the downstream is already struggling, which is the retry trade-off in miniature.

See also: Retry, Dead letter

A write acknowledged but not yet saved to the store

Every dirty write is data the caller believes is safe. A crash of the buffer loses all of them at once, after they were confirmed, which is why the count is worth watching.

See also: Write-behind cache

A search answered from the old index

A write only becomes searchable after the refresh delay, so a search inside that window cannot see it. The stale rate rises with the delay and with write volume.

See also: Search index, Stale read

The request had nowhere to go

A component tried to pass work downstream but nothing was wired to it. Usually a gap in the diagram rather than a capacity problem: check the component has an outgoing connection.

See also: Dropped

The request bounced between components too many times

A safety net for loops. If your components are wired in a circle, a request would travel forever, so it is stopped after a fixed number of hops. Seeing this means the topology has a cycle.

See also: No route

Refused by a rate limiter

The limiter had no tokens left, so the request was turned away instantly rather than being allowed to pile up. Deliberate protection, not a malfunction.

See also: Rate limiter, Shed, Burst

Refused by an open circuit breaker

The breaker had already decided the dependency was unhealthy, so it failed immediately without calling it. This is the breaker doing its job: the dependency gets quiet time to recover.

See also: Circuit breaker, Retry storm

The component is down

Something you knocked over deliberately, or a fault that was injected. Requests that were in flight are lost and new ones fail immediately until it comes back.

See also: Dropped, Region

The connection between two components is cut

Both components are running fine, but they cannot reach each other. Network partitions are the failure people forget to plan for, because nothing looks broken from either side on its own.

See also: Crashed, Region

Turned away because every connection slot was held

The gateway was full of live connections, not busy with requests. More slots, shorter connection lifetimes, or fewer connections are the levers that help.

See also: WebSocket gateway, Connection slot

Refused by the gateway’s auth check

The request spent a rate-limit token and then failed authentication. A rising rate here with steady traffic usually means bad credentials in a client, or an attack, rather than overload.

See also: API gateway

Refused because the concurrency pool was occupied

Every allowed call to the dependency was already in flight, usually because that dependency slowed down. This failure is protection: the alternative is queueing behind a sick service.

See also: Bulkhead

Dropped by a load shedder protecting higher priority

The token bucket was into its reserve, so low-priority traffic was refused first. If high-priority traffic is being dropped too, the protection itself is exhausted.

See also: Load shedder, Shed

Lost during a failover between regions

The active region failed and traffic has not landed on the next one yet. Failover takes time, and requests arriving in that gap have nowhere to go. Failover is never truly free.

See also: Region, Crashed

How much of a component’s capacity is in use

The single best early warning. Waiting time rises gently up to about 70%, then sharply, and becomes unbounded as it approaches 100%. A component at 95% is not "nearly fine". It is about to fall over.

See also: Capacity, p99, Queue time

How many requests a component can work on at once

Think of it as the number of checkout tills open. Anything beyond that waits in line. Capacity times (1000 ÷ service time) gives the most requests per second a component can possibly handle.

See also: Service time, Utilisation, Instances

How long the component takes to do the work itself

Excludes any queueing. It is the best case for that component: a request can never be faster than this, and under load it will be much slower.

See also: Capacity, Queue time

Work piled up waiting to be processed

A backlog that keeps growing means arrivals are outpacing the workers, and it will never recover on its own. A backlog that spikes and drains is a system absorbing a burst, which is what queues are for.

See also: Queue limit, Queue time

The longest the waiting line is allowed to get

Once it is full, new requests are shed instead of queued. Setting it is a real trade-off: a long queue absorbs bursts but makes waits longer, while a short one fails fast and stays responsive.

See also: Backlog, Shed

How many copies of a component are running

Scaling out means running more copies. Doubling instances roughly doubles the work you can do, right up until a shared dependency further down becomes the new bottleneck.

See also: Capacity, Autoscaler, Bottleneck

The one component that limits the whole system

A system is only as fast as its narrowest point. Adding capacity anywhere else changes nothing, which is why finding the bottleneck matters more than optimising everything.

See also: Utilisation, Instances

Whether every request costs the same, or some cost far more

A component where every request takes exactly 20 ms behaves very differently from one averaging 20 ms with occasional 200 ms outliers. Unevenness alone makes queues form and pushes up the slowest requests, even with the average unchanged.

See also: Service time, p99, Queue time

How many times more traffic this could take

The ceiling divided by what is actually arriving. Above 1x it is keeping up, below 1x it cannot, and the component with the smallest number is your bottleneck. This is the fastest way to find one without reading a chart.

See also: Bottleneck, Capacity, Utilisation

How much of the traffic reads rather than writes

It decides how much a replica can help. At 95% reads, adding replicas scales almost everything; at 50%, half the traffic still lands on the one primary no matter how many replicas you add.

See also: Read replica, Database

The busyness an autoscaler tries to hold a component at

Set it low and you pay for idle capacity; set it high and there is no slack left to absorb a spike while new capacity boots. Around 60 to 70% is the usual compromise, and that is not a coincidence: past 70% waiting time climbs sharply.

See also: Autoscaler, Utilisation, Warm-up

The independent lanes a log is split into

A consumer group takes one message per partition at a time, so partition count, not consumer count, caps how fast a group can drain. Adding consumers beyond the partition count does nothing at all.

See also: Stream broker, Consumer lag, Shard

How far behind the head of the log a group is

It grows while a group’s consumers are slower than the producers and drains when they catch up. Lag that keeps growing will eventually cross the retention limit, and then messages are lost to that group.

See also: Stream broker, Retention, Backlog

How far back a log keeps messages

A consumer group that falls further behind than the retention limit skips ahead, and the skipped messages are gone for it. The producer never notices; only the lagging group pays.

See also: Consumer lag, Stream broker

The spendable budget inside a rate limiter’s bucket

One token is spent per admitted request and the bucket refills at the configured rate. An empty bucket is a limiter working hard: everything beyond the refill rate is being refused.

See also: Rate limiter, Burst

Room for one held connection on a gateway

A connection occupies its slot for its whole lifetime, not just while data flows. Held connections settle at connect rate times lifetime, so lifetime matters as much as traffic.

See also: WebSocket gateway, Capacity

A query that scans many points at once

It costs hundreds of appends worth of work, so a small share of range queries can dominate a store sized for pure ingest. Watch the mix, not just the total rate.

See also: Time-series database

How many hops a graph query walks

Every extra hop multiplies the edges visited by about three. Depth 3 costs roughly nine times depth 1, which is why deep traversals get expensive so fast.

See also: Graph database

How close to perfect a similarity search must be

Cost scales with one over one minus recall: the last percent costs more than everything before it. Trading a little recall buys a lot of latency back.

See also: Vector database

The fraction of requests answered at the edge

It is a property of your code, not your traffic: raising it is engineering work, not luck. Everything outside the share pays the full trip to the origin.

See also: Edge compute, CDN

Writes waiting for other writes to finish

A database protects shared data with locks, so each write entering service waits on the writes already holding them. The wait grows with concurrent writers, not with fleet size; adding instances speeds up reads and does nothing here. When lock wait dominates, the fixes are fewer or faster writes, read replicas, or sharding the data itself.

See also: Database, Read replica, Shard

A blob store limit that applies per prefix, not store-wide

Object storage partitions by key prefix, and each prefix sustains only so many requests per second. A hot prefix gets refused with a slowdown while the rest of the store idles, so the fix is spreading your keys, not buying capacity. This is the S3 503 SlowDown.

See also: Object store, Throttled

The hard per-request execution limit of an edge runtime

Edge platforms kill a request whose code runs past the budget and your traffic falls through to the origin anyway. Push heavier logic outward and the budget starts eating the very share you were trying to answer at the edge, so the measured at-edge share is the honest number, not the share your code claims.

See also: Edge compute, Edge share

Spreads incoming requests across several servers

Lets you scale out by adding servers. It does not create capacity, though. If every server talks to the same database, that database is still the bottleneck.

See also: Bottleneck, Instances

Remembers recent answers so the work is not repeated

A hit is answered immediately; a miss costs a full trip to whatever is behind it. Hit rate is the whole story: at 90%, the database sees only a tenth of the traffic.

See also: Hit rate, CDN

The share of requests a cache can answer by itself

Small changes here have outsized effects. Going from 90% to 80% hit rate does not add 10% more database load. It doubles it.

See also: Cache, CDN

A cache close to your users, in front of everything

Serves the common requests before they ever reach your servers. The cheapest traffic to handle is traffic your system never sees.

See also: Cache, Hit rate

Accepts work now so it can be done later

The caller gets an answer immediately while the work is buffered for a worker. This is how a system absorbs a spike instead of collapsing under it, but only if the workers eventually catch up.

See also: Worker, Backlog, Asynchronous

Answering before the work is finished

Keeps the user waiting only for an acknowledgement rather than the whole job. The trade-off is that the answer is a promise, and the work can still fail after you have replied.

See also: Queue, Worker

Processes queued work in the background

Workers set the drain rate. If they are slower than arrivals, the backlog grows without limit no matter how big the queue is.

See also: Queue, Backlog

Caps how many requests are let through per second

Protects what is behind it by refusing excess traffic cheaply and immediately. Errors go up, but latency stays flat and the service survives, which is often the better failure.

See also: Shed, Burst

How much traffic can arrive at once before limiting starts

Lets a rate limiter absorb a short spike without punishing normal, uneven traffic. Real traffic arrives in clumps, not at a steady drip.

See also: Rate limiter

Stops calling a dependency that is already failing

When errors pass a threshold it opens and fails instantly without calling downstream, giving the struggling service room to recover. It then lets a few test requests through to check whether it is safe to resume.

See also: Retry storm, Timeout

Adds and removes instances to track a utilisation target

Watches a component and scales it to keep utilisation near a setpoint. New instances take time to start, so capacity always lags load. That is why a sharp spike still hurts even with autoscaling on.

See also: Instances, Utilisation, Warm-up

The delay before a new instance can serve traffic

Machines boot, caches fill, connections open. Scaling up is never instant, and this gap is when the system is most likely to fail.

See also: Autoscaler, Instances

A slice of the data, held on its own partition

Splitting data across shards multiplies capacity, as long as traffic spreads evenly. One popular key breaks that assumption: a single shard saturates while the rest sit idle, and the total looks healthy while users see failures.

See also: Hot key, Bottleneck

One piece of data that far more requests want

Defeats sharding, because a single key can only live on one shard. Average utilisation stays low while that one shard is on fire.

See also: Shard

A copy of the database that serves reads only

Most workloads read far more than they write, so replicas scale the common case cheaply. The cost is that a replica can be slightly behind the primary.

See also: Replication lag, Stale read

How far behind the primary a replica is

A write takes time to reach the replicas. Read during that window and you get the old value, which is why a user can save a change and then not see it.

See also: Read replica, Stale read

Getting an out-of-date answer from a replica

The visible cost of scaling reads. Usually harmless, occasionally serious. Showing a stale balance or an unsaved edit is how this becomes a real bug.

See also: Replication lag, Read replica

A separate location your system runs in

Running in more than one region means one can fail without taking everything down. Failover is not instant, though, and requests arriving mid-switch are still lost.

See also: Bottleneck

Where the traffic comes from

Stands in for your users. It is also where end-to-end latency is measured, so the numbers it reports are what a real person would actually experience, including every hop and retry along the way.

See also: RPS, Latency

A server that handles requests and calls its dependencies

The ordinary building block of a system. It waits for every dependency it calls before it can answer, so it is only ever as fast as its slowest one.

See also: Capacity, Bottleneck, Instances

Stores the data. Usually the first thing to saturate

Databases are harder to scale than servers: you can add application servers freely, but they all tend to talk to the same database, which is why it so often turns out to be the bottleneck.

See also: Bottleneck, Read replica, Shard

Blob storage: high flat latency, very wide pool

Stores files and blobs rather than rows. Every request pays a high flat latency, but the pool is so wide that saturating it takes deliberate effort. Keep user-facing reads behind a cache and it is the cheapest storage you have.

See also: Cache, Database

Search cluster where writes pay extra and appear late

Searches are cheap, but each write pays an indexing surcharge and only becomes searchable after a refresh delay. Bulk writes can starve the queries that were never the problem, and a search inside the delay window returns the old result.

See also: Stale search, Database

Built for appends; range queries cost hundreds of them

It swallows appends by the thousand, while a range query scans and aggregates many points at once. A few percent of range queries can dominate the whole store, which is why metrics do not live in your main database.

See also: Range query, Database

Stores relationships; deeper queries multiply the work

Each extra hop of traversal multiplies the edges visited by about three. A friends-of-friends query costs several times a direct lookup, and one more hop multiplies it again, so depth is the knob to watch.

See also: Traversal depth, Database

The archive tier: cheap to keep, seconds to read

Retrieval takes seconds per request, and that is its whole identity. It is fine for a trickle of restores fed from a queue and hopeless for anything a user is actively waiting on.

See also: Object store, Queue

Similarity search over embeddings

Finds the items most similar to a query rather than exact matches. Cost grows with the size of the index and explodes as recall approaches perfect, so the recall setting is really a latency setting read backwards.

See also: Recall, Database

A partitioned, replayable log between services

Producers are acknowledged instantly and each consumer group reads at its own pace, so a slow consumer builds lag instead of slowing anyone else. Partition count, not consumer count, caps how fast a group can drain.

See also: Consumer lag, Retention, Queue

One publish becomes one delivery per subscriber

Total load is multiplied by the number of subscribers whether you noticed or not. A slow subscriber fails on its own without delaying the others, which is the point of the decoupling.

See also: Fan-out, Queue

Holds long-lived connections; the scarce thing is slots

Capacity here is concurrent connections held open, not requests per second. At 30 new connections a second, each held for 8 seconds, 240 slots are simply occupied, which is why chat systems run out of sockets long before CPU.

See also: Connection slot, Capacity

The front door: authenticates, rate limits, and routes

Three kinds of refusal happen here so the backends only ever see traffic worth serving. Watch the split between admitted, throttled, and failed auth to tell an attack from an outage.

See also: Rate limiter, Tokens

A proxy beside one service: a tax paid for resilience

It charges its own service time on every request and pays you back with retries, deadlines, and ejecting an upstream that keeps failing. Put one at every hop and the taxes stack; that is a service mesh.

See also: Outlier ejection, Circuit breaker

Scales instantly to a cap, with a cold-start penalty

There is no queue: past the concurrency cap, requests are refused outright. A request that finds no warm instance pays the cold start, so idle periods and sudden bursts are exactly when it is slowest.

See also: Cold start, Capacity

Fires on a schedule and dumps its whole batch at once

Between firings it does nothing at all. The database that handles the daytime load falls over at midnight not because traffic grew, but because the whole batch arrived in the same instant.

See also: Backlog, Queue

Caps calls in flight to the dependency behind it

When that dependency slows down, the pool fills within one round trip and the excess fails fast here instead of queueing behind a sick service. Watching the pool fill is your earliest warning of a slow dependency.

See also: Circuit breaker, Timeout

Delivers messages, retries failures, keeps the dead

It acknowledges the sender instantly, delivers downstream itself, and redelivers failures. Messages that fail every attempt land on the dead letter shelf: counted, not vanished.

See also: Dead letter, Redelivery, Queue

A batch farm where each job takes seconds

Throughput is boxes times jobs-per-box divided by job time, so size it against arrivals. Undersize it by even 10% and the backlog grows forever, because the deficit is structural rather than a burst.

See also: Worker, Backlog

A small function running close to the user

The share of requests it can fully answer never touches your origin; the rest pass through and pay the full path. Unlike a cache hit rate, that share is a property of your code, not your traffic.

See also: Edge share, CDN

Acknowledges writes from memory, flushes them later

The caller sees a one-millisecond write while the store sees the same load smoothed out. Crash it, and every write still in the buffer is lost after being confirmed, which is the trade you are making.

See also: Dirty write, Cache

Refuses traffic by priority when tokens run low

Low-priority traffic must leave a reserve of tokens untouched, so under saturation it is dropped first while the traffic that matters keeps being admitted. Degradation as a policy rather than an accident.

See also: Rate limiter, Tokens, Shed

A proxy dropping an upstream that keeps failing

After enough consecutive failures the proxy stops calling the upstream for a while and fails fast instead. Simpler than a windowed error rate on purpose; this is how sidecar health checking actually works.

See also: Sidecar proxy, Circuit breaker

Requests per second

How much traffic is arriving. Every system has a point where more requests per second stops meaning more work done and starts meaning longer queues, so this is the dial you turn to find that point.

See also: Throughput, Goodput

Milliseconds, one thousandth of a second

The unit for how long something took. For scale: 1 ms is imperceptible, 100 ms feels instant, 1000 ms (one second) is a noticeable wait, and past about 3000 ms most people give up.

See also: Latency