Your average latency is lying to you: p90, p95 and p99 explained

Hello, I’m Maneshwar, and I’m building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Your latency dashboard says 42ms.

Your support inbox says “the app is unusably slow”.

Both of these are true at the same time, and the reason is that you are looking at an average.

Averages are where latency problems go to hide.

They are the single most comfortable lie in observability, because they always look fine right up until a customer churns.

So let’s talk about the numbers that actually tell you something: p90, p95, and p99.

What an average actually hides

Say 10,000 requests hit your service. Most of them are quick.

A handful take three seconds because a cache missed, a GC paused, a connection pool ran dry, or a neighbour on the same box decided to compile something.

Take the mean of all that and the slow ones get diluted into nothing.

Ten thousand fast requests will happily absorb fifty terrible ones and hand you back a number that looks great on a slide.

Here is the framing that fixed this for me.

Nobody experiences your average.

Every user experiences exactly one request at a time.

They do not feel the mean of your traffic, they feel the single call they are currently waiting on.

So the question worth asking is not “how fast is my service typically”. It is “how bad is it for the unluckiest people, and how many of them are there?”

That question has a name. It is a percentile.

What p90, p95 and p99 actually mean

A percentile is a promise about a share of your traffic.

p90 = 400ms means 90 out of every 100 requests finished in under 400ms. Ten did not.

p95 = 900ms means 95 finished under 900ms. Five did not.

p99 = 3,100ms means 99 finished under 3,100ms. One did not.

Notice the shape of that. As you climb from p90 to p99, you are not describing more of your users, you are describing worse experiences for fewer of them. Each step up the ladder zooms further into the tail.

That is exactly why you want all three rather than one favourite. p90 tells you what normal feels like. p99 tells you what your worst hour looks like on a good day.

And the maths behind it is almost insultingly simple.

Sort, index, read

There is no clever statistics here. To find a percentile you sort your response times and read the value at the right position.

With 100 requests recorded:

latencies.sort()                        # ascending, always

p95 = latencies[int(100 * 0.95) - 1]    # index 95  ->  95ms
p99 = latencies[int(100 * 0.99) - 1]    # index 99  ->  99ms

That is genuinely it. Sort the list, multiply the count by the percentile, read the value sitting at that index.

Which means a percentile is not a measurement of anything. It is a position in a sorted list. Hold onto that, because it is about to matter a lot.

Quick tangent: why there is no p100

You will notice nobody ever quotes a p100. There is a nice reason for that.

A percentile answers “what fraction of the values are below this one?” So p99 means “this value beats 99% of the others”.

For a value to be p100, it would have to be greater than 100% of the values, including itself. Which is not a thing.

Put it in a room. If you are the tallest of ten people, you are taller than nine of them, not ten. You are not taller than yourself.

Same with requests. The slowest request is not slower than the slowest request.

If you genuinely need to see the worst case, the metric you want is just called max. Say max. And if you need to go deeper into the tail without falling off the end of it, that is what p99.9 and p99.99 are for, which is where the big shops actually live.

Median vs p99, and why you need both

The median is p50. Sort your data, take the middle value. Half your requests were faster, half were slower.

It is a genuinely useful number, and it is much more honest than the mean, because it is not dragged around by outliers. One 30 second request moves your average. It barely nudges your median.

But that immunity to outliers is exactly the problem.

The median is deliberately blind to the tail. It is designed to ignore extremes. Which is wonderful if you are asking “what is a typical request”, and useless if you are asking “how badly are we failing anyone”.

So they answer different questions, and you want both on the same graph:

flowchart TD
    A["A latency question"] --> B{"Are you asking about
the typical user, or the
worst-off user?"}
    B -->|typical| C["median / p50
ignores outliers by design"]
    B -->|worst-off| D{"How deep into
the tail?"}
    D -->|"most users"| E["p90"]
    D -->|"nearly all"| F["p95 / p99"]
    D -->|"at real scale"| G["p99.9 / p99.99"]
    C --> H["central tendency"]
    E --> I["worst-case experience"]
    F --> I
    G --> I

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef mid      fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef tail     fill:#ff9a5c,stroke:#c25f22,color:#1a1a1a
    classDef out      fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class B,D decision
    class A start
    class C,E mid
    class F,G tail
    class H,I out

A gap between your p50 and your p99 is not noise. It is a measurement of how inconsistent your service is, and inconsistency is what users actually perceive as unreliability.

A service that always takes 300ms feels more trustworthy than one that usually takes 40ms and sometimes takes four seconds. Same average, completely different product.

The part that changes how you think: tail amplification

Here is where p99 stops being an academic nicety and starts being the whole ballgame.

Your service is 99% fast. Excellent. Now, how many services does one page load actually touch?

Because a modern page is not one request. It is a fan-out. The frontend calls the orchestrator, which calls auth, profile, feed, recommendations, pricing, inventory, notifications, and a couple of third parties who have never heard of your SLA.

And the page is only as fast as its slowest call.

Run the numbers, because they are genuinely startling.

If each call has a 1% chance of being a p99 straggler, the chance a page avoids all of them is 0.99^n:

1 call    ->  1% chance of a slow page
10 calls  ->  9.6%
100 calls ->  63%
500 calls ->  99%

At 100 backend calls per page, most of your page loads contain a p99 event. The one-in-a-hundred thing is now the common case. You did not get slower, you just rolled the dice more times.

This is the core of Dean and Barroso’s The Tail at Scale, which is the paper to read if this section grabbed you.

The takeaway I keep coming back to: at scale, your p99 is what a normal Tuesday feels like to somebody. Not an edge case. Not a rare event. Just Tuesday, for a real and growing number of people.

Which means “we only fail 1% of requests” quietly becomes “we fail most page loads” the moment your architecture gets interesting.

The mistake almost everyone makes

Right, back to that line from earlier. A percentile is a position in a sorted list, not a measurement.

Now consider what most dashboards do. You have three hosts, each reporting its own p99. The dashboard shows one line for the service, so it averages them.

You cannot average percentiles. The result is not a worse p99, it is not a p99 at all. It is a number with no relationship to any request that ever happened.

Think about why. Host C’s p99 of 400ms came from its sorted list. Host A’s 100ms came from a different list.

Averaging two positions in two different lists gives you a position in a list that does not exist anywhere in your infrastructure.

The same trap applies across time. Averaging this minute’s p99 with last minute’s p99 to get an hourly p99 is the same mistake wearing a different hat.

The fix is to stop shipping summaries and start shipping distributions. Export histogram buckets, merge the buckets, and compute the percentile once from the merged distribution.

This is exactly why Prometheus histograms exist and why histogram_quantile() takes buckets rather than pre-computed quantiles.

If your metrics pipeline reports a single p99 number per host, you do not have a p99. You have a rumour.

Putting it in an SLA without regretting it

Percentiles are how latency promises get written, because they are the only latency numbers that are actually falsifiable.

“Our API is fast” is marketing. “99% of requests complete in under 300ms, measured over a rolling 30 day window” is something you can be held to, and something you can alert on.

Three things worth getting right when you write one.

Name the percentile and the window together. A p99 over a minute and a p99 over a month are wildly different promises.
A one minute window is twitchy and pages you at 3am for nothing. A 30 day window hides a genuinely awful afternoon.

Pick the percentile that matches the blast radius. For an internal batch job, p90 is plenty. For a checkout flow, the tail is the product, because the people in your tail are the ones abandoning carts.
Google’s SRE workbook is good on choosing these deliberately rather than by vibes.

Measure where the user is, not where it is convenient. Server-side latency excludes queuing, DNS, TLS, and the mobile network.
It is the number that makes you look best, which should be your first clue.

That last one has a sharp edge worth knowing about. If your load generator waits for each response before sending the next request, it stops sending traffic exactly when your system is struggling, so your measurements skip the worst moments entirely.

Gil Tene named this coordinated omission, and it is why a benchmark can report a beautiful p99 for a system that was frozen for two full seconds during the run.

Most cloud tooling now gives you these by default. AWS CloudWatch supports p90, p95, p99 and custom percentiles directly on metrics, so there is no real excuse for a mean-only dashboard in 2026.

So what do you actually change on Monday

Concretely, four things.

Put p50, p95 and p99 on the same chart. Not one of them. The gap between the lines is the signal, and you cannot see a gap with a single line.

Delete the average from your latency dashboards. Not demote, delete. As long as it is there, someone will quote it in a status update.

Check whether your metrics pipeline is averaging percentiles. It probably is somewhere. Find it, and move that panel to histograms.

Count the fan-out on your slowest page. Multiply it out.

If the answer is “most page loads hit at least one straggler”, you now know exactly which number to go optimise, and it was never the mean.

The average tells you how your system behaves.

The tail tells you whether people will keep using it.

Your team’s attention is limited, and the deluge of AI-generated code is making it harder to keep production code safe without slowing you down.