What production API optimization actually looks like

Performance work has a reputation for being about clever tricks — the right cache, the right index, the right algorithm swap. In practice, most of it is unglamorous, and the first unglamorous fact is that "add caching" is rarely the correct first move, because caching a slow thing just means you're slow the first time and fast every time after, without ever answering why it was slow in the first place.

Finding out where the time actually goes

The step that has to come before any fix is measurement — tracing a slow request end to end and finding out, concretely, where the time is actually spent, instead of guessing based on which part of the code looks most complicated. The complicated-looking part is often not the slow part; a simple, unremarkable database call made in a loop is a far more common villain than the elaborate business logic sitting next to it. Optimizing the part that looks hard, without measuring first, is a good way to spend a week improving something that was never the bottleneck.

The fixes that actually move the number

Once the actual bottleneck is identified, the fix is usually one of a small set of unglamorous, well-understood problems: a missing index on a column a query filters or joins on, a response payload sending far more data than the client actually uses, or an N+1 query pattern — a loop that makes one database call per item instead of one call for the whole batch. None of these are clever. All three show up constantly, precisely because they're easy to introduce without noticing: a query that was fine against a small table becomes slow once the table has grown, and nothing about the code changed to signal that a problem had quietly appeared.

An N+1 pattern in particular is worth calling out because it's the easiest of the three to write by accident — a loop over a list of items, with a database call inside the loop that looks completely reasonable in isolation. It works correctly at every step. It's also, silently, doing one round trip per item instead of one round trip total, which is invisible in a small test dataset and very visible in production traffic.

What monitoring changes about what you optimize next

The honest reason performance work in production differs from performance work in a benchmark: production traffic has a shape a synthetic test doesn't, and that shape changes over time. Monitoring — actual latency percentiles per endpoint, not just an average — tells you which endpoint is genuinely worth attention right now, which is a very different answer than "which endpoint looks slowest in a local test." An endpoint that's fast on average but has a bad p99 has a different problem (something intermittent — a lock, a cold cache, a specific input shape) than one that's uniformly slow, and monitoring is what tells you which situation you're actually in before you start guessing at a fix.

The overall shift, moving from ad hoc performance work to something closer to a discipline: stop guessing based on what looks slow, measure where the time actually goes, fix the specific unglamorous thing that measurement points at, and let monitoring — not intuition — decide what gets attention next.