Architecture

Performance Architecture

Performance problems are not solved by guessing and adding infrastructure. They are solved by measuring, locating the actual bottleneck, and applying the cheapest architectural change that moves the constraint.

Mangesh MandlikLast updated: 2 November 2026

What Is Performance Architecture?

Performance architecture, latency, throughput, and bottleneck analysis

Performance architecture is the discipline of designing systems to meet latency and throughput requirements through measurement-driven decisions. Performance asks how fast and efficiently a system handles one operation or a given load level; scalability asks how well that system continues to work as load grows. If you need the growth-planning side of that discussion, continue to Scalability & Capacity.

A common mistake is jumping straight to a familiar fix such as caching, replicas, or more application servers before confirming which resource is actually saturated. The right optimisation depends on evidence about where time is spent and which resource is constraining the system.

Latency, throughput, tail latency, queueing, and saturation

Latency

How long one request or operation takes from the caller's perspective.

Service time

How long the service itself spends doing the work, excluding time spent waiting in a queue.

Tail latency

High-percentile response times such as p95 or p99; this matters because a slow minority of requests can still dominate user pain and timeout risk.

Throughput

How much work completes per unit time, such as requests per second or jobs per minute.

Concurrency

How many requests, threads, workers, or jobs are in progress at the same time.

Utilization

How busy a resource is relative to its capacity, such as CPU%, connection use, or worker occupancy.

Saturation

The point where a resource is fully busy and additional load cannot be served immediately.

Queueing

Work waiting because the resource is already busy; as utilization approaches 100%, queueing delay grows non-linearly rather than a little at a time.

Visual Story

Traffic grows → database bottleneck → cache + async resolves it

Performance Architecture

Baseline

0 / 0

🧮Up nextN+1 Query

⤢ Expand for synchronized narration timeline

In this scenario, the database becomes the bottleneck in Scene 2. The larger lesson is still measurement-first: scaling the API tier before finding the real constraint can add cost without improving latency.

Problem It Solves

Why performance degrades under load

Every system has finite capacity on each resource: CPU, memory, storage I/O, network, locks, downstream concurrency limits, queue workers, and database connections. As load increases, one of those resources saturates first. Requests then wait behind that resource, and tail latency usually degrades before averages look alarming.

Performance architecture solves this by baselining normal behaviour, locating the real bottleneck, choosing the intervention that matches that bottleneck, and then measuring again. The bottleneck may move after each fix, so the workflow is iterative rather than a one-time checklist.

Evidence-Based Bottleneck Analysis

Use tools to find the bottleneck before choosing the fix

Good performance work starts with evidence. Metrics show whether CPU, memory, queue depth, error rate, or connection usage are trending toward saturation. Distributed tracing shows where end-to-end request time is spent. Profiling reveals hot code paths. Slow-query analysis exposes missing indexes, lock contention, and inefficient access patterns. Load testing and baselines tell you whether a change improved the right thing or merely moved the pain elsewhere.

Tracing

Use request traces to break end-to-end latency into database time, downstream calls, queue waits, and application work. This distinguishes a slow dependency from slow local code.

Profiling

Use CPU, memory, and allocation profiles to find hot loops, excessive serialization, GC pressure, or blocking work inside the process.

Metrics + baselines

Track p50/p95/p99 latency, throughput, concurrency, resource utilization, and queue depth against a known baseline so regressions are visible.

Slow-query analysis

Inspect query plans, lock waits, index usage, and cardinality estimates before assuming you need replicas or sharding.

Load testing

Increase load gradually to find the first saturated resource, observe queue growth, and verify that a proposed change actually raises the ceiling.

Measured intervention

Choose caching, async processing, connection pooling, query tuning, compute scaling, or partitioning based on the bottleneck you measured, not on a universal sequence.

For example, a payment request or file upload does not universally need fully synchronous completion to be correct. Many systems synchronously validate and accept the request, return a stable identifier or authorisation result, and finish the heavier processing asynchronously while the client polls, waits for a webhook, or receives a later update.

How Architects Think

Performance is a moving target — not a solved problem

Every optimisation removes one bottleneck and reveals the next. Systems that meet targets at 100 RPS can hit different ceilings at 1,000 RPS, and again at 10,000 RPS. Performance architecture is therefore an iterative practice: measure, understand the constraint, change the system, and re-measure.

The most expensive performance fix is often the one applied to the wrong tier. If tail latency comes from lock contention, database connection saturation, or a slow dependency, adding more stateless application nodes may only increase contention and queueing. Evidence should decide whether you tune code, queries, concurrency limits, storage, network paths, or system boundaries.

What would change your decision? If a cache hit rate drops because the working set no longer fits in memory, the next intervention may shift from cache tuning to data-model or partitioning work. The trigger is a measured change in behaviour, not a preferred pattern.

Illustrative Scenario

P99 latency spiking under moderate load (illustrative scenario; the latencies are invented)

Context: an API handles product search. At 500 requests/second, median response time is 45ms. P99 is 2,300ms. The team assumes it is a database problem and adds two read replicas. The P99 does not improve.

Investigation: distributed tracing reveals that the slow requests are not dominated by query execution time; they are waiting for a connection from the pool. The pool is sized at 10 connections; at 500 RPS, queueing forms well before the rest of the database is saturated. The replicas were the wrong answer because the immediate bottleneck was connection saturation, not read capacity.

Fix: deploy a connection pooler such as PgBouncer between the application and PostgreSQL, re-check query patterns, and load-test again. P99 drops to 180ms at 500 RPS without adding more replicas.

Lesson: the bottleneck moved more than once during scaling, and each stage needed a different fix. No single optimisation sequence is universally correct.

Trade-offs

Every performance fix introduces a new constraint

Caching vs consistency

A cache that serves stale data reduces database load and latency, but the staleness window must be acceptable for the domain. A cache TTL that is fine for a product listing may be unacceptable for an account balance.

Synchronous vs asynchronous processing

Making work asynchronous improves throughput and reduces user-facing latency, but the user may only get acceptance or authorisation immediately while final completion happens later. For payments, file uploads, and similar workflows, the contract must make that state transition explicit.

Optimise the query vs change the architecture

A slow query is often a missing index, a Cartesian join, or an N+1. Those measured issues are usually cheaper to fix than adding replicas, sharding, or another distributed cache tier.

Throughput vs tail latency

Increasing concurrency can raise throughput but also worsen P95 or P99 latency as resources contend. Interactive systems often optimise tail latency; batch systems may accept higher latency in exchange for more total work completed.

What Can Go Wrong

Common performance anti-patterns

Scaling the wrong tier

Added app servers when the bottleneck was database connections. P99 stayed high while cost increased.

Cache with low hit rate

Working set does not fit in memory. Every miss still goes to the database, now with extra cache coordination overhead.

N+1 query

Loop loads 1 record, then queries for each related record individually. What looks fine at low volume becomes expensive on the hot path.

Missing connection pool

Application opens or holds too many database connections. The database spends effort on connection management instead of useful work.

Optimising a cold path

Team spent a week improving a rarely-hit code path. The hot path query or slow dependency was the real bottleneck.

Unbounded async queue

Producers enqueue work faster than workers can consume it. Tail latency moves from requests to backlog growth and delayed completion.

When To Use / When Not To Use

✓ Invest in performance architecture when

→ P99 latency violates user-experience or SLO targets
→ Throughput ceilings block growth without disproportionate cost
→ CPU, I/O, queue depth, or connection usage trends near saturation
→ Cost per request is rising and you need evidence for which tier to optimise

⚠️ Avoid premature optimisation when

→ System is not yet under meaningful load
→ No profiling, tracing, or metrics data identifies the bottleneck
→ Engineering time on optimisation exceeds the cost of the inefficiency
→ Added complexity would harm maintainability for marginal measured gain

Architect's Challenge

An API has P50 latency of 45ms but P99 latency of 2,300ms. The team added two read replicas. P99 did not improve. What is the most likely cause?

Remember· 30-second mental model

Measure first, optimise second. Tail latency, throughput, and concurrency problems often need different fixes.

1Performance is not scalability: performance asks how the system behaves at a given load; scalability asks how it behaves as load grows.
2Track p50, p95, and p99. Averages can hide the slow requests that users actually feel.
3As utilization approaches saturation, queueing delay grows non-linearly.
4Use tracing, profiling, metrics, slow-query analysis, and load testing to find the bottleneck before choosing a fix.
5Async processing changes the workflow contract. Use it when delayed completion is acceptable and clearly communicated.

Key lesson

The expensive mistake is not missing a fancy optimisation. It is adding the wrong optimisation before you have evidence about the real bottleneck.