Architecture

Scalability & Capacity

Scalability patterns — horizontal and vertical scaling, stateless services, caching, read replicas, connection pooling, and sharding — only pay off when matched to the actual bottleneck: define the terms clearly, find the saturated resource, choose the pattern the evidence supports, then load test and measure again.

Mangesh MandlikLast updated: 1 November 2026

What Is Scalability & Capacity?

Scalability is how gracefully a system handles more load when you add resources or change the design. Capacity is the amount of load the current design can carry before it violates latency, error-rate, or cost targets. Elasticity is how quickly that capacity can expand or contract.

The surrounding terms matter too: load is the incoming work, throughput is completed work per unit time, concurrency is how much work is in flight, utilization is how busy a resource is, saturation is the point where queueing and contention begin to dominate, headroom is spare capacity for spikes or failures, and latency is how long each request or job waits plus executes.

Key terms you should distinguish

Scalability

How the system behaves as load grows after you add resources or redesign a bottleneck.

Capacity

The maximum load the current design can sustain within its targets.

Elasticity

How fast capacity can increase or decrease in response to changing demand.

Throughput

Completed requests, jobs, or messages per unit time.

Concurrency

How many requests or jobs are in flight at once.

Utilization

How busy a resource is relative to its current limit.

Saturation

The point where added work mostly creates queues, lock waits, retries, or contention.

Headroom

Spare capacity kept for spikes, failures, and scaling lag.

Latency

Time spent waiting, being serviced, and sometimes retrying.

Load

The demand arriving at the system: users, requests, events, bytes, or jobs.

Common scalability patterns

Vertical Scaling

Make one node bigger. Simple, no code change — but has a ceiling and keeps more risk in one place.

Horizontal Scaling

Add more nodes behind a load balancer. Powerful, but still limited by coordination cost and the slowest shared dependency.

Stateless Services

Hold no session state on the instance, so any request can land on any node. Stateful workloads still need a shared store, sharding, or client affinity.

Load Shedding

Reject or defer excess work once a resource nears saturation, trading some requests to protect the system from cascading overload.

Autoscaling

Expand or contract capacity automatically with load. Saves cost versus provisioning for peak, but reacts with a lag during sudden spikes.

Connection Pooling

Caps concurrent database connections so a scaled-out app tier doesn't overwhelm a single primary. See Database Scaling for the mechanics.

Caching

Serves hot data from a fast layer instead of hitting the primary store on every request — though an expired key under heavy traffic can trigger a cache stampede.

Read Replicas

Offloads read-heavy traffic from the primary once query tuning and caching aren't enough. See Database Scaling for the trade-offs.

Sharding / Partitioning

Splits data across nodes when write throughput or data volume outgrow a single node — deferred until simpler options are exhausted, and only with a shard key that avoids hot partitions.

Visual Story

Traffic grows → bottleneck appears → architecture evolves

Scalability & Capacity

Baseline

0 / 0

📈Up nextTraffic Grows 5×

⤢ Expand for synchronized narration timeline

Notice that scaling the app tier first (Scene 2) makes the database worse, not better. The bottleneck must be identified before scaling — otherwise you amplify the wrong tier.

Problem It Solves

Why naïve scaling often makes things worse

Real ceilings can be CPU on the application tier, but they can also be shared or constrained resources: a single primary database, memory pressure, a saturated network link, storage IOPS, lock contention, a hot partition, exhausted DB connections, downstream rate limits, underprovisioned workers, or a queue backlog that grows faster than consumers drain it.

That is why scalable architecture is less about a fixed recipe and more about a method: measure the load, find the saturated resource, understand the queueing or contention, choose a targeted intervention, load test it, and then re-measure to see whether the bottleneck really moved.

Measure Before You Scale

The repeatable method: measure → diagnose → intervene → re-measure

Measure the baseline

Capture throughput, latency percentiles, error rate, concurrency, utilization, and queue depth under representative load.

Find the saturated resource

Check CPU, memory, network, storage IOPS, DB connections, locks, hot keys or partitions, downstream quotas, worker slots, and backlog growth.

Explain the waiting

Ask where time is spent: service time, lock wait, connection wait, queue wait, retry delay, replication lag, or throttling by a dependency.

Choose the intervention that matches the bottleneck

Tune queries or indexes, add pooling, cache hot data, split a hot partition, raise worker count, shed load, scale a tier, or change the architecture only when the measurements justify it.

Load test the change

Use a realistic workload mix so you test the actual constraint, not a synthetic happy path.

Re-measure and look for the new ceiling

A good change often reveals the next bottleneck rather than eliminating limits forever.

Typical interventions after you know the bottleneck

App CPU saturation

If CPU is the real limit, horizontal scaling or code-path optimisation may help — provided the dependencies underneath can absorb the extra load.

Database read pressure

Start by checking query shape and indexes. If reads are still the issue, then consider connection pooling, caching, replicas, or a larger node based on what the measurements show.

Contention and hot spots

Hot keys, hot partitions, and lock contention usually need a data-model or access-pattern fix, not just more app servers.

Worker and queue limits

A growing backlog means worker capacity, downstream latency, or retry behavior is capping throughput. Measure service time before simply adding more consumers.

Dependency or infrastructure ceilings

Network bandwidth, storage IOPS, rate limits, and connection caps can dominate. Your system cannot scale past its slowest constrained dependency.

Memory and working-set limits

If caches thrash or heap usage stays pinned, more instances may only spread misses around. The fix may be a smaller working set, better eviction, or more memory per node.

How Architects Think

Scaling decisions are evidence-driven, not a universal sequence

There is no universal progression of pooler → replica → cache → bigger box → sharding. Those are common options, often in rough cost and complexity order, but the right move depends on which resource is actually saturated. If the bottleneck is a database, query and index work is often cheaper than infrastructure changes and should usually be checked first.

Capacity planning is a budget, not a guess: know your baseline load, expected peak, growth projection, tested per-instance ceiling, failure headroom after losing N instances, autoscaling lag, and dependency ceilings. If you do not know those values, you are estimating from hope rather than evidence.

Worked example (illustrative planning numbers, not a measured system)

Assumptions: baseline 1,200 req/s; peak factor 3×; 25% growth headroom over the planning horizon; a load-tested ceiling of 250 req/s per instance, planned to run at no more than 70% of it; instances spread over 3 availability zones.

Peak: 1,200 × 3 = 3,600 req/s; with growth, 3,600 × 1.25 = 4,500 req/s. Usable capacity per instance: 250 × 0.7 = 175 req/s. Instances for the peak: 4,500 / 175 ≈ 25.7, so 26.

Failure headroom: to keep serving the peak after losing one whole zone, the two surviving zones must hold those 26 instances, so each zone runs 13 and the fleet totals 39. After the zone loss the survivors run at 4,500 / (26 × 250) ≈ 69% of their tested ceiling; with all 39 healthy, peak utilisation is about 46%. Losing one instance out of 26 would instead need only 27 (N+1).

Autoscaling lag: instances that take minutes to start cannot cover a spike that arrives faster than that, so the always-on fleet must already carry the ramp (or you shed load). And the 4,500 req/s must also fit the database and every other dependency ceiling, not only the app tier.

If the main question is why latency rose at the same load, go deeper in Performance Architecture. If the key trade-off is how much headroom you can afford, continue to Cost & Efficiency.

Illustrative Scenario

A launch that outgrows a single database (constructed for teaching, not a real incident)

A product goes viral. The app tier is stateless, so the team scales it out from 3 to 30 instances in minutes — and latency gets worse. Adding servers made it slower. Why?

In this illustrative scenario, the bottleneck was not the app tier. Every request hits a single primary PostgreSQL, and 30 instances open far more concurrent connections. The database now spends time on connection churn, lock contention, and slow queries. Horizontal scaling of the wrong tier amplified the real bottleneck.

The team follows the method instead of a canned sequence: first profile slow queries and indexes, then add a connection pooler to cap connection pressure. If read traffic remains the measured limit, they add a replica or cache for the read-heavy paths. If write throughput, storage IOPS, or data volume later become the true ceiling, only then do they consider vertical scaling or sharding.

Trade-offs

Every scaling choice buys one thing at a price

Vertical vs horizontal

Scaling up one machine is simple and avoids distribution work, but has a ceiling and a larger single-node failure domain. Scaling out can extend capacity significantly, but coordination overhead, data-layer bottlenecks, and downstream limits still cap the system.

Stateless vs stateful

Stateless services are easier to spread across instances. Stateful workloads can still scale, but the design may need a shared store, sharding, client affinity, or another strategy matched to the workload.

Provision for peak vs autoscale

Provisioning for peak wastes money most of the day but absorbs spikes instantly. Autoscaling saves cost but reacts with a lag — cold starts and warm-up can miss a sudden surge.

Throughput vs latency

Batching and queuing raise throughput but add latency per request. Optimising tail latency often means leaving capacity idle. You rarely maximise both at once.

What Can Go Wrong

Classic scaling anti-patterns

📈Scaling the wrong tier

More app servers amplify DB load. The bottleneck gets worse, not better.

🔗No connection pool

N app instances × M threads = N×M DB connections. The database can fail from connection overhead alone.

🔥Hot partition

Shard key is a timestamp. New writes pile onto one partition while the others sit mostly idle.

💥Cache stampede

Cache expires. Many requests miss together and flood the origin. The origin collapses.

📊No capacity headroom

System runs close to saturation in normal operation. Any spike or instance failure causes cascading overload.

⚙️Premature sharding

Team shards before cheaper, evidence-based options are exhausted. Cross-shard queries become the new bottleneck.

When To Use / When Not To Use

Match the scaling strategy to the actual bottleneck

✓ Horizontal scaling when

→ App-tier CPU or concurrency is the measured bottleneck
→ Dependencies underneath can absorb the additional load
→ The workload is naturally parallelisable across instances
→ Fault isolation across failure domains matters

⚠️ Defer sharding until

→ Query/index work and simpler infrastructure options have been evaluated
→ Write throughput or data volume still exceed a single-node design
→ Hot keys or partitions cannot be solved with access-pattern changes
→ Team is prepared for cross-shard correctness and operational cost

Architect's Challenge

Your e-commerce service handles 500 req/s normally. Black Friday load is expected to reach 5,000 req/s. You've already horizontally scaled the app tier to 20 instances. Latency is still climbing. What do you investigate next?

Remember· 30-second mental model

Measure the load, find the saturated resource, and move that bottleneck on purpose.

1Scalability is how the system behaves as load grows; capacity is the load the current design can sustain right now.
2Horizontal scaling helps only when the measured bottleneck is in a tier that can truly spread across instances.
3Database bottlenecks often justify query, index, pooling, cache, replica, or vertical-scale work before sharding.
4Capacity planning needs baseline load, peak load, growth, tested per-instance ceilings, failure headroom, autoscaling lag, and dependency ceilings.
5Load test after each intervention and re-measure — every successful fix reveals a new limiting resource.

Key lesson

Adding servers to the wrong tier is a common scaling mistake. The important question is not 'how do we scale?' but 'what is saturated right now?'