Cassandra in Production

A partition key with low cardinality (e.g. a single tenant ID for a huge multi-tenant table) concentrates data and traffic on a handful of nodes. Composite keys (e.g. tenant_id + day) spread load more evenly.

Very large partitions (millions of cells) hurt read latency, compaction, and repair times. A commonly quoted guideline (a heuristic, not a hard limit — the right ceiling depends on row width, hardware and Cassandra version) is to keep partitions under ~100MB and a few hundred thousand rows.

Places replicas on the next N-1 nodes clockwise on the ring, ignoring datacenter or rack topology. Fine for single-DC test clusters, but not recommended for production.

Lets you specify a replication factor per datacenter (e.g. {'dc1': 3, 'dc2': 3}), and is rack-aware within a DC to avoid placing all replicas on the same physical rack. This is the production standard.

The most common production setting for multi-datacenter clusters: quorum within the local datacenter only, avoiding cross-DC latency on the hot path while still requiring a majority.

To guarantee a client reads its own recent write, ensure R + W > RF for that keyspace's consistency levels — e.g. QUORUM writes + QUORUM reads with RF=3.

Modern Cassandra assigns each physical node many small token ranges ('virtual nodes') instead of one large range, which spreads the rebalancing load across the whole cluster when a node joins or leaves, rather than concentrating it on two neighbors.

When a node joins, the cluster streams the relevant data ranges to it directly from existing replicas — a bandwidth- and disk-intensive operation that should be scheduled and monitored carefully in production.

Compares Merkle trees of data ranges across replicas and streams over any differences. Every token range should be repaired at least once within the gc_grace_seconds window (default 10 days); a weekly cadence is a common way to keep margin under that window, not a rule. Missing it risks resurrecting deleted data (the 'zombie' tombstone problem). Full and incremental repair behave differently and incremental repair was reworked in 4.0 — check the docs for your version before choosing.

Read/write latency (p99, not just average), pending compactions, dropped mutations, hinted handoff backlog, and JVM garbage collection pause times are the highest-signal metrics for cluster health.

Pending compactions is a symptom; the per-table strategy is often the cause. Use STCS (the default) for write-heavy tables, LCS when reads dominate and latency matters and you can afford the extra write I/O, and TWCS for write-once time-series data with a TTL (avoid updates, manual deletes and out-of-order writes on it). Whatever you choose, keep disk headroom: STCS can temporarily need a large share of free space for big merges. Mechanics and trade-offs are explained in the write path and compaction chapter.

Poor partition key choice concentrates load on a few nodes, causing localized latency spikes even though the rest of the cluster is healthy.

Partitions that grow unbounded slow down reads, compaction, and repair — and can even cause out-of-memory errors during compaction.

Frequent deletes on the same partition accumulate tombstones that must be scanned on every read, sometimes causing reads to fail outright once a threshold is exceeded.

nodetool status shows per-node health and load; nodetool tpstats shows thread pool queues and dropped messages — the fastest way to spot an overloaded node.

TRACING ON in cqlsh (or per-query tracing) shows exactly which replicas were contacted and how long each step took — invaluable for diagnosing a single slow query.