DNS Production Engineering

MX records, NS records, and long-lived A records for stable infrastructure can often use long TTLs (hours to a day) because they rarely change and benefit from lower resolver load.

Records that need to react quickly to health changes should use relatively short TTLs (often tens of seconds to a few minutes). That narrows the stale-cache window, but DNS failover is still bounded by health-check detection and by whatever browsers, OS resolvers, and recursive resolvers have already cached.

The authoritative nameserver estimates where the recursive resolver is located (usually from the resolver IP, sometimes improved by ECS/EDNS Client Subnet) and returns a region-specific answer — for example, a European resolver may receive a European edge or regional ingress IP.

The same IP address is announced from many physical locations at the network (BGP) layer; routers deliver traffic to the topologically nearest announcing location. DNS commonly hands out an anycast address, while the CDN and network layer do the detailed work of getting the user to a nearby edge.

Some DNS providers (e.g. Route 53) measure real network latency from various vantage points to each candidate region and route users to the lowest-latency option rather than pure geographic proximity.

The authoritative server can rotate record order on each response, or return several IPs and let the client/resolver use them in its own order, spreading traffic only roughly over time.

Some providers support assigning traffic percentages per record (e.g. 90% to the primary region, 10% to a canary region) instead of pure even rotation.

The DNS/traffic-management provider itself sends periodic HTTP or TCP probes to each endpoint from multiple global vantage points, marking it unhealthy after a threshold of consecutive failures (to avoid flapping on transient blips).

Some setups also feed real traffic error rates or synthetic transaction results into the same health signal, catching failures that a simple TCP handshake wouldn't.

Track resolution time and NXDOMAIN/SERVFAIL rates from multiple regions — a spike often precedes a broader outage.

Continuously verify that live DNS records match what's declared in infrastructure-as-code (Terraform, etc.) — manual out-of-band changes are a classic source of incidents.

Domain registration and DNSSEC key expirations are a recurring, entirely preventable cause of outages — automated expiry alerts are standard practice.

A highly avoidable outage class — a domain lapses and every record under it vanishes. Prevented entirely by auto-renewal and expiry alerts.

A CNAME pointing at a decommissioned third-party resource (e.g. a deleted S3 bucket or old load balancer) — this is also a security risk known as subdomain takeover, where an attacker claims the abandoned resource and hijacks the subdomain.

During an incident requiring an emergency IP change, a long-lived TTL from before the incident means many clients keep hitting the failed IP long after the fix is live.

If your DNS provider itself goes down, your domain is unreachable regardless of how healthy your actual servers are — this is why some large organizations run DNS across two independent providers.