Kubernetes in Production: Rolling Updates, Probes, Autoscaling & Debugging

01 · The workload primitive

Deployments

A Deployment is the standard way to run stateless applications in production — it manages ReplicaSets so you never have to manage Pods by hand.

Beyond replica count, a Deployment spec controls the update strategy, revision history limit, and Pod template (image, resources, probes, env). Almost every production workload starts here.

02 · Deploying without downtime

Rolling Updates

RollingUpdate is the default and recommended Deployment strategy — new Pods are brought up before old ones are torn down.

Swipe to pan

maxSurge and maxUnavailable together control how aggressive the rollout is.

maxSurge

How many extra Pods (above the desired count) are allowed to exist temporarily during a rollout — higher values roll out faster but use more resources. Defaults to 25% (rounded up) when unset.

maxUnavailable

How many Pods are allowed to be unavailable during the rollout — 0 guarantees full capacity throughout, at the cost of a slower rollout. Defaults to 25% (rounded down) when unset, so a default rollout may briefly run at 75% of desired capacity.

Zero-downtime rollout checklist

yamlIllustrative manifest: the settings that make rollouts and node drains safe (values are examples to adapt, not recommendations for every workload)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 4
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0        # never drop below desired capacity
  template:
    spec:
      terminationGracePeriodSeconds: 45   # must cover preStop + app shutdown
      containers:
        - name: app
          image: registry.example.com/app:1.4.2
          resources:
            requests: { cpu: 250m, memory: 256Mi }   # scheduling + HPA math
            limits:   { memory: 256Mi }              # OOMKilled above this
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
          lifecycle:
            preStop:
              exec: { command: ["sleep", "10"] }     # let endpoints propagate
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app
spec:
  minAvailable: 3
  selector:
    matchLabels: { app: app }

Readiness gates the rollout

With maxUnavailable: 0, old Pods are only removed as new Pods become Ready, so a correct readiness probe is what keeps traffic off Pods that are still booting.

preStop and terminationGracePeriodSeconds

When a Pod is deleted, removal from Service endpoints and the shutdown sequence happen concurrently, so a Pod can still receive a few requests after it starts terminating. A short preStop delay lets endpoint updates propagate before SIGTERM is sent. The grace period (default 30 s) is one budget covering the preStop hook and the application's own shutdown; when it expires the container is SIGKILLed. The app must also handle SIGTERM by draining in-flight requests. The exec example needs a sleep binary in the image.

PodDisruptionBudget

A PDB limits how many replicas can be taken down by *voluntary* disruptions such as kubectl drain and cluster upgrades, which go through the Eviction API. It does not protect against node crashes, and Deployment rolling updates are not limited by it — maxUnavailable governs those. Set it below the replica count (e.g. minAvailable: 3 of 4): a PDB that allows zero disruptions blocks node drains indefinitely.

Requests vs limits

The scheduler places Pods using requests; HPA utilisation is computed against requests. A memory limit is a hard ceiling (exceeding it means OOMKilled), while a CPU limit causes throttling rather than a kill. Setting memory request equal to the limit avoids over-commit surprises under node memory pressure.

03 · Undoing a bad release

Rollbacks

Every Deployment keeps a revision history — rolling back reuses a previous ReplicaSet's Pod template instead of requiring a new build.

bashRolling back to the previous revision, or a specific one.

kubectl rollout undo deployment/app
kubectl rollout undo deployment/app --to-revision=3
kubectl rollout history deployment/app

04 · Knowing what's actually healthy

Health Checks

The kubelet doesn't just trust that your container is fine because the process is running — probes verify actual application health.

Liveness Probe

Answers "is this process stuck/broken and should be restarted?" A failing liveness probe causes the kubelet to kill and restart the container.

Readiness Probe

Answers "can this Pod currently serve traffic?" A failing readiness probe removes the Pod from Service endpoints, without restarting it.

Startup Probe

Gives slow-starting applications extra time before liveness/readiness probing begins, avoiding premature restarts during a long boot sequence.

05 · The distinction that matters

Readiness vs Liveness Probes

These two probes answer genuinely different questions, and conflating them can cause outages.

Liveness failing → restart the container (assumes a restart will fix it). Readiness failing → stop sending traffic, but leave the container alone (assumes it will recover on its own, e.g. finishing warm-up or waiting out a dependency outage). Restarting a Pod that's simply waiting on a slow dependency doesn't help — and can make a cascading outage worse by adding restart churn.

Chapter 06

Kubernetes Requests vs Limits

Kubernetes requests vs limits is the difference between what the scheduler reserves for a container and the ceiling the container is allowed to hit. The scheduler places Pods using requests, and HPA utilization is computed against requests too.

	Request	Limit
Used for	Scheduler placement and HPA utilization math	A hard ceiling on resource use
Memory	Reserved when the Pod is placed	Exceeding it means the container is OOMKilled
CPU	Reserved when the Pod is placed	Exceeding it causes throttling, not a kill

06 · Scaling automatically

Horizontal Pod Autoscaling

The HPA adjusts replica count based on observed metrics, most commonly CPU utilization.

yamlTarget 60% average CPU, scale between 2 and 10 replicas.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Chapter 08

Kubernetes Horizontal Pod Autoscaler Checklist

The Kubernetes Horizontal Pod Autoscaler adjusts the replica count of a workload such as a Deployment. Check these before relying on it:

Key takeaways

01Point scaleTargetRef at the Deployment you want to scale.
02Set a sensible minReplicas and maxReplicas range.
03Set CPU requests on the containers: HPA utilization is computed against the request, not the limit.
04Choose the metric target deliberately, for example the 60% average CPU utilization in the manifest above.

07 · Where Pods actually land

Scheduling

The scheduler's decision is a two-phase process: filter out nodes that can't work, then score and pick the best of what remains.

Filtering (Predicates)

Removes nodes that fail hard constraints — insufficient CPU/memory, taints the Pod doesn't tolerate, node selectors that don't match.

Scoring (Priorities)

Ranks remaining nodes by soft preferences — spreading Pods for availability, balancing resource usage, honoring affinity/anti-affinity rules.

08 · Durable state

Persistent Volumes

PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) decouple storage lifecycle from Pod lifecycle.

PersistentVolume (PV)

A piece of actual storage in the cluster (a cloud disk, NFS share, etc.), provisioned either manually or dynamically via a StorageClass.

PersistentVolumeClaim (PVC)

A Pod's request for storage — Kubernetes binds it to a matching PV. The PVC (and its data) survives Pod deletion and recreation.

09 · Getting traffic in

Networking & Ingress

Internal traffic uses Services; external HTTP(S) traffic typically goes through Ingress.

Ingress

Declares host/path-based routing rules and TLS termination, interpreted by an Ingress Controller (e.g. NGINX Ingress, cloud load balancer controllers).

NetworkPolicy

Firewall-like rules controlling which Pods can talk to which — by default, all Pods can reach all other Pods, which is rarely the right production posture.

10 · Keeping images environment-agnostic

Configuration & Secrets

The same image should deploy to every environment — differences belong in ConfigMaps and Secrets, injected at Pod start.

11 · Seeing what's happening

Monitoring & Logging

Kubernetes doesn't give you observability for free — it gives you the primitives to build it consistently.

Metrics

The metrics-server (or a full Prometheus stack) collects CPU/memory usage per Pod/node — this is what powers kubectl top and the HPA.

Logs

Container stdout/stderr is captured by the container runtime; a log-shipping agent (Fluent Bit, Fluentd) typically forwards it to a central store, since Pod filesystems (and their logs) disappear on restart.

12 · When things go wrong

Common Production Issues & Debugging

Most Kubernetes production incidents fall into a small set of recurring categories.

CrashLoopBackOff

The container keeps crashing right after starting; Kubernetes backs off retry attempts exponentially. Check kubectl logs --previous and the exit code first.

ImagePullBackOff

The node can't pull the specified image — wrong tag, private registry auth missing, or a typo in the image name.

Pending Pods

The scheduler can't place the Pod — usually insufficient cluster resources, or constraints (node selectors, taints, affinity rules) that no node satisfies.

OOMKilled

The container exceeded its memory limit and was killed by the kernel — check kubectl describe pod for the exit reason and consider raising the limit or fixing a leak.

Key takeaways

01kubectl describe pod and kubectl logs (including --previous) are the first two commands for almost every production issue.
02Rolling updates and rollbacks are the same mechanism in reverse — both rely on healthy readiness checks to proceed safely.
03Liveness restarts; readiness reroutes traffic — mixing these up is one of the most common production misconfigurations.
04HPA scales replica count based on requests, not limits — incorrect requests silently break autoscaling math.
05PVCs, ConfigMaps and Secrets exist specifically so Pods can be freely destroyed and recreated without losing state or configuration.