Deployments, self-healing and scaling

In the last lecture a Deployment turned into three running pods. Real systems are messier: processes crash, machines fail, traffic spikes, and new versions ship several times a day. This lecture puts you in control of a cluster model to see how Kubernetes handles each of these — and which settings you, as its operator, are responsible for.

Try it: break the cluster

The model below runs a Deployment named web with three replicas. Use the controls to cause trouble and watch the control loop respond:

• Crash a pod — the ReplicaSet sees two pods where three are desired and creates a replacement.
• Scale 3 ↔ 5 — changing the desired count is all it takes; controllers add or remove pods.
• Roll out v2 — new pods (amber) start one at a time; an old pod is removed only after a new one is running. The service never drops below capacity.
• Fail node-3 — its pods are lost and recreated on the remaining nodes. Restore the node and new pods can be scheduled there again.

Deployments and ReplicaSets

A Deployment doesn’t manage pods directly. It manages ReplicaSets, each of which keeps a fixed number of identical pods — one specific version of your pod template — running. When you change the template, for example the image tag, the Deployment creates a new ReplicaSet and gradually shifts replicas from the old one to the new one.

That structure makes rollbacks simple, because the previous ReplicaSet still exists:

kubectl set image deployment/web web=shop-web:1.5.0   # start a rolling update
kubectl rollout status deployment/web                  # watch it progress
kubectl rollout undo deployment/web                    # go back if something is wrong

The pace of an update is controlled by maxSurge (how many extra pods may exist during the update) and maxUnavailable (how many may be missing). The model above uses one surge and zero unavailable — the safest setting for most services.

Try other settings, and switch off the readiness probe to see why the next section matters:

Probes: telling Kubernetes what “healthy” means

A running process isn’t necessarily a working service. Kubernetes lets you define probes that the kubelet checks regularly:

Readiness probe

Can this pod serve traffic right now? Until it passes, the pod receives no requests from its Service. Essential for zero-downtime deploys.

Liveness probe

Is this pod stuck beyond recovery? If it keeps failing, the kubelet restarts the container.

Startup probe

Gives slow-starting applications time to boot before liveness checks begin.

readinessProbe:
  httpGet: { path: /healthz/ready, port: 80 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /healthz/live, port: 80 }
  initialDelaySeconds: 20
  periodSeconds: 10

Keep the liveness check simple — “is the process responsive?” — and don’t make it depend on databases or other services. Otherwise a brief database hiccup can make Kubernetes restart every pod at once.

Scaling

Scaling a Deployment is just changing its desired replica count, either by hand — kubectl scale deployment/web --replicas=5 — or automatically:

• The Horizontal Pod Autoscaler (HPA) adjusts the number of replicas to keep a metric, usually average CPU utilization relative to the pods’ CPU requests, near a target.
• When new pods don’t fit on existing nodes, a cluster autoscaler (or a tool such as Karpenter on AWS) adds nodes, and removes them again when they are idle.

Watch an autoscaler handle a traffic spike. Change its target and how long it waits before scaling down:

Both depend on accurate resource requests. The scheduler uses requests to decide where pods fit, and the HPA measures utilization against them — so requests that are far too high waste money, and requests that are far too low cause crowded, slow nodes.

That completes the first module. You can now build images, run and connect containers, and explain how Kubernetes keeps applications running. Module 2 continues with Services and networking, configuration and secrets, ingress, Helm and observability.