Scaling out with load balancers

Your service runs on one server. Traffic doubles, then doubles again; CPU is pinned and response times climb. You have two choices. Scale up (vertically): move to a bigger machine. Or scale out (horizontally): run several machines and spread the work across them.

Scaling up is simple and often the right first move — but it has a ceiling, gets disproportionately expensive, and one machine is still a single point of failure. Scaling out has no fixed ceiling and survives the loss of any one machine. It needs one extra component: a load balancer.

What a load balancer does

A load balancer sits in front of a pool of servers and presents one address to clients. For every incoming request or connection, it picks a healthy server and forwards the traffic. Along the way it usually:

• runs health checks — regularly probing each server and removing it from rotation when it stops responding;
• terminates TLS, so certificates live in one place;
• supports zero-downtime deploys, by draining one server at a time.

Layer 4 load balancers route by IP address and port; layer 7 load balancers read HTTP requests and can route by path, header or host — for example, /api to one pool and /images to another.

Run the fleet

The model below sends a stream of requests through a load balancer to three servers. The amber bars show how many requests each server is working on. Server 1 is an older, slower machine. Try these experiments:

• With round robin at 12–15 requests per second, the slow server’s bar keeps growing: it gets the same share as the others but can’t keep up. Switch to least connections and it steadies.
• Fail a server: requests sent to it fail until health checks remove it (about a second and a half here — often 10–30 seconds in real systems). Then the others absorb its traffic.
• Add servers as traffic grows, and watch the per-server load fall.

Stateless servers scale; stateful ones don’t

Horizontal scaling works best when any server can handle any request. That requires stateless application servers — nothing about a user’s session lives only in one server’s memory. Instead:

Sessions

Stored in a shared cache such as Redis, or in signed tokens the client sends with each request.

Uploaded files

Stored in object storage such as Amazon S3, not on a server’s local disk.

Data

In databases shared by all servers — which then become the next thing to scale (lecture 4).

Sticky sessions, where the load balancer always sends a user to the same server, are a common shortcut that undermines all of this: load becomes uneven, and users lose their session when that server fails. See both effects, then move the sessions into a shared store:

Scaling automatically

In the cloud, the pool doesn’t need to be fixed. Autoscaling adds instances when a metric such as CPU utilization or request count crosses a threshold and removes them when load falls. Keep new instances quick to start, keep a sensible minimum for sudden bursts, and remember that the database behind them must cope with the extra connections.

Why not run servers at 95% busy and save money? Because of queueing. As utilization rises, requests increasingly wait behind one another, and response time doesn’t grow steadily — it shoots up near 100%. Push the load towards capacity, then add a server:

That curve is why autoscaling targets usually sit well below 100% — often around 50–70% CPU — leaving headroom for spikes and for the minutes new instances take to start.

Scaling out multiplies capacity. The next technique avoids doing the work at all: caching.