How does Netflix manage to stream to millions at once without buffering? How does Amazon ride massive traffic spikes without melting? A huge part of the answer is thoughtful load balancing. At scale, this isn’t just “distribute requests.” It’s health checks, traffic shaping, global routing, zero-downtime deploys, and graceful failure.
Below are the core ideas, then immediately, practical snippets you can use in real systems.
What is Load Balancing?
At a basic level, you spread incoming requests across multiple servers so no machine gets overwhelmed. At scale, you also need sticky sessions, blue-green or canary rollout support, autoscaling integration, and fast failover when nodes die. Think traffic control for APIs.
L4 vs L7 (quick refresher)
Layer 4 (TCP/UDP) is fast and blind to HTTP details—great for raw throughput (gaming, streaming).
Layer 7 understands HTTP/HTTPS and can route by paths, headers, cookies—perfect for microservices and API gateways.
Common Algorithms, with “when to use”
Round Robin: even distribution when requests are short and similar.
Least Connections: better for long-lived or uneven requests (streams, uploads).
Weighted RR: heterogeneous capacity clusters.
Consistent Hashing: keep the same user on the same backend to improve cache hit rate or locality.
Production snippets you can paste and run
1) NGINX L7 load balancing with least_conn, health checks, and canary
# /etc/nginx/conf.d/api.conf
upstream api_pool {
least_conn; # algorithm
zone api_pool 64k; # share state for health checks
server 10.0.1.11:8080 max_fails=3 fail_timeout=5s;
server 10.0.1.12:8080 max_fails=3 fail_timeout=5s;
# optional canary with weight
server 10.0.1.99:8080 weight=1; # send a small % of traffic
}
server {
listen 80;
server_name api.example.com;
# health endpoint on backends: GET /healthz -> 200 OK
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Request-Id $request_id;
proxy_connect_timeout 1s;
proxy_read_timeout 5s;
proxy_pass http://api_pool;
}
# sticky session via cookie (simple variant)
# map $cookie_route $route { default ""; }
# proxy_cookie_path / "/; HttpOnly; Secure; SameSite=Lax";
}
Why it matters: least_conn stabilizes tail latencies when some requests are heavier. max_fails and fail_timeout act like quick, simple health gates. Canary instance gets just a sliver of traffic via a lower weight.
2) HAProxy with stickiness and circuit-breaker-style outlier ejection
# /etc/haproxy/haproxy.cfg global maxconn 100000 log /dev/log local0 defaults mode http timeout client 30s timeout server 30s timeout connect 5s option httplog backend api_back balance leastconn option httpchk GET /healthz http-check expect status 200 # sticky on a cookie named ROUTE cookie ROUTE insert indirect nocache server s1 10.0.1.11:8080 cookie s1 check fall 2 rise 2 server s2 10.0.1.12:8080 cookie s2 check fall 2 rise 2 # basic outlier ejection: mark as down on failures, quick recovery on rise frontend api_in bind :80 default_backend api_back
Why it matters: stick tables/cookies keep session-bound users on the same node; httpchk gives fast failover. fall/rise mimic circuit breaker feel: eject after consecutive fails, re-admit after consecutive passes.
3) Consistent hashing in 20 lines (Node.js)
This is the mental model behind routing users to the same shard or cache node.
// consistent-hash.js
const crypto = require('crypto');
class ConsistentHash {
constructor(nodes = [], replicas = 128) {
this.ring = new Map();
this.sorted = [];
this.replicas = replicas;
nodes.forEach(n => this.add(n));
}
_hash(key) {
return parseInt(crypto.createHash('md5').update(String(key)).digest('hex').slice(0, 8), 16);
}
add(node) {
for (let i = 0; i < this.replicas; i++) {
const h = this._hash(node + ':' + i);
this.ring.set(h, node);
this.sorted.push(h);
}
this.sorted.sort((a, b) => a - b);
}
get(key) {
const h = this._hash(key);
let idx = this.sorted.findIndex(x => x >= h);
if (idx === -1) idx = 0;
return this.ring.get(this.sorted[idx]);
}
}
// demo
const ch = new ConsistentHash(['nodeA', 'nodeB', 'nodeC']);
console.log('user:42 ->', ch.get('42'));
console.log('user:99 ->', ch.get('99'));
Why it matters: when nodes join/leave, only a small portion of keys remap, which keeps cache hit rates healthy at scale.
4) App-side resilience: retry with jitter and idempotency (Node/Express client)
// fetch-with-retry.js
const fetch = require('node-fetch');
async function fetchWithRetry(url, opts = {}, attempts = 3) {
let delay = 100; // ms
for (let i = 0; i < attempts; i++) {
try {
const res = await fetch(url, { ...opts, timeout: 5000 });
if (res.ok) return res;
if (res.status >= 500) throw new Error('server error ' + res.status);
return res; // 4xx -> don't retry by default
} catch (e) {
if (i === attempts - 1) throw e;
const jitter = Math.floor(Math.random() * 50);
await new Promise(r => setTimeout(r, delay + jitter));
delay = Math.min(delay * 2, 2000);
}
}
}
// usage with idempotency key for POST
const idempotencyKey = crypto.randomUUID();
fetchWithRetry('https://api.example.com/orders', {
method: 'POST',
headers: { 'Idempotency-Key': idempotencyKey, 'Content-Type': 'application/json' },
body: JSON.stringify({ sku: 'A1', qty: 1 })
});
Why it matters: load balancers can reroute after failures, but clients should also be polite. Retrying with backoff+jitter avoids thundering herds, and idempotency keys prevent duplicate charges.
5) Kubernetes: readiness gates, PodDisruptionBudget, and zero-downtime rollouts
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels: { app: api }
template:
metadata:
labels: { app: api }
spec:
containers:
- name: app
image: registry.example.com/api:1.23.0
ports: [{ containerPort: 8080 }]
readinessProbe:
httpGet: { path: /healthz, port: 8080 }
initialDelaySeconds: 2
periodSeconds: 3
livenessProbe:
httpGet: { path: /livez, port: 8080 }
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: api-svc
spec:
selector: { app: api }
ports:
- name: http
port: 80
targetPort: 8080
type: LoadBalancer
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
minAvailable: 5
selector:
matchLabels: { app: api }
Why it matters: the Service fronts your pods with kube-proxy/Cloud LB. Readiness ensures only healthy pods receive traffic. PDB prevents too many pods going down during maintenance, keeping SLAs intact.
6) Edge geo-routing sketch (DNS + Anycast mental model)
Practical recipe most teams use:
- Terminate DNS with a provider that supports geo/latency-based routing.
- Advertise a single Anycast IP from multiple POPs (via your CDN, or a provider like Cloudflare/Akamai).
- At each POP, run L7 proxies (NGINX/Envoy) that route to regional clusters.
- Health probes flip traffic away from a failing region; caches warm automatically.
You don’t always implement Anycast yourself, but understanding the pattern helps you reason about global failover and cold-start penalties.
Real interview walk-through: 50M concurrent live stream
Design choices that interviewers expect to hear, with small, crisp justifications:
- CDN terminates video chunks, origin protected behind WAF and tokenized URLs.
- L7 balancers per region use
least_connand header-based routing. - Sticky sessions only where stateful sessions are unavoidable, otherwise JWT + Redis for rate limits.
- Global balancing: Geo-DNS to the nearest healthy region, automatic failover on health degradation.
- Deployments via canary: 1 weighted instance behind the pool, promote on SLOs meeting thresholds.
- Client behavior: retry with backoff and idempotency, circuit breaker on the SDK to shed load gracefully.
Tools to keep in your kit
Open source: NGINX, HAProxy, Envoy.
Cloud managed: AWS ALB/NLB, GCP External HTTP(S) LB, Azure Front Door.
Service mesh: Istio/Linkerd for per-service traffic policies (retries, timeouts, outlier detection).
FAQs
Which algorithm should I start with?
Round Robin for simplicity, switch to Least Connections when request durations vary. Use weights when hardware differs, and consistent hashing when cache locality matters.
How do I avoid the load balancer being a single point of failure?
Run at least two in active-active behind a VRRP or managed VIP, use health-checked DNS to fail between them.
How do I do zero-downtime deploys?
Combine readiness probes with rolling updates, or do blue-green/canary using separate target groups and weighted traffic.
Do I need sticky sessions?
Prefer stateless services. If unavoidable, use cookies or consistent hashing and store critical state in an external store (Redis, DB).
How does load balancing differ from autoscaling?
Load balancers split traffic, autoscaling changes pool size. You typically need both for steady latency.
