Load Balancing

A load balancer is the traffic cop of a distributed system. It sits in front of a pool of servers and routes each incoming request to one of them, ensuring no single server is overwhelmed. Without a load balancer, adding more servers to handle traffic would be invisible to clients — they would still send all requests to the original server's IP address.

Why Load Balancers Exist

Imagine a single web server handling all traffic. As usage grows:

1. The server becomes a bottleneck — CPU and memory are exhausted.
2. If the server crashes, the entire service goes down.

A load balancer solves both problems:

• Scalability: distribute requests across as many servers as needed.
• High availability: when a server fails, the load balancer stops routing traffic to it and distributes the load among healthy servers.

Load Balancing Algorithms

Round-Robin

Each new request goes to the next server in a rotating list. After the last server, it starts again from the first.

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A  (back to start)
...

Best for: servers with identical hardware and workloads that have similar duration.

Problem: a long-running request on Server A does not prevent Server A from receiving the next round-robin request. One server might end up with many long-lived connections while another sits mostly idle.

Least Connections

Route each new request to the server with the fewest active connections.

Best for: long-lived connections (WebSockets, database connections) where request duration varies widely.

Weighted Round-Robin / Weighted Least Connections

Servers are assigned a weight proportional to their capacity. A server with weight 3 receives 3× as many requests as a server with weight 1.

Best for: heterogeneous hardware — mixing a powerful dedicated server with a smaller one.

IP Hash

A hash of the client's IP address determines which server receives the request. The same IP always maps to the same server (as long as the server pool does not change).

Best for: applications that store session state in server memory (though the better solution is stateless servers + shared Redis).

Consistent Hashing (advanced)

A more sophisticated form of hashing that minimizes redistribution when servers are added or removed. Covered in depth in the Consistent Hashing subject.

Layer 4 vs Layer 7 Load Balancers

Layer 4 (Transport Layer)

Operates on TCP/UDP headers: source IP, destination IP, and port numbers. It does not inspect the content of the packets.

• Very fast and low latency (minimal CPU overhead).
• Cannot make routing decisions based on HTTP content (e.g., URL path, headers, cookies).
• Example: AWS Network Load Balancer (NLB).

Layer 7 (Application Layer)

Operates on the full content of HTTP requests: URL path, headers, cookies, and body.

• Slower than L4 (must parse HTTP), but can make smart routing decisions.
• Route /api/* to an API server farm and /static/* to an asset server.
• Terminate TLS here and talk HTTP to backend servers (offloads crypto from app servers).
• Example: AWS Application Load Balancer (ALB), nginx, HAProxy.

OmniAtlas uses an ALB — L7, which lets CloudFront and WAF sit in front and route /static/* directly to S3.

Health Checks

A load balancer periodically sends health check requests to each backend server (e.g., GET /health → 200 OK). If a server fails N consecutive checks, it is marked unhealthy and removed from the rotation until it recovers.

Health check configuration:

• Interval: how often to check (e.g., every 10 seconds).
• Threshold: how many consecutive failures before marking unhealthy.
• Timeout: how long to wait for a response.

Your application must expose a /health endpoint that:

• Returns 200 when the server can handle traffic.
• Returns a non-200 status (or times out) when it should not receive traffic — for example, during a graceful shutdown.

Session Persistence (Sticky Sessions)

By default, a load balancer may route successive requests from the same user to different servers. If your application stores session state in server memory (not in Redis), you need sticky sessions — all requests from a given client go to the same backend server.

Sticky sessions are typically implemented via a special cookie (AWSALB on AWS ALB) that encodes the target server.

Drawbacks:

• If the sticky server fails, the session is lost.
• Uneven distribution: one popular user's long session keeps one server busy.
• Complicates deployments — you cannot drain a server without disrupting its sticky users.

Recommendation: make your application stateless. Store sessions in Redis and let the load balancer route freely. This is the architecture OmniAtlas uses.

SSL/TLS Termination

Encrypting and decrypting HTTPS traffic is CPU-intensive. Instead of having each backend server handle TLS, the load balancer terminates TLS — it accepts HTTPS from clients and forwards plain HTTP to the backend servers on the internal network.

Benefits:

• Backend servers do not need SSL certificates or the CPU overhead of crypto.
• Centralized certificate management (one cert on the load balancer).
• Internal traffic stays on a private network where it is implicitly trusted.

Active-Passive vs Active-Active

Active-Passive: one load balancer handles all traffic; a second sits idle and takes over if the primary fails. Simple but wastes resources.

Active-Active: two or more load balancers share traffic simultaneously. Provides higher throughput and true redundancy. Requires DNS round-robin or an Anycast IP in front of them.

AWS ELB/ALB handles this automatically — it is internally multi-AZ and managed.