System Design Basics

When you interview at a top tech company or join a team building a product that needs to serve millions of users, you will be expected to reason about system design. This guide introduces the core vocabulary and trade-offs that underpin every large-scale system.

What Is System Design?

System design is the process of defining the architecture, components, data flows, and interfaces of a system to satisfy specified requirements. It sits at the intersection of software engineering and infrastructure: you are not just writing code, you are deciding how the pieces fit together under real-world constraints.

Key Properties of a Well-Designed System

Scalability

Scalability is a system's ability to handle growing load — more users, more data, more requests per second — without a proportional increase in cost or latency.

Vertical scaling (scaling up) means adding more resources to a single machine: faster CPUs, more RAM, larger disks. It is simple but hits a hard ceiling. A single server can only be so big, and it is a single point of failure.

Horizontal scaling (scaling out) means adding more machines to share the load. This is how the web's largest systems work. Horizontal scaling requires stateless application servers (session state lives in a shared cache or database, not in the server's memory) and a load balancer to distribute traffic.

Rule of thumb: Design for horizontal scaling from day one. Retrofitting a stateful monolith to scale horizontally is expensive.

Reliability

Reliability is the probability that a system performs its intended function correctly over a given time period. A reliable system handles hardware failures, software bugs, and network partitions gracefully — it does not corrupt data or silently return wrong results.

Key reliability techniques:

• Redundancy: run multiple replicas of critical components so failure of one does not halt the system.
• Idempotency: design write operations so retrying them has no additional effect (e.g., Stripe uses idempotency keys on payment API calls).
• Checksums: verify data integrity when reading from disk or network.

Availability

Availability is the fraction of time a system is operational. It is often expressed as "nines":

Availability is achieved through redundancy, health checks, automatic failover, and careful deployment practices (canary releases, blue-green deployments).

Latency vs. Throughput

• Latency is the time from sending a request to receiving a response (milliseconds). Users notice latency above ~100 ms.
• Throughput is the number of requests a system can process per unit of time (requests/second).

They are related but not identical. A pipeline with many parallel workers can have high throughput even if individual tasks take seconds. Optimizing for latency (e.g., caching) often improves throughput too, but not always.

The Classic Trade-Off: CAP Theorem

In a distributed system, you can guarantee at most two of these three properties simultaneously:

• Consistency: every read returns the most recent write.
• Availability: every request receives a response (not an error).
• Partition tolerance: the system continues operating despite network partitions.

Because network partitions are a reality (cables break, routers fail), practical distributed systems choose between CP (consistent + partition-tolerant, sacrifices availability) and AP (available + partition-tolerant, sacrifices strict consistency). We cover CAP Theorem in depth in a separate subject.

Common Components

Load Balancers

A load balancer sits in front of a pool of servers and routes each incoming request to one of them. Common strategies:

• Round-robin: requests go to each server in turn.
• Least connections: route to the server with fewest active connections.
• IP hash: always route a given client IP to the same server (useful for sticky sessions).

Caches

Caches store the results of expensive computations (database queries, API calls) so future requests can be served faster. Key decisions:

• Cache-aside (lazy loading): the application checks the cache first; on a miss, it loads from the DB and populates the cache.
• Write-through: writes go to the cache and the DB simultaneously.
• TTL (time-to-live): how long cached data is valid before expiring.

Redis and Memcached are the industry standard in-memory caches.

Databases

Relational databases (PostgreSQL, MySQL) enforce a strict schema, support ACID transactions, and have powerful query capabilities. They are the right default for most applications.

NoSQL databases (Cassandra, DynamoDB, MongoDB) sacrifice some ACID guarantees for horizontal scalability and flexible schemas. Use them when your access patterns and scale genuinely demand it — not just because they sound modern.

Message Queues

A message queue (SQS, Kafka, RabbitMQ) decouples the producer of a task from the consumer. The producer enqueues a job (e.g., "send this email") and continues immediately; a worker processes the job asynchronously. This improves availability (the producer never blocks), enables retries, and allows independent scaling of producers and consumers.

Design Principles to Internalize

1. Build for failure: assume every component will fail. Use retries with exponential backoff, circuit breakers, and fallbacks.
2. Measure everything: you cannot optimize what you do not measure. Instrument latency, error rates, and throughput on every critical path.
3. Start simple, scale when needed: premature optimization leads to complexity without benefit. Add caching, sharding, and message queues only when profiling shows they are needed.
4. Data is the hardest part: migrating data is painful. Model your data carefully upfront and plan schema migrations.
5. Stateless services, stateful data layers: keep application servers stateless so any instance can serve any request. Push state into databases, caches, and queues.