System Design Foundations
Scalability Fundamentals
Scaling vertically (a bigger machine) always hits a ceiling; scaling horizontally (more machines) is the only path past it — but it turns every design problem into a distributed one.
- Vertical scaling: more CPU/RAM/disk on one box — simple, no distributed-systems complexity, but bounded by the biggest machine money can buy
- Horizontal scaling: more machines sharing the load — unbounded in principle, but requires load balancing, partitioning, and coordination
- Latency is the time for one request; throughput is requests handled per unit time — a system can improve one while making the other worse
- Averages hide the story: always reason in percentiles (p50/p99/p999) — a p99 ten times the p50 means 1-in-100 users have a bad time
- Little's Law: average concurrent requests = throughput × average latency — a fixed worker pool caps throughput once latency rises
- Most real systems are throughput-bound in one component (a database) and latency-bound in another (a synchronous downstream call) at the same time
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| Mechanism | Bigger machine | More machines |
| Ceiling | Hard limit (largest available hardware) | No inherent limit |
| Complexity added | None — same single-node code | Load balancing, partitioning, coordination, partial failure |
| Failure mode | One machine, one point of failure | Redundancy possible — a node can die without an outage |
| Typical use | Databases with strong consistency needs, legacy monoliths | Stateless services, web tiers, most modern backends |
A system that reports "average latency: 80ms" can still be unusable for a meaningful slice of users. If p50 is 80ms but p99 is 4 seconds, 1% of requests — potentially thousands per minute at scale — are hitting that slow path. Percentiles are what SLAs are built on (Availability And Slas), not averages.
static long percentile(long[] latenciesMs, double p) {
long[] sorted = latenciesMs.clone();
Arrays.sort(sorted);
int index = (int) Math.ceil(p / 100.0 * sorted.length) - 1;
return sorted[Math.max(index, 0)];
}
// percentile(samples, 50) -> p50, percentile(samples, 99) -> p99