System Design Case Studies

Designing a Chat System

The hard problem is not sending a message — it is delivering it in real time to a recipient who might be offline, connected to a different server, or on multiple devices at once.

Requirements to size the design: 1:1 and group messaging, online presence, and at-least-once delivery with per-conversation ordering (global ordering is not needed)
WebSockets, with long-polling as a fallback, keep a persistent connection open for real-time delivery — plain HTTP request/response cannot push to an idle client
A connection is pinned to one server instance; delivering to a recipient connected elsewhere needs a pub/sub layer that fans a message out to whichever server holds that connection
Message storage is closer to a per-conversation append-only log, partitioned by conversation ID, than to a mutable relational table
Offline delivery: undelivered messages queue server-side and flush on reconnect or push notification; delivery and read receipts are their own small state machine per message
Group chat fan-out to a large group is a miniature version of the same push-vs-pull tradeoff as a social feed

High-level architecture

Cross-server delivery

1:1 vs group chat
	1:1 chat	Large group chat
Delivery path	one pub/sub hop to one recipient	fan-out to every online member
Storage shape	per-conversation log, two participants	per-conversation log, many readers
Scaling concern	minimal	fan-out cost scales with group size

Routing a message to its recipient

@OnMessage
public void onMessage(ChatMessage msg, Session session) {
    messageStore.append(msg.conversationId(), msg);
    String targetServer = presence.serverFor(msg.recipientId());
    if (targetServer == null) {
        offlineQueue.enqueue(msg.recipientId(), msg);   // recipient offline — deliver on reconnect
    } else {
        pubSub.publish(targetServer, msg);
    }
}