Vinay C.
System Design Interview4 min read

Mock design: chat system

Lesson 2.2 · 4 min read · by Vinay C

We'll cover the following

Prompt: 1:1 and group chat, 500M daily active users, groups up to 1,000 members, media, end-to-end encrypted, global with EU and India data residency. Answers when asked: online delivery under 500 ms p99, no loss after the ack, same order for everyone, server keeps messages up to 30 days, no backdoor.

Requirements: how to split them

Functional (what users do)Non-functional (how well, with a number)
Send and receive 1:1 and group messagesUnder 500 ms online, p99
Send mediaNo loss after the ack
Sent, delivered and read statusSame order for everyone in a conversation
Catch up after being offline30-day retention, regional residency, end-to-end encryption

Numbers

ItemValue
Messages500M users x 10 per day = 5B per day
QPS5B / ~10^5 seconds = ~50K/s (58K exact). Peak 2-3x, so ~150K/s
Text volume5B x 100 B = 500 GB per day, more with ciphertext and metadata
Media10% images x 2 MB: 5x10^8 x 2x10^6 B = 10^15 B = about 1 PB per day raw
ConnectionsDAU is not concurrent. Peak 30-50% = ~200M persistent connections
Gateways~200K connections per box = ~1,000 boxes (1,200-1,500 with redundancy)

Flow

Sending to a recipient who is offline

Sequence between Alice phone, a gateway, the conversation shard, the push service and Bob phone. Alice sends an encrypted message to the gateway, which forwards it to the conversation shard. The shard assigns the next sequence number, replicates it to a majority and acks, so Alice sees one tick. Bob is offline, so the shard asks the push service for a content-free wake-up. Bob phone wakes, connects and sends its cursor 41. The shard replays message 42, Bob acks and Alice sees two ticks.

Alice phoneencrypts for Bob
GatewayWebSocket
Conversation shardowner of the seq
APNs / FCMwake-up only
Bob phonewas asleep
  1. Send and save
  2. Alice phone → Gatewaysend(ciphertext, client_msg_id)
  3. Gateway → Conversation shardforward to the owner of this conversation
  4. Conversation shardAssign seq=42, replicate to a majority
  5. Conversation shard → Alice phoneack: one tick
  6. Bob is offline
  7. Conversation shard → APNs / FCMcontent-free wake-up for Bob devices
  8. APNs / FCM → Bob phonewake the app
  9. Bob syncs
  10. Bob phone → Conversation shardconnect with cursor 41
  11. Conversation shard → Bob phonereplay seq=42
  12. Bob phoneDecrypt locally, show notification text
  13. Bob phone → Conversation shardack 42
  14. Conversation shard → Alice phonedelivered: two ticks
Single tick
The message is stored on a majority of replicas.
Double tick
The receiver device has acked, which is the only proof of delivery.

Where the general lessons live

  • Keys, groups and media: End-to-end encryption and key management.
  • Order, acks, duplicates, cursor replay: Ordering, acks and delivery guarantees.
  • Waking a sleeping phone: Protocols, real-time options and pagination.
  • The arithmetic traps: Back-of-envelope estimation.

Data model, groups, regions

  • messages keyed by conversation id and seq: ciphertext, client_msg_id, sender, type, media reference, 30-day TTL. Stored once per conversation, never once per recipient.
  • device_cursor: per user, device and conversation, the last acked seq. devices: push token and public key.
  • A presence registry (for example Redis): user and device to gateway, with a TTL refreshed by heartbeat.
  • Group fan-out: one ciphertext, one seq. Look up each online member gateway, group by gateway and send one batch per gateway. Offline members get a push plus cursor sync. Cap the group size because deliveries multiply.
  • Regions: each user has a home region (EU and India stay local), GeoDNS sends them to the nearest gateway, and a conversation owner lives in one region. Replicate synchronously across zones and asynchronously across regions, and say what the async gap does to the recovery point.

Where this mock lost points

  • Reversed the key roles in end-to-end encryption.
  • Timestamps for ordering and a blanket "availability over consistency".
  • Stored a message only when the recipient looked offline.
  • Put message text in the push payload, which breaks the encryption requirement.
  • QPS off by 1000x, assumed 95% concurrency, forgot media.
  • No-loss and ordering were missing from the non-functional list.

Summary: say it in the interview

  • "What made this problem special? Ordering and no-loss per conversation, end-to-end encryption, and a push that is only a hint."