Mock design: notification system

Prompt: send push, email and SMS to 100M users. The interviewer gave these numbers when asked.

Trigger (API / campaign)
  -> Notification service
  -> priority queues (otp | transactional | marketing)
  -> channel workers (push / email / SMS)
  -> providers (APNs, FCM, SES, Twilio)

• Partitions do not give priority inside one queue. If OTPs and marketing share a topic, a 100M backlog sits in front of the OTP (head-of-line blocking). More consumers just drain the whole backlog faster.
• Use separate queues or topics per priority (otp, transactional, marketing), each with its own consumer pool.
• Throttle marketing to provider rate limits and spread the send over 30-60 minutes. Blasting 170K/s gets you throttled or blocked by APNs, FCM and email/SMS providers.
• Give OTP reserved capacity, such as a dedicated provider route or SMS sender, so marketing can never use up the quota.
• The load balancer matters little here; the interesting design is behind it.

1. The click creates one campaign job, not 100M requests.
2. A fan-out worker splits users into chunks (say 1,000 user IDs per chunk, by ID range) and puts each chunk on a queue.
3. Many workers take chunks in parallel and expand each into per-user messages.
4. Each chunk is checkpointed, so if a worker dies the job resumes instead of restarting or double-sending.

• Opt-in and preferences: unsubscribed, or marketing push turned off? Legally required (GDPR, CAN-SPAM).
• Channel eligibility: valid push device token, verified email, phone number present.
• Frequency cap: no more than N marketing messages per user per day.
• Quiet hours and time zone: no sale push at 3am local time.
• Dedupe: has this user already received this campaign? Key on campaign_id + user_id.
• Active users first (recently used before rarely used), a good marketing optimization.
• Preferences and caps live in a fast store (Redis, or a DB with a cache): 170K checks/s cannot hit a slow DB.

A provider timeout is ambiguous: the push might have been sent and the reply lost, or not sent at all.

1. Retry transient failures (timeouts, 5xx) with exponential backoff plus jitter (1s, 2s, 4s, with randomness), capped at 3-5 attempts.
2. Do not retry permanent failures (invalid device token, unsubscribed). Mark the token dead and drop it.
3. After max attempts, send to a dead-letter queue (DLQ) for inspection or replay.
4. Every notification has a unique notification_id. Before sending, record it as "sending" with an atomic conditional insert (Redis SET NX or a DB unique key); if it exists, it is a retry or duplicate. After the provider confirms, mark it "sent".
5. Pass the id to the provider as an idempotency or collapse key where supported. APNs and FCM collapse IDs make a repeated push replace the earlier one.

A worker crashes after the provider sent the push but before marking it "sent". On retry it sends again, so the user gets a duplicate. You cannot fully solve this (it is unknowable whether the provider received it), so choose a policy per type:

A rare duplicate is usually acceptable, since it needs a crash in a tiny window. What is unacceptable is a design that creates duplicates routinely, like retrying with no dedupe.

• Data model for preferences and device tokens.
• Monitoring delivery rates, latency per priority, DLQ depth.
• Multi-region.