Message queues and Kafka
We'll cover the following
Queues decouple services: the producer does not wait for the consumer, a slow or dead consumer does not break the producer, and spikes get absorbed. The broker is the queue server itself (RabbitMQ or Kafka) between producer and consumer: it stores messages durably, hands them out, and tracks progress.
Claim-check pattern
Do not put big files on a queue (e.g. a 1B-record output file). Write the file to S3 and put only a small pointer message on the queue: {"file": "s3://bucket/out.parquet", "records": 1000000000}. The next service fetches the file itself.
RabbitMQ vs Kafka
| RabbitMQ | Kafka | |
|---|---|---|
| Model | Task queue, push | Durable log, consumers pull |
| After delivery | Acked, then deleted | Kept (days or forever) |
| Who tracks progress | Broker waits for acks and resends on timeout | Consumer keeps an offset (a bookmark) and commits it |
| Strengths | Simple jobs, routing | Replay, many independent consumers, huge throughput (sequential disk writes) |
- Partitions: a topic is split into partitions for parallelism. Order is guaranteed only within a partition.
- Consumer group: consumers share the partitions; each partition is read by one member of the group.
- The Kafka broker never waits on consumer acks and never resends. After a crash the consumer asks for its last committed offset and re-reads from there.
- Kafka "ack" exists on the producer side:
acks=allconfirms the write is on all in-sync replicas.
Delivery guarantees and commit timing
| Guarantee | Meaning | In Kafka |
|---|---|---|
| At-most-once | May lose messages, never duplicates | Commit offset before processing |
| At-least-once | Never lost, may be delivered twice | Commit offset after processing |
| Exactly-once | Hard and expensive, mostly Kafka-internal flows | Rare in practice |
Idempotency (the fix for duplicates)
A consumer charges $50, crashes before acking, the message is redelivered, and the user pays $100. Fix: every message carries a unique ID (payment_id). Before processing, check whether it was handled (a DB table with a unique constraint, or Redis SETNX); if yes, skip and ack. Record the ID in the same transaction as the charge, and pass an idempotency key to the payment provider too (Stripe supports it).
Rule: Kafka for event streams, high volume and replay. RabbitMQ for simple task queues with routing.
Summary: say it in the interview
- "At-least-once delivery plus idempotent consumers gives effectively-once processing."