Fraud alerts, live dashboards, trending, monitoring
Tools
Spark, MapReduce, SQL in a warehouse
Flink, Kafka Streams, Spark Structured Streaming
Fixing a bug
Re-run the job on the same input
Replay the log from an older offset
Hard part
Scale and scheduling
State, time and out-of-order data
Concepts that come up
Event time vs processing time. Event time is when it happened (on the phone); processing time is when your system sees it. Events arrive late and out of order, so aggregate by event time.
Windows group events: tumbling (fixed, no overlap, e.g. each minute), sliding (overlapping, e.g. last 5 minutes updated every minute), session (closes after a gap of inactivity).
Watermark: the system's estimate that "no events older than T are still coming". It decides when a window closes. Events later than that are dropped, or sent to a side output, or corrected by a batch job. Allowed lateness keeps windows open a bit longer.
State. Counts and joins need memory across events. It lives in the operator (often RocksDB) and is checkpointed to durable storage, so a crash resumes from the last checkpoint instead of starting over.
Exactly-once results come from checkpoints + a replayable source (Kafka offsets) + an idempotent or transactional sink. Same idea as "at-least-once plus idempotent" in Message queues.
Lambda vs kappa architecture
How
Trade-off
Lambda
Two paths: a batch layer (slow, accurate) and a speed layer (fast, approximate), merged when you query.
Two codebases that must produce the same numbers. Heavy to maintain.
Kappa
One streaming path. To fix a bug or change logic, replay the log (Kafka with long retention) through the new code.
Simpler, but needs a replayable log and a stream engine you trust.
Example: trending hashtags
clicks / posts -> Kafka (partitioned by hashtag)
-> Flink: 5-minute sliding window, count per hashtag, keep top 10
-> Redis (the API reads the top 10 from here)
Same Kafka topic also lands in object storage (S3) -> nightly Spark job
recomputes exact numbers and corrects late data.
Check yourselfA phone reconnects after a tunnel and sends events that are 20 minutes old, after your per-minute window has closed. What do you do?
Aggregate by event time with a watermark plus some allowed lateness (say 5 minutes). Events later than that go to a side output or are fixed by the nightly batch job. Choose by how exact the number must be: billing needs the correction, a live dashboard can ignore the stragglers.
Summary: say it in the interview
"Batch when the answer can wait and must be exact; stream when it must be fresh. Kafka as a replayable log lets me do both from one source."