Ordering, acks and delivery guarantees

Any system that moves messages (chat, notifications, event streams) has to answer four questions: in what order, what counts as saved, what counts as delivered, and what happens on a retry.

• Device clocks can be wrong or set by the user. Server clocks drift by milliseconds between machines.
• Two messages can arrive within the same millisecond on different servers, so the timestamps tie or invert.
• A timestamp cannot tell you that a message is missing.

• Pick the unit that needs a shared order (one chat, one account, one partition). Hash its id to one owning shard.
• That shard is the only thing that assigns a growing seq (1, 2, 3...) for the unit. No clock is involved.
• The receiver spots a gap (saw 3 then 5) and asks for 4. A gap is detectable, which a timestamp never is.
• Load is spread across thousands of units. Each unit sends few messages per second, so a counter per unit is trivial.
• Kafka gives the same thing: a partition keyed by the unit id is an ordered log with offsets.

• The owner replicates the message to followers and acks only after a majority has it (2 of 3).
• A crash right after the ack cannot lose it: any new leader needs a majority, and every majority contains a copy.
• Acking after only the leader write risks a tick for a message that exists nowhere.
• Why not wait for all replicas? One dead or slow replica would block every write. A majority tolerates one failure.
• This is CP for that one unit: during a leader election that unit pauses for a moment, while everything else stays available.

• The network can drop the ack, so the client retries. The client generates an idempotency key (client_msg_id) per message.
• The owner checks the key before assigning a seq. On a retry it returns the same seq and the same ack as the first attempt, not an error.
• At-least-once delivery plus dedupe on the key gives effectively-once behaviour.

• Store first, ack the sender second, deliver third. Never skip the store because the receiver looks online: a socket can look alive while the phone is already off the network.
• Handing a message to a socket is not delivery. Delivery is the receiver device sending back an ack.
• Delete the stored copy only after that ack (or when its retention TTL expires).
• On reconnect the receiver sends a cursor per unit ("last seq I have is 41") and the server replays 42 onward. Each device keeps its own cursor.