Design an In-Memory Pub-Sub Message Queue With Offsets
In a 45-minute LLD interview, this problem evaluates append-only log architectures, message immutability, and decoupled consumer offset tracking (inspired by Apache Kafka). Scope the solution to: an in-memory Broker acting as Mediator, topics owning immutable sequential commit logs, independent per-subscriber offset pointers, pull-based (poll) batch consumption, and replayability by cursor rewinding. Multi-broker network consensus (Raft/Paxos) can be cited as a distributed system extension.
Problem statement & clarifying requirements
Design an in-memory distributed message queue inspired by Apache Kafka and RabbitMQ. The system organizes messages into distinct Topics, appends published events to an immutable commit log, and maintains independent consumer offsets. Slow consumers must never block fast consumers, and consumers must be able to replay past events by rewinding their offset cursor back to any point in the log.
No. Topics use an append-only log with separate per-subscriber offset pointers. Each consumer polls and advances its own read index independently.
Yes. Messages are not deleted upon consumption. A consumer can call resetOffset(topic, subscriberId, 0) to replay all past messages.
The architecture implements a pull-based (poll) model mediated by the Broker, giving consumers direct control over batch size and consumption pacing.
The Broker acts as a Mediator: topic append operations are synchronized with internal mutexes, preventing producers and consumers from coupling directly to each other.
First, sketch the design yourself
Spend 5 minutes listing the classes, who owns whom, and which interface hides the part that changes.
- 1.
Traditional queues (like simple FIFOs) pop elements upon receipt, preventing multiple independent subscriber groups and making replay impossible.
- 2.
Decouple publishers from consumers via a central Broker Mediator: publishers only know topic names and payloads; they never hold references to downstream worker instances.
- 3.
Treat the commit log as an immutable sequence of numbered entries: once appended, a message's content and offset never change.
How to think through this problem (5-step walkthrough)
In a 45-minute LLD interview, this problem evaluates append-only log architectures, message immutability, and decoupled consumer offset tracking (inspired by Apache Kafka). Scope the solution to: an in-memory Broker acting as Mediator, topics owning immutable sequential commit logs, independent per-subscriber offset pointers, pull-based (poll) batch consumption, and replayability by cursor rewinding. Multi-broker network consensus (Raft/Paxos) can be cited as a distributed system extension.
Not every noun in the prompt deserves its own class. Separate dumb data structs from stateful managers, and turn trivial nouns into fields or enums.
BrokerCentral Mediator managing topic registries, subscriber mappings, topic creation, publication, and per-subscriber cursor offsets without direct publisher-to-consumer coupling.
TopicEncapsulates an append-only thread-safe vector of Message instances and provides sub-slice batch reads by offset range under mutex protection.
MessageImmutable value struct representing a committed record with sequential offset (int64_t), payload string, and optional routing key.
ISubscriberObserver interface declaring subscriber identity and onMessage(topic, message) callback contract.
WorkerSubscriberConcrete consumer worker that polls message batches from the Broker, processes payloads, and tracks received message history.
PerConsumerQueueCopyTrap noun — beginner candidates often allocate a separate std::queue<Message> for each subscriber and duplicate every published message N times. This balloons memory by O(N × M) and prevents historical seeking; the correct architecture is a single append-only topic log with a lightweight scalar integer offset cursor per consumer.
A traditional naive message queue deletes messages as soon as a consumer dequeues them (pop()). This prevents multiple consumer groups from reading the same topic, makes historical message replay impossible, and causes slow consumers to either block fast consumers or exhaust memory with unbound subscriber buffers. Coupling publishers directly to subscriber instances also creates cascading failure when a consumer crashes.
Direct publisher-to-subscriber calls couple services together, creating circular dependencies and synchronous bottlenecks.
Tightly coupling message consumers to concrete processing logic prevents adding analytics loggers or audit workers without touching publisher code.
Unsynchronized appends to shared message vectors cause data races, offset collision, and iterator invalidation.
- ✓
Append-only log immutability: Once appended to a
Topic, aMessageand its assigned offset index never change or get deleted during standard consumption, allowing safe concurrent reads and rewinds. - ✓
Independent consumer offset isolation: Each consumer maintains its own independent offset cursor (C_i ≥ 0). A slow consumer lagging behind does not delay or block fast consumers reading higher offsets on the same topic.
- ✓
Replayability invariant: Any subscriber can rewind its cursor to an earlier valid offset (
0 <= targetOffset <= latestOffset) viaBroker::resetOffset()to reprocess historical messages from that point onward.
Entities & design patterns used
Unlock Full UML Diagram & C++23 Solution
The complete UML diagram, production-ready C++23 code, follow-up extensions, and design pitfall analysis are part of Brute2Best Pro. All 7 foundation chapters are 100% free.
First anchor question per pattern, Part 1 LLD & Module 1 C++ basics are 100% Free forever.