Part 4 of 6 · 4 min read
The code
Append, read from an offset, retry.
The topic and the message:
record Message(long offset, String topic, String key, String body) {}
class Topic {
final String name; private final List<Message> log = new ArrayList<>();
synchronized Message append(String key, String body) {
Message m = new Message(log.size(), name, key, body); // the offset is the position
log.add(m);
return m;
}
synchronized List<Message> readFrom(long offset, int max) {
int from = (int) Math.min(offset, log.size());
return new ArrayList<>(log.subList(from, Math.min(log.size(), from + max)));
}
}
The consumer reads a batch, calls the subscriber for each message, retries failures, and then moves its position:
class Consumer {
final Subscriber sub; final Topic topic; private long next;
final AtomicInteger failures = new AtomicInteger();
final List<Message> dead = new ArrayList<>(); final int maxRetries;
int poll() {
int done = 0;
for (Message m : topic.readFrom(next, 100)) {
boolean ok = false;
for (int attempt = 0; attempt <= maxRetries && !ok; attempt++) {
try { sub.onMessage(m); ok = true; }
catch (Exception e) { failures.incrementAndGet(); }
}
if (!ok) dead.add(m); // give up on this one, keep going
next = m.offset() + 1; done++;
}
return done;
}
void seek(long o) { next = o; }
}
The position moves only after the message is handled or set aside. If the program stops half way through a batch, the position still points at the first unhandled message, so nothing is skipped. The price is that the last message may be handled twice after a restart. That is at-least-once delivery, and it is the usual promise, since "exactly once" needs extra machinery. Subscribers should therefore be safe to run twice on the same message (an idempotent handler).
The broker is a registry of topics and consumers:
class Broker {
private final Map<String, Topic> topics = new ConcurrentHashMap<>();
private final Map<String, Consumer> consumers = new ConcurrentHashMap<>();
Topic topic(String n) { return topics.computeIfAbsent(n, Topic::new); }
Message publish(String topic, String key, String body) { return topic(topic).append(key, body); }
Consumer subscribe(String topic, Subscriber s, int retries) {
return consumers.computeIfAbsent(topic + "/" + s.name(), k -> new Consumer(s, topic(topic), retries));
}
}
A run. Two subscribers, billing and email, follow the topic "orders". Messages o1 and o2 go to "orders", and x goes to "news".
| Step | Result |
|---|---|
| Publish o1, o2 to orders and x to news, then poll all | Billing and email each saw 0:o1 and 1:o2. Nobody saw x |
| A third subscriber, audit, joins and o3 is published, then poll | Billing saw offsets 0, 1, 2. Audit also saw all three, since new subscribers here start at offset 0 |
| Audit seeks to 0 and polls again | Audit's list now holds all three messages twice. This is the replay, and it shows at-least-once behaviour |
| A flaky subscriber fails on o2 twice, with 2 retries allowed | 2 failures, 0 dead, position 3. The third try worked |
| A broken subscriber always fails on o2, with 1 retry allowed | 2 failures, 1 dead message, position 3. It moved on to o3 |
| 8 publishers each send 1,000 messages at once | 8,000 messages, 8,000 unique offsets, and a consumer sees all 8,000 |
Where a new subscriber should start is a choice. Starting at offset 0 gives it the full history. Starting at the end gives it only new messages. A real broker makes this a setting per subscriber.
Check yourself: 1. Why is the position moved only after the message is handled? 2. What is at-least-once delivery? 3. Why should a subscriber's handler be idempotent?