CSE Roadmap

Specialist tier · about 2 hours · 6 parts

Part 5 of 6 · 2 min read

Consumer groups and windows

Consumer groups and windows

Think of a consumer group as several cashiers sharing one long line of customers by splitting the line into fixed lanes, each cashier serving only their own lane from front to back.

In plain words: A consumer group lets several consumer processes share the work of reading one topic by assigning each partition to exactly one consumer in the group at a time. The number of consumers that can usefully work on one topic in parallel is capped by the number of partitions, since a single partition is only ever read by one member of the group. Many streaming jobs also need to summarize events over time rather than process them one by one forever; a tumbling window groups events into fixed, non-overlapping time slices, such as every one-minute period, and produces one summary per slice once that slice closes.

A worked example. The bookstore's "page-views" topic has six partitions and a consumer group of three processes computing trending books; each process is assigned two partitions. A separate tumbling-window job counts page views per book in each one-minute window, emitting one count per book every minute.

The key lines.

# A one-minute tumbling window, grouping events by floor(timestamp / 60)
def window_key(event):
    return event["viewed_at_epoch"] // 60

Consumer group, tumbling window

A consumer group is a set of consumer processes that share the work of reading a topic by each taking responsibility for a distinct subset of its partitions. A tumbling window is a fixed-length, non-overlapping time interval used to group events for a periodic summary, with every event belonging to exactly one window.

Example: the one-minute window function above assigns every page-view event to exactly one 60-second bucket, with no overlap between buckets.

Quick notes

  • An event is a small, immutable, timestamped record; producers send events, consumers receive them, and a topic groups related events.
  • Polling a database works for occasional checks but grows wasteful as the check interval shrinks; message brokers push notifications instead.
  • A traditional queue typically deletes a message once delivered; a log-based broker keeps an append-only log a consumer can replay.
  • A topic's log is split into partitions for scale; each message gets an offset, so a consumer tracks its place with one number per partition.
  • A consumer group splits a topic's partitions across several consumer processes, capping useful parallelism at the partition count.
  • A tumbling window groups events into fixed, non-overlapping time slices for periodic summaries.