Part 1 of 6 · 1 min read
Why this page comes first
In this module you will
- Define an event, a producer, and a consumer in a stream processing system.
- Explain why polling a database for new rows breaks down at low latency, and what a message broker offers instead.
- Tell a traditional message queue apart from a log-based broker, and say when each fits better.
- Explain partitions, offsets, and consumer groups in a log-based broker such as Kafka.
- Describe a tumbling window and say what it is for.
A warehouse answers questions about yesterday, and a MapReduce job can take hours. Some questions need an answer within seconds of something happening, such as flagging a fraudulent card swipe. Stream processing handles data as a continuous flow of events instead of a batch that waits to be run.
Remember this
A stream processing system never finishes; it keeps consuming new events and producing new output for as long as it runs.