CSE Roadmap

Specialist tier · about 2 hours · 6 parts

Part 1 of 6 · 1 min read

Why this page comes first

In this module you will

  • Define an event, a producer, and a consumer in a stream processing system.
  • Explain why polling a database for new rows breaks down at low latency, and what a message broker offers instead.
  • Tell a traditional message queue apart from a log-based broker, and say when each fits better.
  • Explain partitions, offsets, and consumer groups in a log-based broker such as Kafka.
  • Describe a tumbling window and say what it is for.

A warehouse answers questions about yesterday, and a MapReduce job can take hours. Some questions need an answer within seconds of something happening, such as flagging a fraudulent card swipe. Stream processing handles data as a continuous flow of events instead of a batch that waits to be run.

Remember this

A stream processing system never finishes; it keeps consuming new events and producing new output for as long as it runs.