Part 3 of 8 · 3 min read
What stream processing is used for
Alerts, running statistics, live copies of data and searching incoming items.
Think of a night guard, a scoreboard operator and a post clerk. The guard watches for one odd pattern. The scoreboard operator keeps running totals. The clerk files each item in the right pigeonhole as it arrives. A stream job can play all three roles.
DDIA lists the uses:
Monitoring and complex event processing (CEP). Banks watch for stolen cards, trading systems watch prices, factories watch machines. CEP lets you describe a pattern of events, like a regular expression for events, often in an SQL-like language. The engine keeps a state machine and emits a complex event when a match occurs. Note the reversal compared with a database. A database stores data and forgets the query once answered. A CEP engine stores the query long-term and lets events flow past it.
Stream analytics. Here the interest is in statistics over many events: the rate of an event, a rolling average, or a comparison with the same time last week. These are computed over a window, a time interval such as the last 5 minutes. Averaging over a few minutes smooths out second-by-second noise while staying timely. The book notes that some analytics systems use approximate algorithms such as Bloom filters (to test membership) and HyperLogLog (to estimate the number of different items) because they use far less memory. This does not mean streams are inexact. The approximation is an optimisation.
Maintaining materialized views. The change streams of page 7 keep caches, indexes and warehouses up to date. A materialized view is a stored result of a query that is updated when the underlying data changes. In event sourcing (page 4), the application state is such a view. Unlike analytics, this often needs all events, with a window reaching back to the beginning of time.
Search on streams. Sometimes you need to match single events against complex criteria, such as a full-text query. A news monitoring service stores queries such as "company X" and runs each new article past them. A property site stores your saved search and tells you when a new matching listing appears. Normal search indexes documents and runs queries over the index. Stream search stores the queries and lets documents flow past.
The book also compares stream processing with message passing between services (actors, RPC). Actor systems manage concurrency, often use temporary one-to-one communication and allow cycles. Stream processing is a data management technique with durable, multi-subscriber logs and usually has acyclic pipelines.
A worked example (made up). A fraud rule: "alert when one card makes 3 or more payments in 10 minutes in 2 different countries". Events:
| Time | Card | Country |
|---|---|---|
| 10:00 | 55 | UK |
| 10:03 | 55 | FR |
| 10:07 | 55 | FR |
| 10:20 | 55 | FR |
At 10:07 the engine has seen 3 payments within 10 minutes (10:00, 10:03, 10:07) in 2 countries and emits an alert. At 10:20 the first events are outside the 10-minute window, so they are forgotten.
Check yourself:
- How does a CEP engine reverse the roles of data and queries?
- Why does a materialized view for event sourcing need a window back to the beginning of time?