Part 2 of 7 · 2 min read
The pipeline style
Data flows one way through small independent steps.
Think of a car factory line. Each station does one job and passes the car on. Stations do not talk back. To change the process, add or remove a station.
FSA calls this the pipeline architecture (also pipes and filters). Most developers know it from Unix shells, which Level 6 page 5 showed. Tools that use the MapReduce model follow the same shape.
Pipeline architecture
The pipeline architecture splits a process into independent steps called filters, joined by one-way channels called pipes. Data flows through the filters in order.
Example: a data import: read a file, drop bad rows, convert currencies, write to a database.
Pipes are usually one-way and point to point, for speed. They carry small payloads. Filters are self-contained, stateless and do one task. A larger task is a chain of filters. There are four types:
| Filter | Job |
|---|---|
| Producer | Starts the flow. Output only. Also called the source |
| Transformer | Takes input, changes some or all of it and passes it on (like map) |
| Tester | Checks a condition, and optionally passes data on (like a filter or reduce) |
| Consumer | Ends the flow, for example by saving to a database or showing a screen |
The book's example handles service telemetry streamed through Kafka. A producer filter reads from a Kafka topic. A tester filter asks "is this about request duration?". If yes, the data goes to a transformer that calculates duration. If not, a second tester asks "is this about uptime?", and so on. The final consumer filter saves to MongoDB. Each filter knows only its own task. Adding a new metric means adding one more tester and transformer.
A worked example (made up). A pipeline gets 1,000 events a minute:
| Filter | Type | Events in | Events out |
|---|---|---|---|
| Capture | Producer | 0 | 1,000 |
| Duration test | Tester | 1,000 | 600 duration events (400 pass on) |
| Duration calc | Transformer | 600 | 600 |
| Uptime test | Tester | 400 | 150 uptime events (250 end here) |
| Uptime calc | Transformer | 150 | 150 |
| Save | Consumer | 750 | 0 |
Typical uses are ETL (extract, transform, load) tools, electronic data interchange converters, and orchestration tools such as Apache Camel.
FSA says the pipeline is a technically partitioned style, since the logic is split by filter type. It is usually one deployment, so it is a monolith with one architectural quantum (one independently deployable unit with its own characteristics).
Architecture quantum
An architecture quantum is an independently deployable part of a system with high functional cohesion that includes everything needed to run, such as its database. Two parts that share one database are one quantum.
Example: services A and B that share a single database belong to a single quantum.
Check yourself:
- Which filter type would drop events that do not match a rule?
- Why does adding a step to a pipeline rarely disturb the other steps?