CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 2 of 7 · 2 min read

The pipeline style

Data flows one way through small independent steps.

Think of a car factory line. Each station does one job and passes the car on. Stations do not talk back. To change the process, add or remove a station.

FSA calls this the pipeline architecture (also pipes and filters). Most developers know it from Unix shells, which Level 6 page 5 showed. Tools that use the MapReduce model follow the same shape.

Pipeline architecture

The pipeline architecture splits a process into independent steps called filters, joined by one-way channels called pipes. Data flows through the filters in order.

Example: a data import: read a file, drop bad rows, convert currencies, write to a database.

Pipes are usually one-way and point to point, for speed. They carry small payloads. Filters are self-contained, stateless and do one task. A larger task is a chain of filters. There are four types:

Filter Job
Producer Starts the flow. Output only. Also called the source
Transformer Takes input, changes some or all of it and passes it on (like map)
Tester Checks a condition, and optionally passes data on (like a filter or reduce)
Consumer Ends the flow, for example by saving to a database or showing a screen

The book's example handles service telemetry streamed through Kafka. A producer filter reads from a Kafka topic. A tester filter asks "is this about request duration?". If yes, the data goes to a transformer that calculates duration. If not, a second tester asks "is this about uptime?", and so on. The final consumer filter saves to MongoDB. Each filter knows only its own task. Adding a new metric means adding one more tester and transformer.

A worked example (made up). A pipeline gets 1,000 events a minute:

Filter Type Events in Events out
Capture Producer 0 1,000
Duration test Tester 1,000 600 duration events (400 pass on)
Duration calc Transformer 600 600
Uptime test Tester 400 150 uptime events (250 end here)
Uptime calc Transformer 150 150
Save Consumer 750 0

Typical uses are ETL (extract, transform, load) tools, electronic data interchange converters, and orchestration tools such as Apache Camel.

FSA says the pipeline is a technically partitioned style, since the logic is split by filter type. It is usually one deployment, so it is a monolith with one architectural quantum (one independently deployable unit with its own characteristics).

Architecture quantum

An architecture quantum is an independently deployable part of a system with high functional cohesion that includes everything needed to run, such as its database. Two parts that share one database are one quantum.

Example: services A and B that share a single database belong to a single quantum.

Check yourself:

  • Which filter type would drop events that do not match a rule?
  • Why does adding a step to a pipeline rarely disturb the other steps?