CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 2 of 7 · 1 min read

Three kinds of system

Services answer requests, batch jobs crunch a fixed input, stream jobs react to events.

Think of three ways to get a meal. A waiter takes your order and brings food in minutes. A catering firm takes a big order today and delivers tomorrow. A conveyor-belt sushi bar puts plates by as the chef makes them. All feed people, with different expectations.

DDIA separates three kinds of system:

Kind Takes in Produces Measured mainly by
Service (online) One request at a time A reply Response time and availability
Batch job (offline) A large, fixed set of data Output data Throughput
Stream job (near real time) Events soon after they happen Output events Delay

Batch processing

Batch processing is running a job over a large, fixed amount of input data and producing output data. Nobody waits for the answer, and jobs often run on a schedule, such as once a day.

Example: every night, a job reads all of the day's page views and writes a report of the 10 most viewed pages.

Throughput

Throughput is how much work is finished per unit of time, such as gigabytes of input processed per hour.

Example: a job that processes 600 GB in 2 hours has a throughput of 300 GB per hour.

Batch is not new. DDIA reminds readers that punch card machines in the 1890 US census computed totals from large inputs, and MapReduce looks much like the card-sorting machines of the 1940s and 1950s. The book also calls MapReduce a fairly low-level model, and says its importance is declining, but that it is worth knowing because it shows why and how batch processing is useful.

Check yourself:

  • Why is response time the main measure for a service but throughput for a batch job?
  • Where does a stream job sit between the other two?