CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 4 of 8 · 3 min read

Time: event time and processing time

When something happened and when it was processed are different, and mixing them gives wrong answers.

Think of postcards from a holiday. You receive three on the same day, but they were written on different days. The date on the postcard is when the writer wrote it. The date you read it is when you processed it. If you want a diary of the trip, you use the writing date.

DDIA explains the problem. A batch job reading a year of events in a few minutes must use the timestamp inside each event. The time on the machine running the job has nothing to do with when events happened. Using event timestamps also keeps the job deterministic: the same input gives the same result.

Many stream frameworks use the local clock of the machine, the processing time. This is simple and fine when the delay between an event and its processing is tiny. But delays can be large: queueing, network faults, a restart of the consumer, or reprocessing old events (page 3). Delays also change the order. If a user's first request goes through web server A and the second through B, and B's event reaches the broker first, the processor sees them in the wrong order.

Event time

Event time is the moment an event actually happened, as recorded in the event itself.

Example: a sensor reading carries the timestamp 10:03:07, set when the sensor took the reading.

Processing time

Processing time is the moment the stream processor handles the event, according to its own clock.

Example: the same reading reaches the processor at 10:03:09 and is processed then.

The book gives an example of what goes wrong. A job measures requests per second by processing time. It is shut down for a minute during a redeploy. When it starts again, it processes the backlog, and the chart shows a huge spike, although the real request rate was flat.

Real time Real requests per second Chart by processing time
10:00 to 10:01 100 100
10:01 to 10:02 (job down) 100 0
10:02 to 10:03 (backlog) 100 200 or more

When is a window complete? With event time, you can never be sure that all events for a window have arrived. When has the 37th minute of the hour ended? Some events may be buffered somewhere. Late events are called stragglers. The book gives two options: ignore them, tracking the number of dropped events and alerting if too many; or publish a correction to the window, which may mean retracting the earlier output. Sometimes a special message says "no more events with a timestamp before t will come", but with many producers each has to be tracked.

Whose clock? A mobile app may store events offline and send them hours later. The right timestamp is the time of the action on the device's clock, but the device's clock may be wrong. The server's clock is more trustworthy but says when the event arrived. The book suggests logging three timestamps: when the event happened (device clock), when it was sent (device clock) and when it was received (server clock). Subtracting the second from the third estimates the device's clock error. Applying that offset to the first estimates the real time.

A worked example (made up). A phone logs an event at its clock 14:00, sends it at its clock 14:05, and the server receives it at 14:20 server time.

Quantity Calculation Result
Estimated clock offset 14:20 - 14:05 15 minutes (device clock is 15 minutes behind)
Estimated real time of the event 14:00 + 15 minutes 14:15

This assumes the network delay is small compared with the accuracy needed, and that the offset did not change in between.

Common mistake

Windowing by processing time and treating the result as real behaviour. After a restart or a backlog, rates spike and the data lies.

Check yourself:

  • A phone event says 09:00 and arrives at 11:30. Which time should a daily count use?
  • What two choices exist for an event that arrives after its window was closed?