CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 5 of 8 · 2 min read

Pipeline parts

Six components.

The book names six parts.

  1. Preprocessor. It splits the video into small chunks, called groups of pictures (GOP), each a few seconds long and playable by itself. For old clients that cannot split, the server does it. It builds the DAG from configuration files written by developers. It also stores chunks and metadata in temporary storage, so a failed encoding can be retried from saved data.
  2. DAG scheduler. It divides the graph into stages and places each stage's tasks on a task queue. For example, stage 1: split into video, audio and metadata. Stage 2: video encoding and thumbnail for the video part, and audio encoding for the audio part.
  3. Resource manager. It decides which worker runs which task. It holds a task queue (tasks by priority), a worker queue (workers by how busy), a running queue (tasks matched to workers) and a task scheduler. The scheduler takes the top task, picks the best worker, tells it to run, records the pair in the running queue, and removes it when done.
  4. Task workers. They run the tasks. Different workers may run different kinds of task.
  5. Temporary storage. Small, hot data such as metadata sits in memory. Large video and audio chunks sit in blob storage. It is cleared when the video is done.
  6. Encoded video. The output, for example funny_720p.mp4.

A worked example (made up). Stage 1 takes 2 minutes. In stage 2, video encoding takes 10 minutes, thumbnail 1 minute and audio encoding 3 minutes. Run one after another, the total is 2 + 10 + 1 + 3 = 16 minutes. Run stage 2 tasks in parallel, it is 2 + 10 = 12 minutes, as the longest task decides.

Common mistake

Running the whole transcode as one long job on one machine. A failure near the end wastes hours. Split the video into chunks and tasks, so only the failed piece is redone.

Check yourself: 1. Why does the preprocessor save chunks? 2. What does the resource manager decide? 3. Why is stage 2 faster in parallel?