CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 3 of 8 · 2 min read

High-level design

Upload and stream.

The system has three kinds of part. Clients on phones, browsers and TVs. A CDN that holds videos and streams them. API servers that do everything else: recommendations, upload links, metadata updates and sign-up.

Upload flow. Two things happen in parallel. (a) The video file is uploaded. (b) The video's metadata, meaning its name, size, format, owner and URL, is saved.

Flow (a):

  1. The client uploads the file to original storage, a blob store (a service for large binary files).
  2. Transcoding servers fetch it and convert it.
  3. Then two steps run in parallel. The converted files go to transcoded storage and on to the CDN. And a "done" event goes on a completion queue.
  4. Completion handler workers read the queue and update the metadata database and cache.
  5. The API servers tell the client that the video is ready.

Flow (b): while the file uploads, the client sends the metadata to the API servers, which update the metadata cache and the metadata database.

Streaming flow. Streaming means the device receives the video continuously and plays it as it arrives. It does not download the whole file first. A streaming protocol is the agreed way to send and control this data. Known ones are MPEG-DASH, Apple HLS, Microsoft Smooth Streaming and Adobe HDS. Different protocols suit different players, so the choice follows the target devices. The video is served by the edge server nearest to the viewer, which keeps delay small.

upload:  client -> original storage -> transcoding -> transcoded storage -> CDN
                                           |-> completion queue -> handler -> metadata DB + cache
         client -> API servers -> metadata DB + cache   (in parallel)

watch:   client <---- streaming protocol ---- nearest CDN edge

Metadata

Metadata is data about a file rather than its content, such as name, size, format, owner and location.

Example: for a video: title "funny", size 300 MB, owner 42, URL of the 720p file.

Check yourself: 1. Which two processes run in parallel at upload? 2. What does the completion queue carry? 3. Where does the viewer's video come from?