Part 3 of 8 · 2 min read
Architecture
Edge, mid tier, shield, origin.
Layers.
- Edge servers (PoPs): close to users, in many cities. Each keeps a cache in memory and on disk.
- Regional (mid-tier) caches: fewer, larger caches. An edge that misses asks its regional cache before the origin. This raises the overall hit ratio, because many edges share one regional cache.
- Origin shield: one designated cache layer in front of the origin. All regional caches miss through the shield, so the origin sees at most one request per file, however many regions ask. It protects the origin.
- Origin: the real source: web servers or object storage.
- Control plane: configuration, certificates, purge commands, metrics and routing decisions. Kept apart from the data plane (the servers that serve users) so a control failure does not stop delivery.
Request flow with layers.
- The user's request reaches the nearest edge.
- Edge hit: return it. Miss: ask the regional cache.
- Regional hit: return it to the edge, which stores a copy. Miss: ask the shield.
- Shield miss: fetch from the origin, store and return along the same path.
Request collapsing. If 1,000 users ask for a new file within the same second, the edge sends a single request upstream and holds the others until the answer arrives (also called request coalescing). Without it, a popular new file causes a stampede on the origin.
Within a PoP. Many cache servers sit behind a local load balancer. To avoid every server caching every file, route by the URL with consistent hashing (Level 3): the same URL always goes to the same server of the PoP. That multiplies the effective cache size and raises the hit ratio. Very hot files are replicated to several servers to avoid a hot spot.
Storage on a cache server. Memory for the hottest objects, then SSD for the warm ones. An eviction rule decides what to drop: least recently used (LRU) is a common base (Level 2). Large video files are cut into short segments, so a segment is the unit of caching.
A worked example (made up). A file is requested for the first time in 40 regions. With no shield, the origin receives up to 40 requests. With a shield, the origin receives 1, and the shield answers the other 39 from its copy. A popular launch day thus spares the origin 97.5 per cent of that traffic.
Origin shield
An origin shield is a cache layer placed directly in front of the origin, so that all other caches miss through it and the origin sees few requests.
Example: 40 regional caches all need a new file. The shield fetches it once from the origin and passes it to all 40.
Check yourself: 1. Why add a regional layer? 2. What does request collapsing prevent? 3. Why route by URL inside a PoP?