CSE Roadmap

Core tier · about 3 hours · 6 parts

Part 5 of 6 · 3 min read

Passthrough

Passthrough, the IOMMU and SR-IOV

Think of lending a company car to one employee. The employee drives it without asking each time, which is fast, and the others cannot use it. The analogy stops in one place: a car cannot damage another car by itself, and a device that does DMA could write over any memory if nothing limited it.

In plain words: In passthrough, the hypervisor maps the real device into one guest, so the guest's driver reads and writes the real registers without any exit. Two problems must be solved. First, the device does DMA with physical addresses (Level 8), and a guest driver gives it guest physical addresses, so the device could read or write memory outside the guest. The IOMMU is a table, like a page table, for device accesses: it translates the device's address (the guest physical address) to the real host address and blocks the others. Second, one device can have only one owner, unless the device splits itself. SR-IOV (single-root I/O virtualisation) makes one physical function present many virtual functions, each a small copy that a different guest can own.

A worked example (made up to show the idea). A server has one network card of 100 Gbit/s with SR-IOV and 8 guests. With virtio, each guest shares the host's software switch, and the host CPU copies or moves every packet, which limits the rate to a few tens of Gbit/s and uses host cores. With 8 virtual functions, each guest owns one and the card puts packets into the guest's memory by DMA, with the IOMMU checking each address. The host CPU is not on the data path. The costs are that the guest depends on the real hardware (so live migration is harder, since the device state cannot simply move), and that the cards must support SR-IOV.

The key lines. Emulation is the most flexible and the slowest, passthrough is the fastest and the least flexible, and virtio is in the middle and the most common.

IOMMU

An IOMMU is a memory management unit for devices. It translates and checks the addresses that a device uses for DMA, so that a device given to a guest can reach only that guest's memory.

Example: a passed-through network card writes to a guest physical address, and the IOMMU maps it to the right host page and rejects any other.

Interview angle

A common question is "How does a VM do I/O fast?" A good answer names the three models, explains that emulation causes an exit per register access, that virtio batches requests in shared rings and kicks once, and that passthrough with an IOMMU and SR-IOV removes the host from the data path, at the cost of flexibility.

Check yourself: 1. Why does a passed-through device need an IOMMU? 2. What does SR-IOV give? 3. Name one cost of passthrough.

Words you will meet

Word Meaning
Emulated device A virtual copy of a real card that traps on each register access.
Virtio The standard for paravirtual devices.
Virtqueue A shared ring for requests between guest and host.
Kick A notification from guest to host.
IOMMU Address translation and checking for devices.
SR-IOV One device that presents many virtual functions.

Quick notes

  • Three models: emulation (slow, no guest change), virtio (fast, guest driver), passthrough (fastest, one owner).
  • A virtqueue is a shared ring, and a kick is one exit for many requests.
  • With a batch of 32, exits fall 128 times in the example (4 per request to 1 per 32).
  • The test machine has 10 virtio devices: six disks, one network card, balloon, rng and vsock.
  • Passthrough needs an IOMMU for safe DMA, and SR-IOV to share one card.