CSE Roadmap

Core tier · about 3 hours · 6 parts

Part 5 of 6 · 4 min read

Cost of a trap

What a trap costs

Think of two kinds of errand: picking up a pen from your own desk, and sending a request to head office and waiting for a reply. The first takes a second, the second takes a day. The analogy stops in one place: head office also does useful work, while the hypervisor's time is mostly the cost of switching.

In plain words: A trap is a VM exit: the processor stops the guest, saves its state, loads the hypervisor's state, and runs the hypervisor. Later a VM entry goes back. The pair costs at least a thousand cycles, and more if the hypervisor must do work. On x86 the instruction cpuid (which asks for processor information) always traps in a VM, since the hypervisor must be able to choose what the guest sees. The instruction rdtsc (read the cycle counter) can be left untrapped. Timing both shows the gap.

A worked example.

#include <cpuid.h>
#include <stdio.h>
#include <time.h>

static double now(void) {
    struct timespec t;
    clock_gettime(CLOCK_MONOTONIC, &t);
    return t.tv_sec + t.tv_nsec * 1e-9;
}

int main(void) {
    unsigned a, b, c, d;
    enum { N = 200000 };
    double t0 = now();
    for (int i = 0; i < N; i++) __cpuid(0, a, b, c, d);   /* always trapped by a hypervisor */
    double t1 = now();
    for (int i = 0; i < N; i++) { unsigned lo, hi; __asm__ volatile("rdtsc" : "=a"(lo), "=d"(hi)); }
    double t2 = now();
    printf("cpuid: %.0f ns each   rdtsc: %.0f ns each\n", (t1 - t0) / N * 1e9, (t2 - t1) / N * 1e9);
    return 0;
}

Three runs on the test machine:

cpuid: 5244 ns each   rdtsc: 14 ns each
cpuid: 6443 ns each   rdtsc: 14 ns each
cpuid: 5486 ns each   rdtsc: 14 ns each

One cpuid cost about 5,000 to 6,000 ns, and one rdtsc 14 ns, so the trapped instruction was about 400 times slower (5,500 / 14 = 393). The test machine is itself a guest, so an exit may pass through more than one layer. That makes it slower than on a simple server, where a cpuid exit usually costs a small number of microseconds. The lesson holds on any machine: code that runs without exits runs at native speed, and each exit costs thousands of cycles.

The key lines. A guest that does many I/O operations or many page-table changes (page 2) causes many exits and runs slower. Good virtualisation designs reduce exits: this is why virtio (page 3) uses shared queues and batches.

VM exit

A VM exit is the transfer of control from a running guest to the hypervisor, caused by an event that the hypervisor asked to intercept, such as a privileged instruction or an access to an emulated device. The matching VM entry resumes the guest.

Example: every cpuid in a guest causes one VM exit and one VM entry.

Common mistake

Reading "a VM is as fast as a real machine" as a rule for every workload. Pure computation is near native speed, and work that traps often (I/O, page-table changes, timers) is not.

Check yourself: 1. What is a VM exit? 2. Why does cpuid always trap? 3. How many times slower was cpuid than rdtsc in the test?

Words you will meet

Word Meaning
Virtual machine A software copy of a whole computer.
Hypervisor The program that runs virtual machines.
Host / guest The real system / the system inside a VM.
Trap and emulate Catch a privileged instruction and perform a safe version.
Hypercall A deliberate call from a guest to the hypervisor.
VM exit / entry Switch from guest to hypervisor / back.

Quick notes

  • A VM is a software computer, and the hypervisor shares the real hardware among them.
  • Type 1 runs on the hardware, type 2 runs on a host operating system.
  • Guests run deprivileged, and privileged instructions trap and are emulated.
  • Hardware assistance (VT-x, AMD-V) makes this fast and needs no change to the guest.
  • A VM exit costs thousands of cycles: on the test machine cpuid took about 5,500 ns against 14 ns for rdtsc.
  • lscpu, the hypervisor flag and systemd-detect-virt show whether code runs in a VM.