Part 5 of 6 · 4 min read
Cost of a trap
What a trap costs
Think of two kinds of errand: picking up a pen from your own desk, and sending a request to head office and waiting for a reply. The first takes a second, the second takes a day. The analogy stops in one place: head office also does useful work, while the hypervisor's time is mostly the cost of switching.
In plain words: A trap is a VM exit: the processor stops the guest, saves its state, loads the hypervisor's state, and runs the hypervisor. Later a VM entry goes back. The pair costs at least a thousand cycles, and more if the hypervisor must do work. On x86 the instruction cpuid (which asks for processor information) always traps in a VM, since the hypervisor must be able to choose what the guest sees. The instruction rdtsc (read the cycle counter) can be left untrapped. Timing both shows the gap.
A worked example.
#include <cpuid.h>
#include <stdio.h>
#include <time.h>
static double now(void) {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return t.tv_sec + t.tv_nsec * 1e-9;
}
int main(void) {
unsigned a, b, c, d;
enum { N = 200000 };
double t0 = now();
for (int i = 0; i < N; i++) __cpuid(0, a, b, c, d); /* always trapped by a hypervisor */
double t1 = now();
for (int i = 0; i < N; i++) { unsigned lo, hi; __asm__ volatile("rdtsc" : "=a"(lo), "=d"(hi)); }
double t2 = now();
printf("cpuid: %.0f ns each rdtsc: %.0f ns each\n", (t1 - t0) / N * 1e9, (t2 - t1) / N * 1e9);
return 0;
}
Three runs on the test machine:
cpuid: 5244 ns each rdtsc: 14 ns each
cpuid: 6443 ns each rdtsc: 14 ns each
cpuid: 5486 ns each rdtsc: 14 ns each
One cpuid cost about 5,000 to 6,000 ns, and one rdtsc 14 ns, so the trapped instruction was about 400 times slower (5,500 / 14 = 393). The test machine is itself a guest, so an exit may pass through more than one layer. That makes it slower than on a simple server, where a cpuid exit usually costs a small number of microseconds. The lesson holds on any machine: code that runs without exits runs at native speed, and each exit costs thousands of cycles.
The key lines. A guest that does many I/O operations or many page-table changes (page 2) causes many exits and runs slower. Good virtualisation designs reduce exits: this is why virtio (page 3) uses shared queues and batches.
VM exit
A VM exit is the transfer of control from a running guest to the hypervisor, caused by an event that the hypervisor asked to intercept, such as a privileged instruction or an access to an emulated device. The matching VM entry resumes the guest.
Example: every cpuid in a guest causes one VM exit and one VM entry.
Common mistake
Reading "a VM is as fast as a real machine" as a rule for every workload. Pure computation is near native speed, and work that traps often (I/O, page-table changes, timers) is not.
Check yourself:
1. What is a VM exit?
2. Why does cpuid always trap?
3. How many times slower was cpuid than rdtsc in the test?
Words you will meet
| Word | Meaning |
|---|---|
| Virtual machine | A software copy of a whole computer. |
| Hypervisor | The program that runs virtual machines. |
| Host / guest | The real system / the system inside a VM. |
| Trap and emulate | Catch a privileged instruction and perform a safe version. |
| Hypercall | A deliberate call from a guest to the hypervisor. |
| VM exit / entry | Switch from guest to hypervisor / back. |
Quick notes
- A VM is a software computer, and the hypervisor shares the real hardware among them.
- Type 1 runs on the hardware, type 2 runs on a host operating system.
- Guests run deprivileged, and privileged instructions trap and are emulated.
- Hardware assistance (VT-x, AMD-V) makes this fast and needs no change to the guest.
- A VM exit costs thousands of cycles: on the test machine
cpuidtook about 5,500 ns against 14 ns forrdtsc. lscpu, thehypervisorflag andsystemd-detect-virtshow whether code runs in a VM.