Interview questions
Operating Systems Interview Questions
100 operating systems interview questions, answered using a four-part structure: the core concept in plain terms, the kernel mechanics underneath it, the trade-off it accepts, and its real production or debugging impact. Covers fundamentals, processes and threads, scheduling, concurrency and deadlocks, memory management, file systems and storage, and Linux internals and troubleshooting.
Jump to a level
- Level 0: How an operating system works — 8 questions
- Level 1: Processes — 10 questions
- Level 2: Threads — 5 questions
- Level 3: CPU scheduling — 12 questions
- Level 4: Concurrency and synchronisation — 12 questions
- Level 5: Deadlocks — 5 questions
- Level 6: Memory management — 11 questions
- Level 7: Memory in practice — 8 questions
- Level 8: Storage and I/O devices — 7 questions
- Level 9: File systems — 8 questions
- Level 10: Security and protection — 1 question
- Level 11: Virtualisation and containers — 2 questions
- Level 12: Linux in practice and interview prep — 11 questions
Level 0: How an operating system works
How an operating system works — interview questions
Answer in your head first, then open it.
What is an Operating System, and what are its core responsibilities?
Concept: An operating system is the software layer that sits between raw hardware and every running program, managing CPU time, memory, storage, and devices so applications don't have to talk to hardware directly. Kernel Mechanics: It does this through the kernel, which runs in a privileged CPU mode and owns the hardware-facing data structures, process tables, page tables, device queues, exposing controlled access to user programs only through system calls. Trade-off: Centralizing all hardware access through one kernel adds a layer of indirection and overhead to every hardware interaction, but it's what lets many untrusted programs share one machine safely. Production Impact: Commands like top, vmstat, and strace are really windows into the kernel's own bookkeeping, process states, memory stats, syscall traces, which is why understanding the kernel's responsibilities is the foundation for reading any of them.
Explain the difference between Kernel Mode and User Mode. How does the CPU switch between them?
Concept: User mode restricts a running program from executing privileged instructions or touching hardware directly; kernel mode lifts that restriction for the OS itself. Kernel Mechanics: The CPU enforces this with a hardware privilege bit; a user program triggers a mode switch via a system call, interrupt, or trap, which causes the CPU to jump to a fixed kernel entry point, save the user context, and raise its privilege level before running kernel code. Trade-off: The switch itself costs real cycles, flushing pipeline state and touching shared kernel data structures, so frequent mode switching, many small syscalls, measurably slows a program down. Production Impact: High %sy time in top usually means a process is spending unusually large shares of its time in kernel mode, often from excessive syscalls or page faults, a direct signal to profile with strace.
What is a System Call, and what happens step-by-step during fork() or read()?
Concept: A system call is the controlled entry point a user program uses to ask the kernel to do something on its behalf, since the program itself can't touch hardware or other processes' memory directly. Kernel Mechanics: Calling fork() traps into the kernel, which allocates a new PCB, duplicates the calling process's address space via copy-on-write page tables, copies file descriptors, and schedules the new process; read() traps in, the kernel checks the file descriptor table, locates the backing inode or socket buffer, copies data into the user buffer, and returns the byte count. Trade-off: Every syscall costs a full user-to-kernel-to-user round trip, so batching I/O or using mmap to avoid repeated read() calls is a common performance technique. Production Impact: strace -c on a slow program quickly shows which syscall is dominating, a common first diagnostic step for an unexplained slowdown.
Compare Monolithic Kernels (Linux), Microkernels (Minix/QNX), and Hybrid Kernels (Windows/macOS).
Concept: These describe how much OS functionality actually runs inside the privileged kernel versus in separate, isolated user-space servers. Kernel Mechanics: A monolithic kernel like Linux runs drivers, file systems, and networking all in one address space with direct function calls between them; a microkernel like Minix or QNX keeps only scheduling, IPC, and basic memory management in the kernel and runs drivers and file systems as user-space servers communicating over message passing; a hybrid kernel runs most services in kernel space for speed but keeps some subsystems modular. Trade-off: Monolithic kernels are fast, since everything talks via direct calls, but a bug in any driver can crash the whole system; microkernels isolate failures but pay a real performance cost on every cross-component IPC call. Production Impact: A Linux kernel panic from a bad driver is the direct cost of the monolithic trade-off, part of why Linux increasingly uses loadable, more sandboxed kernel modules.
What happens step-by-step from pressing the power button until the OS login screen appears?
Concept: Booting is the sequence that takes a powered-off machine from nothing but firmware to a running OS with a login prompt. Kernel Mechanics: Power-on runs the BIOS or UEFI firmware, which initializes hardware and locates a bootloader (GRUB, systemd-boot); the bootloader loads the kernel image and an initial filesystem into memory and hands off control; the kernel initializes its own drivers and mounts the real root filesystem, then starts the first user-space process, init or systemd, which brings up services in dependency order. Trade-off: UEFI with Secure Boot adds cryptographic verification at each stage, closing off a class of boot-time malware but adding real complexity when something needs to be swapped, like a new kernel module. Production Impact: A server stuck before reaching a login prompt is almost always failing at one of these exact stages, which is why checking dmesg or journalctl -b starts with figuring out which stage it got stuck in.
Explain the difference between an Interrupt, a Trap (Exception), and a Signal.
Concept: All three redirect execution away from a program's normal instruction flow, but they come from different sources. Kernel Mechanics: A hardware interrupt comes asynchronously from a device, a timer, a keyboard, a NIC, and the CPU jumps to a kernel interrupt handler regardless of what was running; a trap is synchronous, triggered by the currently executing instruction itself, a syscall, a divide-by-zero, a page fault; a signal is a software notification the kernel delivers to a specific process, which the process can catch, ignore, or let trigger its default action. Trade-off: Interrupts need to be handled fast and can't block, so heavy work gets deferred to a lower-priority context, a Linux softirq or tasklet, rather than done inline in the handler. Production Impact: A segfault seen in dmesg is the kernel's trap handler detecting an invalid memory access and converting it into a SIGSEGV signal delivered to the offending process.
What is the difference between Firmware, Device Drivers, and the Kernel?
Concept: These are three distinct layers between a physical device and the application that wants to use it. Kernel Mechanics: Firmware is small, permanent code embedded directly on a hardware device that initializes it at the lowest level; a device driver is kernel-loadable code that knows the specific device's register-level protocol and exposes it through the kernel's generic device interfaces; the kernel itself provides those generic interfaces, block devices, character devices, network interfaces, that let the rest of the OS talk to any device the same way. Trade-off: Pushing device-specific logic into drivers rather than the kernel core keeps the kernel hardware-agnostic, but a single bad or outdated driver can still crash or destabilize the whole monolithic kernel. Production Impact: A kernel panic right after plugging in new hardware is almost always a driver bug, which is why dmesg right after the event is the first place to look.
What are POSIX standards, and why do they matter across Unix-like systems?
Concept: POSIX is a standardized set of APIs, including system calls like fork(), read(), and pthread functions, that Unix-like systems agree to implement the same way. Kernel Mechanics: A POSIX-compliant kernel exposes these calls with the same signatures and semantics regardless of the underlying implementation, so the same C source calling fork() and read() behaves consistently whether compiled for Linux, BSD, or macOS. Trade-off: Standardizing around POSIX guarantees portability across systems, but it can lock an OS into semantics that don't fit a newer hardware reality, part of why Linux has grown non-POSIX extensions like epoll and io_uring alongside the standard calls. Production Impact: Portable server software, databases, web servers, is written against POSIX precisely so it can run unmodified across different Unix-like deployment targets.
Level 1: Processes
Processes — interview questions
Answer in your head first, then open it.
What is a Context Switch, what state is saved in the PCB, and why is it expensive?
Concept: A context switch is the kernel saving the full execution state of the currently running process or thread and loading the saved state of the next one, so several tasks can share one CPU core. Kernel Mechanics: Triggered by a timer interrupt or a blocking syscall, the CPU traps into kernel mode, saves registers, the program counter, and stack pointer into the outgoing thread's PCB, loads the incoming thread's saved state, and, if switching between processes rather than threads of the same process, reloads the page-table base register to point at the new address space. Trade-off: The direct register save/restore takes only microseconds, but switching address spaces flushes the TLB and evicts whatever was warm in the CPU cache, a much larger indirect cost than the switch itself. Production Impact: A high cs count in vmstat alongside high %sy time usually points to too many threads competing for too few cores, part of why high-concurrency servers often prefer an event loop over one OS thread per connection.
What is the difference between a Program, a Process, and a Thread?
Concept: A program is static code sitting on disk; a process is that code actually running, with its own memory and resources; a thread is a unit of execution inside a process that shares the process's memory with any sibling threads. Kernel Mechanics: Loading a program creates a process: the kernel allocates an address space, a PCB, and an initial thread; spawning an additional thread inside that process reuses the same address space but gives the new thread its own stack, registers, and TCB. Trade-off: Threads are cheaper to create and communicate between, since they share memory directly, but that shared memory means one thread's bug, a stray write, can corrupt another's state, unlike the strong isolation between processes. Production Impact: A crash that takes down an entire multi-threaded server, instead of just one request, is usually exactly this: shared memory letting one thread's fault reach every other thread in the process.
Draw and explain the Process Lifecycle (New, Ready, Running, Waiting/Blocked, Terminated).
Concept: Every process moves through a small set of states as the kernel decides who gets the CPU and who's waiting on something. Kernel Mechanics: A process starts New while the kernel builds its PCB and address space, moves to Ready once it's eligible to run, is picked by the scheduler into Running on a CPU core, moves to Waiting/Blocked if it issues a blocking syscall like a disk read, returns to Ready once that I/O completes, and finally reaches Terminated when it exits, at which point the kernel reclaims its resources. Trade-off: Keeping a process in Waiting rather than busy-polling frees the CPU for other work, but the transition back to Ready and eventually Running still costs a scheduling decision and a context switch once the wait condition is met. Production Impact: A process stuck in Waiting/Blocked for an unusually long time, visible as D state in ps, is a strong signal of a stuck I/O operation, like a hung NFS mount, rather than a CPU-bound problem.
What is stored inside a Process Control Block (PCB) vs. a Thread Control Block (TCB)?
Concept: The PCB holds everything the kernel needs to manage a process as a whole; the TCB holds the narrower per-thread state needed to resume one specific thread of execution inside it. Kernel Mechanics: A PCB contains the process ID, process state, the page-table pointer for its address space, open file descriptor table, scheduling priority, and accounting info; a TCB contains that specific thread's saved registers, program counter, stack pointer, and its own stack region, while pointing back to the shared PCB for everything process-wide. Trade-off: Splitting state this way lets threads resume independently and cheaply, since a thread switch within the same process doesn't touch the page table, but it means thread-local versus process-wide state has to be tracked correctly everywhere in the kernel. Production Impact: Tools like ps -eLf list threads (TCBs) under their parent process (PCB), which is exactly this two-level structure surfacing in a shell command.
What is a Zombie Process vs. an Orphan Process? How do you create and clean up each in Linux?
Concept: A zombie is a process that has finished executing but still has an entry in the process table because its parent hasn't collected its exit status; an orphan is a process whose parent has exited while the child is still running. Kernel Mechanics: When a child exits, the kernel keeps its PCB around as a zombie (state Z) until the parent calls wait() or waitpid() to read the exit code, which lets the kernel finally free the entry; an orphan's parent dying causes the kernel to reparent the orphan to init (PID 1) or a subreaper, which is responsible for eventually reaping it when it exits. Trade-off: Keeping zombies around until explicitly reaped lets a parent reliably retrieve a child's exit status even some time after it finished, at the cost of a PCB entry sitting unreclaimed until that happens. Production Impact: A long-running parent that never calls wait() on its children accumulates zombies over time, which, while not consuming much memory individually, can exhaust the system's PID table if left unbounded.
How do fork(), vfork(), exec(), and wait() work together in Unix/Linux?
Concept: These four calls together implement Unix's split model of process creation: duplicate, then replace, then collect. Kernel Mechanics: fork() creates a near-identical copy of the calling process with a new PID; vfork() does the same but, as an optimization, suspends the parent and lets the child share the parent's address space until it calls exec() or exits, to avoid an unnecessary copy; exec() replaces the calling process's own code and address space with a new program entirely, keeping the same PID; wait() blocks the parent until a specific child terminates, returning its exit status. Trade-off: The classic fork()-then-exec() pattern (spawn a shell command) pays the cost of duplicating an address space it's about to immediately discard, which is exactly what vfork() and the copy-on-write optimization under fork() itself are designed to avoid. Production Impact: A shell running a command is literally fork() followed by exec() in the child and wait() in the parent, the textbook example of this four-call pattern in everyday use.
What is Copy-on-Write (CoW) during fork(), and how does it optimize memory and speed?
Concept: Copy-on-write lets fork() appear to duplicate a process's entire address space instantly, without actually copying any of the underlying memory pages up front. Kernel Mechanics: The kernel duplicates only the parent's page table, marking every page read-only and shared between parent and child; the first time either process tries to write to a shared page, a page fault fires, the kernel allocates a real private copy of just that page for the writer, and execution resumes. Trade-off: This defers the cost of copying from fork() time to the first actual write, which is a huge win for the extremely common fork()-then-exec() pattern that never writes to most of the copied memory at all, but it adds a small page-fault cost the first time a page genuinely diverges. Production Impact: Without CoW, forking a process with gigabytes of resident memory would momentarily double that memory usage and take real time to copy; CoW is why fork() stays cheap even for large processes.
Compare all major Inter-Process Communication (IPC) mechanisms: Pipes, Named Pipes (FIFOs), Message Queues, Shared Memory, and Sockets.
Concept: These are the kernel-provided channels unrelated processes use to exchange data, since they don't share memory by default the way threads do. Kernel Mechanics: A pipe is an anonymous, kernel-buffered byte stream between related processes, one-directional and gone once both ends close; a named pipe (FIFO) is the same mechanism given a filesystem path so unrelated processes can open it; a message queue lets processes exchange discrete, typed messages that persist in the kernel until read; shared memory maps the same physical pages into multiple processes' address spaces for direct access with no kernel copying; a socket provides a bidirectional, addressable channel that works both locally and over a real network. Trade-off: Shared memory is by far the fastest, since there's no data copying through the kernel at all, but it provides zero synchronization on its own, so it has to be paired with semaphores or mutexes; sockets are the most flexible, working across machines, at the cost of extra kernel-mediated copying per message. Production Impact: High-throughput systems like databases often use shared memory for their buffer pools specifically to avoid the IPC copying overhead that pipes or sockets would add.
Why is Shared Memory the fastest IPC mechanism, and what extra synchronization does it require?
Concept: Shared memory is the fastest IPC mechanism because the kernel is involved only in setting it up, not in moving the actual data. Kernel Mechanics: The kernel maps the same set of physical memory pages into the virtual address spaces of multiple processes, so once set up, one process writing to that memory is immediately visible to another reading it, with zero data copying or kernel mediation per access. Trade-off: That speed comes at the cost of all safety: the kernel provides no coordination over who's reading or writing when, so the processes themselves must implement their own synchronization, typically with a semaphore or named mutex also placed in the shared region, to avoid racing on the data. Production Impact: A shared-memory segment with no synchronization layered on top is a guaranteed source of race-condition bugs under concurrent access, which is why nearly every real shared-memory IPC use pairs it with POSIX semaphores.
What is a Daemon Process, and how is it detached from a terminal?
Concept: A daemon is a process designed to run continuously in the background, with no controlling terminal and no interactive session attached to it. Kernel Mechanics: The standard pattern to create one is a double fork: the process forks, the parent exits, the child calls setsid() to become a session leader with no controlling terminal, forks again so the new child is no longer a session leader either, and then closes or redirects its standard file descriptors so it's fully detached from whatever terminal launched it. Trade-off: Fully detaching from the terminal means the daemon survives the terminal closing or the launching shell exiting, which is exactly the point, but it also means its own crash output has to be captured through logging rather than just appearing on a screen. Production Impact: This is why production daemons write to syslog or a log file rather than stdout, since by the time something goes wrong, there's no terminal left attached to see it.
Level 2: Threads
Threads — interview questions
Answer in your head first, then open it.
Which memory segments are shared among threads of the same process, and which are private?
Concept: Threads of the same process share most of the process's memory but each still needs some private state to execute independently. Kernel Mechanics: Code (the program's instructions), Data (global/static variables), and Heap (dynamically allocated memory) are shared across all threads in the process's single address space; each thread gets its own private Stack for local variables and call frames, and its own set of CPU registers saved in its TCB. Trade-off: Sharing the heap and globals is what makes threads useful for cooperating work without explicit IPC, but it's exactly that sharing that creates race conditions whenever two threads touch the same heap data without synchronization. Production Impact: A heap corruption bug that only shows up under concurrent load, and not in single-threaded testing, is almost always two threads racing on shared heap or global state.
Compare User-Level Threads (ULT) and Kernel-Level Threads (KLT) (1:1, N:1, M:N threading models).
Concept: The question is whether the kernel actually knows about and schedules individual threads, or whether a user-space library is managing them itself behind the kernel's back. Kernel Mechanics: Under the N:1 model, a user-space library multiplexes many user threads onto a single kernel thread, so the kernel only ever sees one schedulable entity; under 1:1, every user thread maps to its own kernel thread, giving the kernel full visibility and true parallelism; M:N maps many user threads onto a smaller pool of kernel threads, trying to get benefits of both. Trade-off: N:1 threads are extremely cheap to create and switch between, since no kernel involvement is needed, but one thread blocking on a syscall blocks the entire process since the kernel sees only one thread; 1:1 avoids that but pays a real kernel-level context-switch cost per thread. Production Impact: Most modern production systems, including Linux's pthreads, use 1:1 threading, while high-concurrency user-space schedulers like Go's goroutines implement their own M:N-style model on top of it for cheaper concurrency.
What happens when a multi-threaded process calls fork()?
Concept: fork() on a multi-threaded process duplicates the entire address space, but the new child process ends up with only one thread, not a copy of every thread. Kernel Mechanics: The kernel copies the calling thread's state into the single thread the child gets; every other thread that existed in the parent simply doesn't exist in the child at all, but any locks those other threads were holding at the moment of the fork are copied into the child in whatever state they were in, held or not, with no guarantee the thread that would normally release them still exists. Trade-off: This keeps fork() itself simple and fast, since it doesn't have to recreate every thread's state, but it's a well-known trap: a lock held by a thread that doesn't exist in the child can never be released there, deadlocking any child code that needs it. Production Impact: This is exactly why the POSIX guidance is to avoid calling complex, lock-using code between fork() and exec() in a multi-threaded program, and why libraries register pthread_atfork handlers to reset locks safely across a fork.
What are Green Threads / Coroutines / Fibers, and how do they differ from OS threads?
Concept: Green threads, coroutines, and fibers are all forms of concurrency implemented entirely in user space, without the kernel knowing individual units exist. Kernel Mechanics: A user-space runtime maintains its own scheduler and its own set of lightweight stacks, switching between them with plain function calls and stack-pointer swaps rather than a kernel-mediated context switch, so from the kernel's point of view, it's all still just one OS thread doing all the work. Trade-off: Because there's no syscall or kernel involvement in switching between them, these are dramatically cheaper to create and switch between than real OS threads, but they can't achieve true CPU parallelism on their own, since the kernel still schedules only the single underlying OS thread they run on, unless the runtime explicitly maps them across a pool of OS threads. Production Impact: Go's goroutines and Python's asyncio coroutines both use this model to support tens of thousands of concurrent logical tasks on a handful of real OS threads, which is how they scale concurrency far beyond what one-OS-thread-per-task could support.
Why is creating a thread faster than creating a process, and when would you still choose multi-processing over multi-threading?
Concept: Thread creation is faster because it reuses almost everything the process already has, while process creation has to build a whole new address space from scratch. Kernel Mechanics: Creating a thread allocates just a new stack, a TCB, and registers, pointing at the same existing page table and file descriptor table; creating a process (even with copy-on-write) still requires a new PCB, a new page table, and duplicating or re-establishing the file descriptor table, all real kernel bookkeeping a new thread skips entirely. Trade-off: That shared state is exactly why multi-threading is cheaper but also why one thread's crash or memory corruption can take down the whole process; multi-processing costs more to set up but gives each worker genuine memory isolation, so one crashing worker doesn't corrupt or kill the others. Production Impact: A web server handling many untrusted or unstable third-party plugins often isolates each into its own process specifically for that crash containment, accepting the higher per-worker cost in exchange for not losing the whole server to one bad plugin.
Level 3: CPU scheduling
CPU scheduling — interview questions
Answer in your head first, then open it.
Differentiate between Preemptive and Non-Preemptive scheduling.
Concept: The question is whether the scheduler can forcibly take the CPU away from a running process, or has to wait for the process to voluntarily give it up. Kernel Mechanics: In non-preemptive scheduling, the running process keeps the CPU until it blocks on I/O or exits on its own, so the scheduler only ever runs at those natural transition points; in preemptive scheduling, a timer interrupt fires periodically, trapping into the kernel, which can forcibly switch to a different, higher-priority or simply next-in-line process even if the current one wants to keep running. Trade-off: Preemption guarantees responsiveness, no single process can hog the CPU indefinitely, but every forced switch costs a context switch that non-preemptive scheduling avoids entirely. Production Impact: Every general-purpose OS, Linux, Windows, macOS, uses preemptive scheduling specifically because a non-preemptive one would let a single runaway process freeze the whole interactive system.
Compare Long-Term, Medium-Term, and Short-Term schedulers.
Concept: These operate at three different timescales and decide three different things: which jobs get admitted, which get swapped, and who runs next on the CPU right now. Kernel Mechanics: The long-term scheduler decides which jobs are admitted into the system at all, controlling the degree of multiprogramming; the medium-term scheduler can swap a process out of main memory entirely to reduce memory pressure, and swap it back in later; the short-term scheduler, the one invoked on every context switch, picks which ready process actually runs on the CPU next, typically many times a second. Trade-off: Running the short-term scheduler frequently keeps the CPU responsive but means its own decision logic has to be extremely fast; the long- and medium-term schedulers run far less often and can afford more expensive, holistic decisions about overall system load. Production Impact: Modern general-purpose OSes mostly collapse the long-term scheduler into simple load-based admission and rely heavily on the short-term scheduler, since full process swapping to disk is rare on systems with ample RAM.
Explain FCFS, SJF (Shortest Job First), SRTF (Shortest Remaining Time First), and Priority Scheduling.
Concept: These are four different rules for picking which ready process runs next, each optimizing for something different. Kernel Mechanics: FCFS runs processes strictly in arrival order with no reordering; SJF always picks the ready process with the shortest total burst time, requiring the scheduler to know or estimate that length in advance; SRTF is the preemptive version of SJF, re-evaluating on every new arrival and switching to an even shorter job if one appears mid-execution; Priority scheduling always runs the highest-priority ready process, which can be either preemptive or not. Trade-off: SJF/SRTF minimize average waiting time provably, but they need burst-time knowledge the scheduler usually doesn't actually have, and Priority scheduling can starve low-priority processes indefinitely unless paired with aging. Production Impact: Because burst times aren't known in advance, real-world schedulers approximate SRTF's behavior using historical run-time estimates and dynamic priority adjustment rather than implementing it literally.
What is the Convoy Effect in FCFS scheduling?
Concept: The convoy effect is when one long-running process at the front of an FCFS queue forces every shorter process behind it to wait far longer than its own burst time would suggest. Kernel Mechanics: Because FCFS never reorders the ready queue, a CPU-bound process ahead of several short I/O-bound processes keeps the CPU for its full burst, during which every process behind it, even ones needing only a few milliseconds, sits idle in Ready, and the devices those short processes would have used sit idle too. Trade-off: FCFS's complete lack of reordering makes it trivially simple and perfectly fair in arrival order, but that simplicity is exactly what creates the convoy effect whenever burst lengths vary widely. Production Impact: This is the textbook argument against using plain FCFS for any interactive or mixed workload, and is the direct motivation for Round Robin and priority-based scheduling in real systems.
How does Round Robin (RR) scheduling work, and how do you choose the optimal time quantum?
Concept: Round Robin gives every ready process a small, fixed slice of CPU time in turn, cycling through the ready queue so no process can monopolize the CPU for long. Kernel Mechanics: The scheduler runs the process at the head of the ready queue for one time quantum; if it's not finished when the quantum expires, a timer interrupt preempts it and moves it to the back of the queue, and the next process is dispatched. Trade-off: A very short quantum makes the system feel responsive but drives up the total number of context switches, each with its own overhead; a very long quantum reduces that overhead but starts behaving like FCFS and reintroduces the convoy effect. Production Impact: Choosing a quantum is a direct latency-versus-throughput tuning knob, which is why real systems pick it empirically, typically in the 10-100ms range, rather than from a formula.
What is Starvation, and how does Aging solve it?
Concept: Starvation is when a ready process never actually gets to run because the scheduler keeps choosing other processes ahead of it, usually under a strict priority scheme. Kernel Mechanics: Under pure priority scheduling, a low-priority process can sit in Ready indefinitely if higher-priority processes keep arriving; aging fixes this by having the scheduler gradually increase a waiting process's priority the longer it waits, so eventually even a low-priority process's effective priority rises high enough to be scheduled. Trade-off: Aging guarantees eventual progress for every process, but it adds extra scheduler bookkeeping, tracking and updating wait times, and slightly undermines strict priority ordering by letting low-priority work occasionally jump ahead. Production Impact: Linux's scheduler incorporates aging-like mechanisms specifically so background, low-priority tasks aren't permanently locked out by a steady stream of higher-priority interactive work.
Explain Multi-Level Feedback Queue (MLFQ) scheduling and why modern OSes use variations of it.
Concept: MLFQ uses several priority queues with different time quanta and automatically moves a process between them based on its observed behavior, rather than relying on a fixed priority assigned up front. Kernel Mechanics: A new or recently-scheduled process starts in the highest-priority queue with a short quantum; if it uses its full quantum without blocking (behaving like a CPU-bound job), it's demoted to a lower-priority queue with a longer quantum; a process that blocks for I/O quickly is kept at or promoted toward higher priority, since it's likely interactive. Trade-off: This gives interactive, I/O-bound processes good responsiveness without needing to know anything about them in advance, which plain priority scheduling can't do, but it adds real complexity and tunable parameters, queue count, quanta, demotion rules, that have to be chosen carefully. Production Impact: Linux's CFS and its EEVDF successor don't use MLFQ literally but inherit its core goal, favoring processes that behave interactively, through a different mechanism built around tracked CPU usage.
How does the Linux Completely Fair Scheduler (CFS) / EEVDF work under the hood (Red-Black trees, vruntime)?
Concept: CFS's goal is to give every runnable task a fair share of CPU time proportional to its weight, approximating an idealized processor that could run everything simultaneously. Kernel Mechanics: CFS tracks each task's vruntime, accumulated virtual runtime weighted by priority, and always picks the runnable task with the smallest vruntime to run next, storing all runnable tasks in a red-black tree keyed by vruntime so that minimum is always at the leftmost node, an O(log n) operation; EEVDF, its successor, instead tracks each task's eligibility and virtual deadline to decide who runs next, aiming for more precise fairness especially under mixed latency-sensitive and throughput-oriented workloads. Trade-off: The red-black tree gives CFS good scalability to many runnable tasks, but tracking and comparing vruntime precisely adds real per-schedule overhead compared to a simpler, fixed-priority queue. Production Impact: nice values in Linux directly set a task's scheduling weight, which is what makes a process with a higher nice value accumulate vruntime faster and get picked less often.
Define CPU Utilization, Throughput, Turnaround Time, Waiting Time, and Response Time.
Concept: These are the standard metrics used to judge how well a scheduling policy is actually performing. Kernel Mechanics: CPU Utilization is the percentage of time the CPU is doing useful work rather than idle; Throughput is the number of processes completed per unit time; Turnaround Time is the total time from a process's submission to its completion; Waiting Time is the time a process spends in the Ready queue, not running and not blocked; Response Time is the time from submission until the process's first response, its first time actually getting the CPU. Trade-off: Optimizing for one can hurt another, maximizing throughput often means running short jobs first, which can increase turnaround time for the few long jobs that keep getting deprioritized. Production Impact: Interactive systems prioritize low Response Time, since users notice lag immediately, while batch-processing systems care more about Throughput and overall Turnaround Time across the whole job queue.
What is Processor Affinity (Soft vs. Hard affinity), and how does it affect CPU cache warmth?
Concept: Processor affinity is the scheduler's tendency, or requirement, to keep running a given process on the same CPU core it ran on before. Kernel Mechanics: Soft affinity is just a scheduler preference, it tries to reschedule a process on the same core it last ran on but will move it if needed for load balancing; hard affinity is an explicit, binding constraint, set via a syscall like sched_setaffinity, that pins a process to a specific set of cores no matter what. Trade-off: Keeping a process on the same core lets it keep reusing data still warm in that core's L1/L2 cache, avoiding the cost of refilling the cache from scratch elsewhere, but strict pinning can leave that core overloaded while other cores sit idle, hurting overall load balance. Production Impact: Low-latency applications, trading systems, high-throughput databases, often pin critical threads to dedicated cores specifically to guarantee that cache-warmth benefit and avoid unpredictable migration-induced latency spikes.
What is Priority Inversion (the Mars Pathfinder bug), and how does Priority Inheritance fix it?
Concept: Priority inversion happens when a high-priority task is blocked waiting on a lock held by a low-priority task, which itself gets preempted by some unrelated medium-priority task, effectively letting that medium-priority task indirectly block the high-priority one. Kernel Mechanics: The high-priority task can't proceed until the low-priority lock holder releases the lock, but the low-priority holder isn't being scheduled because a medium-priority task keeps preempting it; priority inheritance fixes this by temporarily boosting the lock holder's priority to match the highest-priority task waiting on that same lock, so it can't be preempted by anything below that level until it releases the lock. Trade-off: Priority inheritance solves the inversion but adds real bookkeeping cost to every lock acquisition, since the kernel or lock implementation has to track and adjust priorities dynamically. Production Impact: This is literally the bug that caused the Mars Pathfinder rover's software resets in 1997, and it's the reason real-time operating systems implement priority-inheriting mutexes as standard practice.
How does NUMA (Non-Uniform Memory Access) affect CPU scheduling and memory allocation?
Concept: On a NUMA system, each CPU socket has its own locally attached memory that it can access faster than memory attached to a different socket. Kernel Mechanics: The scheduler tries to keep a process running on CPUs within the same NUMA node as the memory it's already allocated, and the memory allocator tries to allocate new pages from the node local to the requesting CPU, since a cross-node memory access has to traverse an inter-socket interconnect and is measurably slower. Trade-off: Keeping processes and their memory co-located on one NUMA node maximizes memory-access speed, but it can leave that node's CPUs over-subscribed while another node's CPUs sit idle, forcing a trade-off between NUMA locality and load balancing. Production Impact: Databases and other memory-intensive workloads on multi-socket servers are often explicitly NUMA-pinned, and an unexpectedly high memory-latency profile on such a server is a common sign that a process got scheduled onto the wrong node relative to its allocated memory.
Level 4: Concurrency and synchronisation
Concurrency and synchronisation — interview questions
Answer in your head first, then open it.
What is a Race Condition, and what defines a Critical Section?
Concept: A race condition is a bug where the correctness of a program's result depends on the unpredictable timing of concurrently executing threads or processes. Kernel Mechanics: It happens specifically inside a critical section, the part of code that reads or writes shared data, where if two threads execute it at overlapping times without coordination, their individual read-modify-write sequences can interleave and produce a result neither thread would have produced alone. Trade-off: Protecting every critical section with synchronization eliminates races but serializes access to that shared data, trading concurrency for correctness. Production Impact: A bug that only reproduces under load or appears intermittently, and vanishes when you add logging that slows execution down, is the classic fingerprint of a race condition in an unprotected critical section.
What are the three requirements for a valid Critical Section solution (Mutual Exclusion, Progress, Bounded Waiting)?
Concept: Any correct solution to the critical-section problem has to satisfy three properties simultaneously, not just prevent simultaneous access. Kernel Mechanics: Mutual Exclusion means at most one process can be inside the critical section at a time; Progress means that if no process is in the critical section, one of the processes wanting to enter must eventually be allowed to, so the decision can't stall forever with nobody chosen; Bounded Waiting means there's a finite limit on how many times other processes can enter the critical section before a specific waiting process is granted its turn. Trade-off: Satisfying all three is harder than satisfying mutual exclusion alone, naive solutions often provide exclusion but can still starve a specific waiter indefinitely, which is why real synchronization primitives are built and proven against all three properties together. Production Impact: A lock implementation that only guarantees mutual exclusion but not bounded waiting can produce a thread that technically never deadlocks but effectively starves under contention, a subtle and hard-to-diagnose fairness bug.
Compare a Mutex, a Binary Semaphore, and a Counting Semaphore.
Concept: All three limit how many threads can access a resource at once, but they differ in ownership rules and how many concurrent accesses they allow. Kernel Mechanics: A mutex allows exactly one thread in at a time and strictly enforces that only the thread that locked it can unlock it; a binary semaphore also only allows a count of 0 or 1, but any thread can signal it, release it, not just the one that originally waited on it; a counting semaphore generalizes this to allow up to N concurrent holders, decrementing on acquire and blocking once it hits zero, incrementing on release. Trade-off: A mutex's strict ownership rule prevents accidental release by the wrong thread, a common bug class with semaphores, but a counting semaphore is what you actually need to limit access to a pool of N identical resources rather than a single exclusive one. Production Impact: A connection pool limiting concurrent database connections to a fixed number is a textbook counting-semaphore use case; a mutex protecting a single shared data structure is the textbook mutex use case, and mixing the two up is a common source of subtle bugs.
Compare a Spinlock and a Mutex—when is busy-waiting actually faster than sleeping?
Concept: Both enforce mutual exclusion on a critical section, but a mutex puts the waiting thread to sleep, whereas a spinlock keeps the CPU core spinning in a tight loop checking the lock. Kernel Mechanics: When a thread fails to acquire a mutex, the kernel moves it to a wait queue and triggers a context switch to run another thread; a spinlock uses an atomic hardware instruction like Compare-and-Swap to poll the lock variable continuously without yielding the CPU. Trade-off: A mutex saves CPU cycles during long waits but pays the cost of two context switches, sleep and wake; a spinlock avoids context-switch overhead completely but wastes 100% of a CPU core if the lock is held for long, and is useless on a single-core CPU where the lock holder can't even make progress to release it. Production Impact: Inside the Linux kernel or low-latency trading engines, spinlocks are used when the critical section takes only a few nanoseconds, less than the roughly one-to-two-microsecond cost of a context switch, while mutexes are used whenever I/O or longer operations happen inside the lock.
What is a Monitor and a Condition Variable? Why must wait() always be called inside a while loop (spurious wakeups)?
Concept: A monitor bundles a lock together with the shared data and the operations on it, guaranteeing only one thread executes inside the monitor at a time; a condition variable lets a thread inside the monitor wait for some condition to become true, releasing the lock while it waits and reacquiring it on wakeup. Kernel Mechanics: Calling wait() on a condition variable atomically releases the monitor's lock and suspends the thread; another thread later calls signal() or notify(), which wakes the waiter, but the kernel doesn't guarantee the condition the waiter cared about is still true by the time it actually resumes and reacquires the lock, another thread may have gotten in first, or the wakeup itself can happen spuriously with no signal at all on some implementations. Trade-off: Re-checking the condition in a while loop after every wakeup adds a small, cheap re-check, but it's the only way to guarantee correctness against both legitimate race-y wakeups and spurious ones. Production Impact: A bug where a thread wakes up and proceeds on a stale condition, because wait() was called inside an if instead of a while, is a classic concurrency bug that only surfaces under specific timing and contention.
Explain the Producer-Consumer (Bounded Buffer) problem and write its synchronization logic.
Concept: Producer-Consumer models one or more producer threads filling a fixed-size buffer and one or more consumer threads draining it, needing coordination so producers don't overflow a full buffer and consumers don't read from an empty one. Kernel Mechanics: The standard solution uses two counting semaphores, empty (initialized to the buffer size, tracking free slots) and full (initialized to zero, tracking filled slots), plus a mutex protecting the buffer index itself; a producer waits on empty, acquires the mutex, inserts an item, releases the mutex, then signals full, and a consumer does the mirror image, waiting on full, removing an item, then signaling empty. Trade-off: Using two separate semaphores for empty and full slots rather than one shared counter lets producers and consumers block independently and correctly without racing on the count itself, at the cost of needing three synchronization primitives working together correctly. Production Impact: Every bounded message queue or task queue implementation, from Python's queue.Queue to Java's BlockingQueue, is an implementation of exactly this pattern under the hood.
Explain the Readers-Writers problem (Reader-preference vs. Writer-preference vs. Fair).
Concept: The Readers-Writers problem coordinates access to shared data where multiple readers can safely read simultaneously, but a writer needs fully exclusive access. Kernel Mechanics: A reader-preference solution lets any number of readers in concurrently and only blocks a writer until all current readers finish, which can starve writers if readers keep arriving; a writer-preference solution gives waiting writers priority over new readers, which can starve readers instead; a fair solution uses an explicit queue or ticketing so readers and writers are granted access roughly in arrival order, avoiding starvation of either side. Trade-off: Reader-preference maximizes read throughput, which suits read-heavy workloads, but risks writer starvation under constant read load; the fair solution avoids that risk but adds the overhead of maintaining and checking that ordering queue on every access. Production Impact: A read-write lock (like a pthread rwlock) is a direct implementation of this problem, and which preference policy a given rwlock implementation defaults to is a real, documented difference between platforms that matters under high contention.
What are atomic hardware instructions like Compare-and-Swap (CAS) and Test-and-Set?
Concept: These are CPU instructions that perform a read-modify-write sequence as a single, indivisible hardware operation, guaranteeing no other core can interleave in the middle of it. Kernel Mechanics: Test-and-Set reads a memory location's current value and unconditionally sets it to a new value in one atomic step, returning the old value, commonly used to implement a simple lock; Compare-and-Swap reads a location, compares it to an expected value, and only if they match writes a new value, also atomically, returning whether the swap succeeded, which lets software implement optimistic, lock-free updates to shared data. Trade-off: These instructions are the foundation every higher-level lock (mutex, semaphore, spinlock) is ultimately built on, but using them directly for lock-free algorithms is far harder to get correct than just using an existing lock, since the retry-on-failure logic has to be carefully reasoned about. Production Impact: Lock-free data structures in high-performance systems, concurrent queues, counters, are built directly on CAS loops specifically to avoid the overhead of a full mutex for very short, simple updates.
What is the ABA Problem in lock-free programming, and how do you solve it?
Concept: The ABA problem is when a CAS operation succeeds because a memory location holds the same value it started with, A, but the value actually changed to something else, B, and back to A in between, invisibly to the thread doing the compare. Kernel Mechanics: A thread reads value A, gets preempted, another thread changes it to B and back to A before the first thread resumes and performs its CAS, which succeeds because the value matches A again, even though the underlying object or state it pointed to may now be entirely different, a classic risk in lock-free stacks or queues reusing freed memory. Trade-off: The standard fix, pairing the value with a version counter or tag that increments on every modification, so a CAS checks both the value and the version, closes the hole but adds extra memory and a slightly more complex comparison to every atomic operation. Production Impact: Lock-free data structure libraries, like those used in high-performance concurrent queues, explicitly use tagged pointers or double-word CAS specifically to guard against this, since it is a subtle bug that can pass most testing and only manifest under rare interleavings in production.
What is a Memory Barrier (Fence), and why do modern CPUs reorder instructions?
Concept: A memory barrier is an instruction that forces a specific ordering on memory operations around it, preventing the CPU or compiler from reordering reads and writes across that point. Kernel Mechanics: Modern CPUs reorder instructions, within limits, to keep execution pipelines full and hide memory latency, which is invisible and harmless for single-threaded code but can break multi-threaded code that assumes writes become visible to other cores in the order they were issued; a memory barrier instructs the CPU to complete all memory operations before the barrier before starting any after it, restoring the ordering guarantee where it matters. Trade-off: Barriers are necessary for correct lock-free and low-level concurrent code, but they're expensive, since they force the CPU to stall and synchronize caches across cores, so they're used sparingly and only exactly where ordering actually matters. Production Impact: Writing correct lock-free code without understanding where memory barriers are needed is one of the most common sources of concurrency bugs that work fine on one architecture, like x86, which has stronger default ordering, and fail on another, like ARM, which reorders more aggressively.
Explain False Sharing in multi-core CPU caches and how cache-line padding fixes it.
Concept: False sharing happens when two threads on different cores modify logically unrelated variables that happen to sit on the same CPU cache line, causing unnecessary cache invalidation traffic between the cores even though the threads aren't actually sharing data. Kernel Mechanics: CPU caches operate in fixed-size cache lines, typically 64 bytes; if variable A used by core 1 and variable B used by core 2 fall within the same line, every write to A invalidates that whole cache line in core 2's cache, forcing it to refetch, even though core 2 never touches A at all, and vice versa, creating constant, unnecessary cross-core cache traffic. Trade-off: Padding each thread's hot variable out to its own full cache line eliminates this traffic entirely, but it wastes memory, since most of each padded line goes unused, a direct memory-for-speed trade-off. Production Impact: High-performance concurrent counters and per-thread statistics structures routinely pad their fields to cache-line size specifically to avoid this invisible but measurable performance cliff under multi-core contention.
What is RCU (Read-Copy-Update) in the Linux kernel?
Concept: RCU is a synchronization technique optimized for workloads with far more readers than writers, letting readers access shared data with essentially zero locking overhead. Kernel Mechanics: A reader simply dereferences a pointer to the current version of the data with no lock at all; a writer that wants to update the data creates an entirely new copy, modifies it, and then atomically swaps the pointer so new readers see the new version, while readers already in flight continue safely finishing against the old copy, which is only freed once the kernel confirms no reader could still be using it. Trade-off: Readers pay essentially no synchronization cost, which is exactly what very hot, read-heavy kernel data structures need, but writers pay more, a full copy plus a deferred-reclamation delay, and the technique only works cleanly for data where copy-and-swap is a sensible update model. Production Impact: The Linux kernel uses RCU extensively for data structures like routing tables, read constantly on every packet but updated comparatively rarely, specifically because a traditional lock there would be a serious bottleneck.
Level 5: Deadlocks
Deadlocks — interview questions
Answer in your head first, then open it.
Explain the Dining Philosophers problem and three ways to prevent deadlock in it.
Concept: Dining Philosophers models N processes each needing two shared resources, forks, at once to proceed, where the naive approach of each grabbing its left fork then its right fork can deadlock if everyone grabs their left fork simultaneously. Kernel Mechanics: One fix imposes a strict ordering, every philosopher picks up the lower-numbered fork first, breaking the circular wait; another introduces an arbitrator or waiter that only allows a philosopher to pick up both forks if both are actually available, checked atomically; a third limits the number of philosophers who can attempt to eat simultaneously to N-1 via a semaphore, guaranteeing at least one can always complete. Trade-off: The ordering fix is cheap and requires no central coordinator but can be awkward to apply to arbitrary resource-acquisition patterns beyond this specific problem; the arbitrator approach generalizes better but adds a centralized bottleneck. Production Impact: This is the canonical teaching example for deadlock from circular resource dependency, and the lock-ordering fix is exactly the real-world technique used to avoid deadlocks when code needs to acquire multiple locks: always acquire them in the same global order.
What are the four necessary conditions for a Deadlock (Coffman conditions)?
Concept: A deadlock can only occur if all four of these conditions hold simultaneously; breaking any single one prevents it. Kernel Mechanics: Mutual Exclusion means at least one resource must be held in a non-shareable mode; Hold and Wait means a process holding at least one resource is waiting to acquire additional resources held by others; No Preemption means resources can't be forcibly taken away from a process, only voluntarily released; Circular Wait means there's a cycle of processes each waiting on a resource held by the next one in the cycle. Trade-off: All four are necessary, so a system doesn't need to prevent all of them; it only needs to reliably break one, which is why different deadlock-avoidance strategies target different conditions, with correspondingly different costs. Production Impact: The lock-ordering convention used to avoid deadlocks in real code works by directly eliminating Circular Wait, the condition that's usually cheapest to break without sacrificing concurrency.
Compare Deadlock Prevention, Deadlock Avoidance (Banker's Algorithm), Deadlock Detection, and Deadlock Ignorance (Ostrich Algorithm).
Concept: These are four fundamentally different strategies for handling the possibility of deadlock, ranging from stopping it structurally to not worrying about it at all. Kernel Mechanics: Prevention restructures the system so one of the four necessary conditions can never hold, like enforcing lock ordering; Avoidance, like the Banker's Algorithm, grants a resource request only if doing so still leaves the system in a provably safe state where every process could still eventually finish, requiring upfront knowledge of each process's maximum resource needs; Detection lets deadlocks happen but periodically runs an algorithm to check the resource-allocation graph for cycles, then recovers by killing or rolling back a process; Ignorance simply assumes deadlocks are rare enough not to engineer around, and reboots if one ever actually locks the system up. Trade-off: Prevention and Avoidance guarantee no deadlock but add real runtime overhead or design constraints; Detection accepts deadlocks can happen but pays a periodic checking cost and a messier recovery; Ignorance costs nothing until a deadlock actually happens, then costs everything for that incident. Production Impact: Most general-purpose OSes, including Linux, use Ignorance for most resources precisely because the overhead of true avoidance (needing every process to declare its maximum resource needs upfront) isn't practical, reserving real prevention techniques, like lock ordering, for specific known-risky code paths.
Differentiate between Deadlock, Livelock, and Starvation.
Concept: All three describe a process failing to make progress, but for different underlying reasons. Kernel Mechanics: Deadlock is when two or more processes are each waiting on a resource the other holds, with neither able to proceed and both staying permanently blocked and idle; Livelock is when processes aren't blocked at all, they're actively running and repeatedly changing state, like both backing off and retrying in sync, but their combined actions never actually let either one complete; Starvation is when a process is technically eligible to proceed but a scheduling or resource-allocation policy keeps indefinitely favoring other processes over it. Trade-off: Deadlock is usually easier to detect, since the involved processes are visibly idle and blocked, while livelock is harder to spot since CPU usage looks active even though no real progress is happening; starvation can persist for a long time before it's even noticed as a problem distinct from normal contention. Production Impact: A retry loop with naive backoff that two threads happen to synchronize on is a real-world livelock, visible as high CPU usage with zero throughput, which is why good backoff logic adds randomized jitter specifically to break that synchronization.
How do you debug a deadlock in a live production process?
Concept: Debugging a live deadlock means identifying which threads are blocked, what each is waiting on, and whether those waits form a cycle, without necessarily being able to stop and restart the process. Kernel Mechanics: On Linux, taking a thread dump, via gdb attaching to the process, or a language runtime's own thread-dump facility, shows each thread's current stack and what lock or resource it's blocked on; cross-referencing which thread holds the lock another thread is waiting for reveals the resource-allocation graph, and a cycle in that graph confirms a true deadlock rather than just slow contention. Trade-off: Attaching a debugger to a live production process can itself pause or disrupt it momentarily, so some systems instead build in lightweight, always-on lock-state logging specifically so this diagnosis never requires attaching at all. Production Impact: Thread dumps, jstack for Java, gdb's thread apply all bt for native code, are the standard first tool reached for when a service suddenly stops responding but hasn't crashed, exactly the symptom profile of a live deadlock.
Level 6: Memory management
Memory management — interview questions
Answer in your head first, then open it.
What is the difference between 32-bit and 64-bit operating systems in terms of memory addressing and register width?
Concept: The bit-width of an OS and CPU determines both the size of its general-purpose registers and the range of memory addresses a single pointer can reference. Kernel Mechanics: A 32-bit system's pointers and address registers are 32 bits wide, capping addressable virtual memory at 4GB per process, often less after kernel reservation, while a 64-bit system uses 64-bit pointers, letting a single process address far more virtual memory, even though hardware and the OS typically expose fewer than the full 64 bits of that range today. Trade-off: Wider pointers mean every pointer-sized value in memory takes twice the space, increasing a program's memory footprint, in exchange for addressing room that modern workloads, especially in-memory databases, actually need. Production Impact: A 32-bit process hitting the 3-4GB address ceiling and failing to allocate more memory, despite free RAM existing system-wide, is a classic diagnostic pointing straight at the architecture's address-width limit.
Differentiate between Logical (Virtual) Address and Physical Address, and explain the role of the MMU.
Concept: A logical address is what a running program uses to reference memory; a physical address is the actual location in RAM that data lives at, and the two are almost never the same number. Kernel Mechanics: Every memory reference a program makes goes through the Memory Management Unit, a hardware component that translates the logical address into a physical one using the current process's page table, so the CPU core itself never directly touches a physical address a user program specified. Trade-off: This indirection is what makes virtual memory possible at all, letting a process use a contiguous-looking address space that's actually scattered across physical RAM and even disk, but every single memory access now requires a translation step, which is why hardware caches that translation, the TLB, aggressively. Production Impact: A process can safely be moved, swapped, or have its physical pages relocated by the OS without the process itself ever noticing, precisely because its logical addresses stay stable while the MMU's mapping underneath changes.
Compare Internal Fragmentation and External Fragmentation.
Concept: Both describe memory that's technically available but can't actually be used, for two different structural reasons. Kernel Mechanics: Internal fragmentation happens when a fixed-size allocation unit, a page, or a fixed-size memory block, is larger than what was actually requested, wasting the unused remainder inside that block; external fragmentation happens when free memory exists in enough total quantity but is scattered across many small, non-contiguous gaps, so no single gap is large enough to satisfy a new, larger request even though the sum of free space would be. Trade-off: Paging, using fixed-size blocks, eliminates external fragmentation almost entirely but introduces internal fragmentation in the last partial page of every allocation; segmentation does the reverse, no internal fragmentation but real external fragmentation as segments of varying sizes are allocated and freed over time. Production Impact: Modern systems overwhelmingly use paging specifically because internal fragmentation, capped at less than one page per allocation, is far more predictable and manageable than the external fragmentation a long-running segmented system would otherwise accumulate.
Compare Paging and Segmentation, and how modern x86-64 combines them.
Concept: Paging divides memory into fixed-size blocks regardless of what's stored in them; segmentation divides memory into variable-size, logically meaningful chunks like code, stack, and data. Kernel Mechanics: A paged system maps virtual pages to physical frames through a page table purely by address, with no concept of what the memory represents; a segmented system maps each named segment to a base-and-length region of physical memory, letting each segment be an arbitrary size matching its actual logical content; x86-64 in practice uses paging as its primary mechanism but keeps minimal vestigial segmentation, mostly for backward compatibility and a few specific base-register uses, rather than true variable-size segment-based memory management. Trade-off: Paging's fixed block size is simpler to manage and avoids external fragmentation, while segmentation's variable-size, meaning-aware chunks map more naturally to how programs are logically structured, but reintroduce external fragmentation as segments of different sizes are allocated and freed. Production Impact: This is why nearly every modern OS presents memory to applications as one flat, paged virtual address space, even though segmentation was historically important, because paging's fragmentation behavior is simply easier to manage at scale.
Walk through step-by-step how a virtual address is translated to a physical address using Multi-Level Page Tables.
Concept: Multi-level page tables split a virtual address into several index fields, each selecting an entry in a different level of a tree of page tables, rather than using one giant flat table. Kernel Mechanics: The CPU's MMU takes the virtual address, uses its top bits to index into the top-level page table, pointed to by a register like CR3 on x86, which yields a pointer to a second-level table, uses the next bits to index into that, continuing down each level, until the final level yields the actual physical frame number, which is then combined with the address's low-order offset bits to form the complete physical address. Trade-off: Multi-level tables let large, mostly-unused regions of a virtual address space go completely unrepresented, since an entirely empty top-level entry means none of the lower-level tables under it need to exist at all, saving huge amounts of memory compared to one flat table covering the whole address space, but each extra level adds another sequential memory access to every address translation. Production Impact: This is exactly why TLB caching matters so much: without it, every single memory access would require walking three or four levels of page tables in sequence, a real and measurable cost the TLB exists specifically to avoid paying on every access.
Why do we need Multi-Level Page Tables (or Inverted Page Tables) instead of a single flat page table?
Concept: A single flat page table covering a full 64-bit virtual address space would itself be enormous, even for processes that only actually use a tiny fraction of that space. Kernel Mechanics: Multi-level page tables solve this by letting entire unused branches of the address space go unallocated, since an empty top-level entry means nothing below it needs to exist; an inverted page table takes a different approach entirely, sized by physical memory rather than virtual address space, with one entry per physical frame recording which process and virtual page currently occupies it, looked up via a hash on the virtual address rather than a direct tree walk. Trade-off: Multi-level tables scale well with sparse address space usage but still require walking several levels per lookup; inverted page tables are compact and scale with physical memory size regardless of virtual address space, but a hash-based lookup can be slower on a collision and is a fundamentally different, less common architecture choice. Production Impact: x86-64 systems use multi-level tables because typical processes use a sparse, clustered subset of the available address space, exactly the pattern multi-level tables are efficient for.
What is the TLB (Translation Lookaside Buffer), what happens on a TLB Miss, and what is a TLB Shootdown?
Concept: The TLB is a small, fast hardware cache inside the CPU that stores recently used virtual-to-physical address translations, so the MMU can skip the full page-table walk for addresses it's seen recently. Kernel Mechanics: On a TLB hit, translation is essentially instant; on a TLB miss, the MMU has to perform the full multi-level page-table walk to find the translation, then cache the result in the TLB for next time; a TLB shootdown happens when one CPU core modifies a page table entry, such as during a page being unmapped or permissions changed, and must send an interprocessor interrupt to every other core that might have cached the old, now-stale translation in its own TLB, forcing them to invalidate it. Trade-off: The TLB makes the common case of repeated access to the same pages extremely fast, but a TLB miss costs a full page-walk's worth of extra memory accesses, and a shootdown, needed for correctness on multi-core systems, stalls other cores briefly to guarantee they're not using a stale mapping. Production Impact: Workloads that frequently remap memory, certain garbage collectors, some virtualization scenarios, can suffer real performance hits from frequent TLB shootdowns, which is part of why techniques like Huge Pages, fewer, larger translations to cache, exist.
Explain Demand Paging and what happens step-by-step during a Page Fault (Minor vs. Major page fault).
Concept: Demand paging means a process's pages are only actually loaded into physical memory when they're first accessed, not all at once when the process starts. Kernel Mechanics: When a program accesses a page not currently mapped to a physical frame, the MMU raises a page fault, trapping into the kernel's fault handler; a minor fault happens when the page is already in memory somewhere, perhaps in the page cache or shared with another process, and the kernel just needs to update the page table to point to it; a major fault happens when the page genuinely has to be read from disk, requiring the kernel to allocate a frame, issue the disk I/O, and block the process until that I/O completes. Trade-off: Demand paging means a process can start running with only a fraction of its code and data actually loaded, dramatically speeding up startup and reducing wasted memory for rarely-used pages, but it means the first touch of any given page always pays a fault, and a major fault specifically pays a real disk-latency cost. Production Impact: A sudden spike in major page faults, visible in vmstat's majflt column, is a strong signal a process's working set no longer fits comfortably in available RAM and is paging from disk.
Compare page replacement algorithms: FIFO, Optimal, LRU, Clock (Second-Chance), and LFU.
Concept: When physical memory is full and a new page needs to be loaded, a replacement algorithm decides which currently resident page gets evicted. Kernel Mechanics: FIFO evicts whichever page has been resident the longest, regardless of how recently it was used; Optimal evicts whichever page won't be needed for the longest time in the future, a theoretical best case requiring knowledge of the future; LRU evicts the page that hasn't been accessed for the longest time in the past, approximating Optimal using actual history; Clock, or Second-Chance, approximates LRU cheaply with a circular list and a per-page reference bit, giving a page a second chance if its bit is set before evicting it; LFU evicts whichever page has been accessed the fewest total times. Trade-off: True LRU gives good results but is expensive to track precisely on every access; Clock approximates it with far less bookkeeping, which is why it, not true LRU, is what most real kernels actually implement. Production Impact: Linux's page reclaim algorithm is a Clock-like approximation specifically because maintaining exact LRU ordering across millions of pages on every single memory access would be prohibitively expensive.
What is Belady's Anomaly, and which replacement algorithms suffer from it?
Concept: Belady's Anomaly is the counterintuitive result that, for some page-reference sequences, increasing the number of available physical frames can actually increase the number of page faults rather than decrease it. Kernel Mechanics: This occurs specifically under FIFO replacement, because FIFO's eviction choice, the oldest resident page, doesn't account for how soon that page will be needed again, so a larger frame count can shift which pages are resident at the moment of a future access in a way that happens to produce more faults than the smaller frame count would have. Trade-off: This anomaly is a structural weakness specific to FIFO and a few other non-stack-compliant algorithms; algorithms like LRU and Optimal are provably stack algorithms and are mathematically guaranteed never to exhibit it, which is one of the theoretical reasons LRU-family approximations are preferred in practice. Production Impact: This is primarily a theoretical result used to justify the stronger guarantees of LRU-family algorithms over FIFO, rather than something that shows up as a practical production incident, since few real systems use plain FIFO for page replacement today.
How does Virtual Memory enable two processes to safely load the same shared library (.so / .dll) in RAM once?
Concept: Virtual memory lets multiple unrelated processes each map the exact same physical pages of a shared library's code into their own, separate virtual address spaces, so the library only needs to be loaded into physical RAM once no matter how many processes use it. Kernel Mechanics: When a process loads a shared library, the kernel checks whether that library's pages are already resident in the page cache from another process having loaded it; if so, it maps those existing physical pages directly into the new process's page table at whatever virtual address that process uses, with the two processes' virtual addresses for the library potentially being completely different numbers pointing at the identical physical memory. Trade-off: This saves enormous amounts of physical RAM when many processes use common libraries, but it only works cleanly for the code and read-only data, any writable global state in the library typically needs special handling, like copy-on-write, so one process's modifications don't leak into another's. Production Impact: This is exactly why running a hundred instances of the same application doesn't use a hundred times the memory for things like libc, the shared, read-only portions are genuinely shared in physical RAM across all of them.
Level 7: Memory in practice
Memory in practice — interview questions
Answer in your head first, then open it.
What is Thrashing, what causes it, and how does the Working Set Model prevent it?
Concept: Thrashing is when a system spends almost all its time servicing page faults and swapping pages in and out, rather than making real forward progress on actual work. Kernel Mechanics: It's caused by too many processes, or processes with too-large working sets, competing for too little physical memory, so each process's pages keep getting evicted by others before they're actually done using them, triggering a new fault on almost every access; the Working Set Model addresses this by estimating each process's actual working set, the set of pages it's genuinely using in a recent time window, and only admitting or keeping running as many processes as the system can give their full working sets to simultaneously. Trade-off: Enforcing working-set-based admission avoids thrashing but can mean rejecting or suspending a process even though it's technically ready, sacrificing degree of multiprogramming to protect overall system throughput. Production Impact: A system with very high CPU usage but near-zero actual throughput, alongside a massive swap and page-fault rate, is the live symptom of thrashing, and the standard fix in production is simply adding memory or reducing concurrent load rather than tuning the replacement algorithm.
Explain Memory-Mapped Files (mmap) and when mmap() is faster than standard read()/write().
Concept: Memory-mapped files let a process access a file's contents directly as if it were an array in memory, rather than issuing explicit read() and write() calls to move data through a buffer. Kernel Mechanics: mmap() asks the kernel to map a file's pages directly into the calling process's virtual address space; accessing that memory for the first time triggers an ordinary page fault, which the kernel satisfies by reading the needed page from disk into the page cache and mapping it in, after which further access is just normal memory access with no syscall at all. Trade-off: mmap() avoids the extra copy that read() does, from the kernel's page cache into a separate user-space buffer, and avoids repeated syscall overhead for repeated access, but it ties the access pattern to page-fault behavior, which can be worse than a single large, sequential read() call for a file that's going to be read once, start to finish, anyway. Production Impact: Databases often mmap() large data files specifically to let the kernel's existing page cache do the caching work for them, avoiding the need to maintain a separate, redundant userspace cache of the same data.
How do malloc() and free() work under the hood (brk/sbrk vs. mmap, free lists, buddy allocator, slab allocator)?
Concept: malloc() and free() are userspace library functions that manage a heap carved out of memory the kernel grants the process, not direct kernel calls themselves. Kernel Mechanics: For smaller allocations, the allocator typically grows the heap via the brk or sbrk syscall, a single contiguous region, and manages chunks within it using a free list tracking which regions are currently unused and available to satisfy future requests; for larger allocations, it often uses mmap() directly to get a separate memory region from the kernel, since that memory can be independently unmapped and returned to the OS when freed, unlike heap space from brk, which can only shrink from the very end; the kernel itself, internally, often uses different allocators for its own needs, a buddy allocator for splitting and merging power-of-two-sized blocks of physical memory, and a slab allocator for efficiently handling many same-sized, frequently allocated kernel objects. Trade-off: Routing large allocations through mmap() lets that memory be released back to the OS immediately on free(), while small allocations on the brk-based heap can leave fragmented, unreturnable gaps that the process holds onto even after freeing, which is why a long-running process's memory usage, as reported by the OS, doesn't always shrink even after a big free(). Production Impact: Seeing a process's RSS stay high long after it's freed large amounts of memory is a common and often misunderstood symptom of exactly this: the memory was freed back to the allocator's free list, not necessarily back to the OS.
What is the Linux OOM (Out-Of-Memory) Killer, and how does it pick which process to kill (oom_score)?
Concept: The OOM Killer is a Linux kernel mechanism that steps in when the system is critically low on memory with no way to satisfy a new allocation, choosing to kill a process outright rather than let the whole system lock up. Kernel Mechanics: When memory is nearly exhausted, the kernel computes an oom_score for every process, factoring in how much memory it's using, how long it's been running, and an adjustable oom_score_adj value an administrator can set, and kills the process with the highest score, the one judged to free the most memory for the least overall disruption. Trade-off: Killing a process is a drastic, lossy recovery, any unsaved state in that process is gone, but it's the only option once memory really is exhausted and continuing to let every process fail its own allocations unpredictably would be worse. Production Impact: A process that unexpectedly dies with no clear application-level error, especially under memory pressure, is almost always the OOM Killer, confirmed by checking dmesg for an Out of memory, Killed process log line.
What is Memory Overcommit in Linux, and what are its risks?
Concept: Memory overcommit is Linux's default policy of allowing processes to request more virtual memory in total than the system actually has physical RAM and swap to back, on the assumption that most allocated memory is never fully touched. Kernel Mechanics: When a process calls malloc() or mmap(), the kernel typically just reserves the virtual address range and page-table entries without actually allocating physical pages; physical pages are only committed lazily, on first touch via a page fault, which is why overcommit works at all, most allocated memory genuinely never gets fully used. Trade-off: Overcommit lets the system support far more total virtual allocation than physical memory, which benefits efficiency broadly, but it means a later page fault, from a process just touching memory it already successfully allocated, can fail if the system is now truly out of physical memory, triggering the OOM Killer rather than a clean, predictable allocation-time failure. Production Impact: This is exactly why malloc() succeeding doesn't guarantee a later write to that memory will succeed, a subtlety that surprises engineers used to languages where allocation failure is the only expected failure mode.
Explain Huge Pages (Transparent Huge Pages) and how they reduce TLB misses for databases/AI workloads.
Concept: Huge Pages use a much larger page size, 2MB or 1GB instead of the standard 4KB, so a single TLB entry covers far more memory. Kernel Mechanics: With standard 4KB pages, a process touching gigabytes of memory needs an enormous number of distinct translations, far more than the TLB can cache at once, causing frequent TLB misses; using huge pages, the same amount of memory is covered by vastly fewer, larger translations, so a much higher proportion of a large working set's accesses hit in the TLB; Transparent Huge Pages let the kernel do this automatically without the application explicitly requesting huge pages itself. Trade-off: Huge pages dramatically cut TLB miss rates for large, memory-intensive workloads, but they increase internal fragmentation, since any allocation that doesn't need the full 2MB still occupies it, and automatic promotion or demotion of pages can itself introduce unpredictable latency spikes as the kernel reorganizes memory in the background. Production Impact: Databases and large in-memory AI workloads routinely enable huge pages explicitly specifically to reduce TLB-miss overhead, while some latency-sensitive systems actually disable Transparent Huge Pages because its background compaction work introduces exactly the kind of unpredictable pause they're trying to avoid.
What is Swapping vs. Paging, and why does swap thrashing destroy database performance?
Concept: Paging moves individual pages between memory and disk as needed; swapping, in the stricter historical sense, moves an entire process's memory out to disk at once, though the term is now commonly used loosely to mean paging to a swap device. Kernel Mechanics: Under memory pressure, the kernel's page replacement mechanism writes infrequently-used pages out to a swap device to free physical RAM for more active pages, reading them back in on a page fault if they're touched again; for a database holding a large buffer pool or working set, this means pages the database assumes are resident in fast RAM can silently be out on slow disk-backed swap. Trade-off: Swap lets a system survive momentary memory pressure gracefully rather than failing allocations outright, but once a workload's actual active working set exceeds physical RAM, every access to swapped-out data costs a full disk round-trip, turning what the database expected to be a RAM-speed operation into a disk-speed one, repeatedly. Production Impact: Database administrators almost universally recommend disabling or minimizing swap on dedicated database servers, since swap thrashing under load can degrade query latency by orders of magnitude compared to simply failing fast or provisioning more RAM.
Differentiate between a Memory Leak, a Dangling Pointer, and a Segmentation Fault (SIGSEGV).
Concept: These are three distinct memory bugs, one about memory never being released, one about a pointer outliving what it points to, and one about the kernel actually catching an invalid access. Kernel Mechanics: A memory leak is memory a program allocated and lost track of, never calling free() on it, so it stays reserved and unusable for the rest of the process's life even though nothing references it anymore; a dangling pointer is a pointer that still holds the address of memory that's already been freed or gone out of scope, so dereferencing it reads or writes memory that may now be reused for something else entirely; a segmentation fault is the kernel's own detection, via the MMU, that a program tried to access memory outside its valid mapped address space, or without the right permissions, raising a trap that typically terminates the process via SIGSEGV. Trade-off: A leak degrades a long-running process slowly and might never crash it outright; a dangling pointer can silently corrupt unrelated data without immediately crashing anything, which makes it especially dangerous; a segfault at least fails loudly and immediately, which, perversely, makes it the easiest of the three to actually notice and debug. Production Impact: Tools like valgrind and AddressSanitizer exist specifically to catch leaks and dangling-pointer use before they reach production, since by the time a dangling-pointer bug actually manifests as a visible segfault, the real corruption may have happened much earlier and elsewhere.
Level 8: Storage and I/O devices
Storage and I/O devices — interview questions
Answer in your head first, then open it.
Explain DMA (Direct Memory Access) and how it reduces CPU overhead during I/O.
Concept: Direct Memory Access lets a device transfer data to or from main memory without the CPU copying every byte itself. Kernel Mechanics: The CPU sets up a DMA controller with a source, destination, and length, the controller then moves the data directly over the memory bus while the CPU is free to run other work, and the controller raises an interrupt only once the transfer completes. Trade-off: DMA frees the CPU from busy-copying data, which matters enormously for disk and network throughput, but it requires physical memory to stay pinned and not be paged out mid-transfer, and coordinating cache coherency between the CPU's cache and the DMA'd memory adds real complexity. Production Impact: A network card or disk controller's advertised throughput is only achievable because of DMA; without it, high-throughput I/O would saturate the CPU just copying bytes.
What is the Linux Page Cache, and how do fsync(), fdatasync(), and O_DIRECT bypass or flush it?
Concept: The page cache is memory the kernel uses to transparently cache file data that's been read from or written to disk, so repeated access to the same data doesn't have to hit the slower disk again. Kernel Mechanics: A normal write() call actually just updates the page cache and marks those pages dirty, returning to the caller immediately, with the kernel writing dirty pages back to disk later in the background; fsync() forces all of a specific file's dirty pages to be written to disk and blocks until that's confirmed complete; fdatasync() does the same but skips flushing certain metadata that doesn't affect how the data itself will be read back; O_DIRECT bypasses the page cache entirely, having the application's I/O go straight to and from the disk, no kernel caching layer involved at all. Trade-off: Relying on the page cache's default background write-back is far faster for the common case, but it means a crash between a write() call returning and the actual disk write happening can lose that data; fsync() guarantees durability at the cost of blocking until the slow disk I/O actually completes. Production Impact: Databases call fsync() on their write-ahead log specifically at the moment they need to guarantee a committed transaction will survive a crash, accepting that latency cost exactly where durability actually matters, while using ordinary cached writes elsewhere.
Compare RAID 0, RAID 1, RAID 5, RAID 6, and RAID 10 in terms of fault tolerance, read/write speed, and usable capacity.
Concept: RAID levels combine multiple physical disks into one logical volume using different strategies that trade off speed, usable capacity, and how many disk failures the array can survive. Kernel Mechanics: RAID 0 stripes data across disks with no redundancy at all, maximizing speed and usable capacity but tolerating zero disk failures; RAID 1 mirrors data identically across pairs of disks, surviving one failure per mirrored pair at the cost of halving usable capacity; RAID 5 stripes data with a single parity block distributed across disks, tolerating one disk failure while using only one disk's worth of capacity for parity; RAID 6 adds a second independent parity block, tolerating two simultaneous failures at the cost of two disks' worth of capacity; RAID 10 combines mirroring and striping, offering both strong fault tolerance and good speed at the cost of using only half the raw capacity, like RAID 1. Trade-off: More redundancy always costs usable capacity or write performance, since every write to a redundant array has to update the mirror or recompute parity, which is why the right choice depends on whether a workload prioritizes raw capacity, write speed, or surviving multiple simultaneous failures. Production Impact: RAID 5's single-parity design has a well-known production risk: a second disk failure during the already-slow rebuild of a failed first disk destroys the whole array, which is exactly why RAID 6 or RAID 10 is now generally recommended for large, important arrays.
Compare disk scheduling algorithms: FCFS, SSTF, SCAN (Elevator), C-SCAN, and LOOK.
Concept: These algorithms decide the order in which a disk's read/write head services a queue of pending requests at different track locations, to minimize the total seek distance traveled. Kernel Mechanics: FCFS services requests strictly in arrival order regardless of their physical location; SSTF (Shortest Seek Time First) always services whichever pending request is physically closest to the head's current position; SCAN moves the head in one direction across the whole disk, servicing every request it passes, then reverses, like an elevator; C-SCAN only services requests in one direction and then jumps back to the start without servicing on the return trip, giving more uniform wait times; LOOK behaves like SCAN but only travels as far as the furthest pending request rather than all the way to the end of the disk. Trade-off: SSTF minimizes total seek distance but can starve requests far from the head's current busy area; SCAN-family algorithms bound the worst-case wait more fairly at some cost to average seek efficiency compared to SSTF. Production Impact: These algorithms matter far less on SSDs, which have no real seek cost, which is exactly why modern Linux I/O schedulers default to simpler, lower-overhead policies on SSD-backed storage and reserve the more elaborate seek-minimizing schedulers for spinning disks.
How do SSDs (NAND Flash) differ from HDDs in OS I/O scheduling, wear leveling, and the TRIM command?
Concept: SSDs have no moving parts and no meaningful seek time, which changes both how the OS should schedule I/O to them and introduces entirely new concerns, like limited write endurance, that HDDs never had. Kernel Mechanics: An HDD's I/O scheduler is built around minimizing physical head movement across tracks; an SSD has roughly uniform access time regardless of logical address, so that scheduling goal is irrelevant, and the OS instead focuses on maximizing parallel queue depth, since flash chips can service many requests concurrently; wear leveling, handled by the SSD's own controller, spreads writes evenly across all flash cells so no single cell wears out from repeated overwriting faster than the rest; TRIM is a command the OS sends telling the SSD which logical blocks no longer hold valid data, letting the controller erase and reclaim those flash cells proactively in the background rather than discovering they're free only on the next write attempt. Trade-off: Without TRIM, an SSD's controller doesn't know which blocks are actually free, since a filesystem delete only removes a directory entry at the OS level, leading to write-amplification and slowing performance over time as free space effectively shrinks from the SSD's perspective; TRIM fixes this but costs extra commands sent to the device. Production Impact: An older SSD without TRIM support enabled is a well-documented cause of gradual, unexplained write-performance degradation over months of use, fixed simply by enabling periodic or continuous TRIM.
Explain epoll, select, and poll for I/O multiplexing—why is epoll O(1) while select is O(n)?
Concept: All three let a single thread monitor many file descriptors at once for readiness, readable, writable, without blocking separately on each one, but they scale very differently as the number of watched descriptors grows. Kernel Mechanics: select() and poll() both require the caller to pass the entire set of watched file descriptors to the kernel on every single call, and the kernel has to linearly scan all of them to check which are ready, an O(n) cost that's paid again on every call regardless of how many are actually ready; epoll instead has the application register its set of watched descriptors once with the kernel, which then maintains that set internally and, via callbacks tied to each descriptor's readiness events, can return just the subset that's actually ready without rescanning everything, an O(1) operation relative to the total watched set. Trade-off: select() and poll() are simpler and more portable across different Unix systems, but epoll's registration-based design is what lets a single process efficiently monitor tens of thousands of connections, the exact workload select's linear scan can't handle well. Production Impact: This is precisely why every high-performance Linux event loop, nginx, Node.js's libuv, Redis, is built on epoll rather than select, since select's O(n) scanning becomes the actual bottleneck once a server holds more than a few hundred open connections.
Compare Blocking I/O, Non-Blocking I/O, I/O Multiplexing, and Asynchronous I/O (io_uring).
Concept: These are four different models for how a program's thread relates to the time an I/O operation actually takes to complete. Kernel Mechanics: Blocking I/O suspends the calling thread entirely until the operation finishes; Non-Blocking I/O returns immediately with an error if the operation can't complete right away, leaving the application to retry later itself; I/O Multiplexing, select, poll, epoll, lets one thread wait on many non-blocking descriptors at once and only act once at least one is actually ready; true Asynchronous I/O, as implemented by io_uring, lets the application submit an operation and get notified via a completion queue once it's fully done, with the kernel handling the entire operation, including the data transfer itself, without the application polling or retrying at all. Trade-off: Blocking I/O is the simplest to program but needs one thread per concurrent operation; non-blocking and multiplexing let one thread handle many operations but push retry and readiness-tracking complexity onto the application; true async I/O removes even that polling overhead but has historically been harder to use correctly and wasn't uniformly well supported until io_uring matured. Production Impact: io_uring's adoption in high-throughput Linux servers and databases is specifically because it eliminates both the thread-per-connection cost of blocking I/O and the syscall overhead that even epoll-based multiplexing still pays on every readiness check.
Level 9: File systems
File systems — interview questions
Answer in your head first, then open it.
What is an Inode in Unix/Linux, and what metadata does it store (and not store—like the filename)?
Concept: An inode is the data structure that holds all the metadata about a file, everything except the file's name, which is stored separately in the directory that points to it. Kernel Mechanics: An inode stores the file's size, permissions, owner, timestamps, link count, and pointers to the actual data blocks on disk that hold its content; a directory itself is really just a list of filename-to-inode-number mappings, so a file's name lives entirely in whichever directory entries reference it, not in the inode. Trade-off: Separating the name from the inode is exactly what makes hard links possible, since multiple directory entries, even with different names, can point to the same inode, but it means an inode on its own, by number, tells you nothing about what the file is called. Production Impact: This is why renaming a file is nearly instant regardless of file size: it only edits a directory entry, not the inode or the data blocks at all.
Explain the difference between a Hard Link and a Soft (Symbolic) Link at the inode level.
Concept: A hard link is a second directory entry pointing directly to the same inode as the original file; a symbolic link is a separate small file that just contains the path string to another file. Kernel Mechanics: Creating a hard link increments the target inode's link count and adds a new directory entry referencing that same inode number, so there's no real distinction between the original and the hard link, both are equally the file; a symbolic link gets its own, different inode, whose data is literally the text of a path, which the kernel resolves by reading that path and following it to whatever it currently points at. Trade-off: Hard links can't cross filesystem boundaries, since inode numbers are only meaningful within one filesystem, and can't point to directories in most systems, while symbolic links can point anywhere, even to a path that doesn't exist, but following one costs an extra lookup, and a dangling symlink, pointing nowhere, fails only when actually accessed. Production Impact: Deleting the original file that a hard link points to doesn't affect the hard link at all, since the inode and data persist as long as the link count is above zero, while deleting a target a symlink points to leaves the symlink intact but now broken.
What happens when you rm a file that is currently open and being read by a running process?
Concept: On Linux, removing a file that a process still has open doesn't actually delete its data immediately, because the kernel tracks file existence through reference counts, not just the directory listing. Kernel Mechanics: rm only unlinks the directory entry and decrements the inode's link count; the kernel does not actually free the underlying disk blocks until both the link count and the count of currently-open file descriptors referencing that inode drop to zero, so a process that already had the file open keeps reading and writing it normally even though it no longer appears in any directory listing. Trade-off: This reference-counting design prevents a process from crashing mid-read or mid-write just because another user deleted the file out from under it, at the cost of hidden disk consumption, the space isn't actually reclaimed, until every handle to that now-invisible file closes. Production Impact: This is the exact cause of the classic df versus du mismatch, where df shows a disk as nearly full but du can't account for all the space, because some of it is held by deleted-but-still-open files; lsof is the standard tool to find the process holding the handle.
Explain the VFS (Virtual File System) layer in Linux and how it supports ext4, XFS, ZFS, and NFS simultaneously.
Concept: The VFS is an abstraction layer inside the Linux kernel that presents one consistent file-operation interface, open, read, write, to user space, regardless of which actual filesystem implementation is handling a given file underneath. Kernel Mechanics: Every concrete filesystem, ext4, XFS, NFS, implements the same set of VFS-defined operations, like a plugin conforming to a common interface, and the VFS dispatches a syscall like read() to the correct filesystem's actual implementation based on which filesystem the target file lives on, translating between the generic VFS data structures (like its own generic inode representation) and that filesystem's specific on-disk or on-wire format. Trade-off: This abstraction is what lets a single system mount several completely different filesystems simultaneously and have every application work with them identically, but it means each filesystem has to be adapted to fit the VFS's generic model, which can leave some filesystem-specific features unreachable through the standard interface. Production Impact: Being able to mount an NFS network share right alongside a local ext4 partition and have ordinary commands like ls and cp work identically on both is the VFS's abstraction working exactly as intended.
Compare contiguous, linked, and indexed (inode-based) file block allocation methods.
Concept: These are three different strategies a filesystem can use to track which disk blocks belong to which file. Kernel Mechanics: Contiguous allocation stores a file's data in one unbroken run of consecutive disk blocks, needing only a start block and length to describe it; linked allocation stores each block with a pointer to the next block of the same file, scattered anywhere on disk; indexed allocation, the approach inodes use, keeps a dedicated list, or tree, of pointers to every block belonging to the file in one central index structure. Trade-off: Contiguous allocation gives the fastest sequential access, since blocks are physically adjacent, but suffers badly from external fragmentation as files are created and deleted over time; linked allocation avoids that fragmentation but makes random access to a specific offset slow, since you have to follow pointers from the start; indexed allocation gives fast random access without needing contiguous space, at the cost of the index structure itself needing storage and, for very large files, multiple levels. Production Impact: Virtually every modern filesystem uses indexed allocation via inodes specifically because real workloads need both fast random access and resilience to fragmentation that contiguous and linked allocation can't both provide.
What is a Journaling File System (e.g., ext4, NTFS), and how does it prevent corruption after a sudden power loss?
Concept: A journaling filesystem records the metadata changes it's about to make in a dedicated log, the journal, before actually applying them to the main filesystem structures. Kernel Mechanics: Before modifying inodes, directory entries, or block bitmaps for an operation, the filesystem first writes a journal entry describing exactly what it's about to do; if the system crashes or loses power mid-operation, on reboot the filesystem replays the journal, either completing or cleanly rolling back any operation that was interrupted, rather than leaving the actual filesystem structures in a half-updated, inconsistent state. Trade-off: Writing to the journal before the real structures costs extra I/O on every metadata change, but it turns what used to require a full, slow filesystem-wide consistency check after any unclean shutdown into a fast, bounded journal replay instead. Production Impact: This is why modern Linux and Windows systems recover from a power loss in seconds rather than running the hours-long fsck a non-journaling filesystem like old ext2 would need on a large disk.
Compare Copy-on-Write (CoW) File Systems (ZFS, Btrfs) with traditional journaling file systems.
Concept: Both protect against corruption from an interrupted write, but a CoW filesystem never overwrites existing data in place at all, while a journaling filesystem still does, just with a log recording the intent first. Kernel Mechanics: A CoW filesystem like ZFS or Btrfs writes any modified block to a brand-new location on disk entirely, then atomically updates the metadata pointers to point at the new block instead of the old one, leaving the old data fully intact until it's explicitly reclaimed, which is also what makes instant, consistent snapshots possible; a journaling filesystem still modifies data in place but protects the metadata about that modification via the journal. Trade-off: Never overwriting in place makes CoW filesystems naturally resistant to corruption and gives them cheap, reliable snapshots almost for free, but it requires more sophisticated space-management and garbage-collection logic to reclaim old, superseded blocks, and can fragment data more over time than in-place writes would. Production Impact: ZFS and Btrfs's instant snapshot capability, used heavily for backups and safe upgrade rollbacks, is a direct consequence of the CoW design, something a traditional journaling filesystem can't offer nearly as cheaply.
What are File Descriptors, and how do the per-process FD table, system-wide open file table, and inode table link together?
Concept: A file descriptor is a small integer a process uses to refer to an open file, but that integer is just an index into a chain of kernel structures that ultimately point at the real file. Kernel Mechanics: Each process has its own file descriptor table mapping small integers to entries in the kernel's system-wide open file table; each open file table entry tracks things specific to that one open() call, like the current read/write offset and the access mode, and points to an entry in the system-wide inode table, which holds the file's actual on-disk metadata and is shared by every open file table entry referencing that same file. Trade-off: This three-level indirection is what lets two processes open the same file independently, each with their own offset and mode, while still sharing the single underlying inode, but it also means the same inode can be referenced by many open file table entries at once, which is exactly the reference-counting mechanism that lets rm on an open file not immediately free its data. Production Impact: lsof walks exactly this chain, listing which process holds which file descriptors pointing at which open files and inodes, which is why it's the standard tool for finding what's holding a file open or consuming unexpected disk space.
Level 10: Security and protection
Security and protection — interview questions
Answer in your head first, then open it.
What is ASLR (Address Space Layout Randomization), and how does it protect against buffer overflow exploits?
Concept: ASLR randomizes the virtual memory addresses where a process's key regions, the stack, heap, shared libraries, executable code, are loaded each time it runs, rather than using the same predictable addresses every time. Kernel Mechanics: When the kernel loads a new process, instead of placing these regions at fixed, well-known addresses, it adds a random offset chosen fresh for that execution, so an attacker who doesn't know this run's specific randomized layout can't reliably predict the address of a specific function or gadget to jump execution to. Trade-off: ASLR makes many classic memory-corruption exploits, like a buffer overflow that overwrites a return address with a hardcoded target, far less reliable without actually fixing the underlying memory-safety bug, but it's probabilistic protection, not a guarantee, and can sometimes be defeated by information leaks that reveal the actual randomized addresses in a given run. Production Impact: ASLR being enabled or disabled is a real, checkable system hardening setting, cat /proc/sys/kernel/randomize_va_space on Linux, and disabling it, sometimes done for debugging reproducibility, measurably weakens a system's exploit resistance.
Level 11: Virtualisation and containers
Virtualisation and containers — interview questions
Answer in your head first, then open it.
What is Virtualization (Type 1 vs. Type 2 Hypervisors), and how do containers (Docker) differ from Virtual Machines at the OS level?
Concept: Virtualization and containers both let multiple isolated workloads share one physical machine, but at very different levels of the stack. Kernel Mechanics: A Type 1 hypervisor runs directly on hardware and hosts full guest OS kernels with their own virtualized CPU, memory, and devices; a Type 2 hypervisor runs as a process inside a host OS and does the same; a container, by contrast, shares the single host kernel and is isolated using kernel features like namespaces and cgroups rather than running a second kernel at all. Trade-off: A VM's full kernel isolation is stronger but far heavier, each is a complete OS with its own memory and boot time, while a container's shared-kernel model is lightweight and starts in milliseconds but means every container on a host is only as isolated as the kernel's namespace and cgroup boundaries actually are. Production Impact: A kernel vulnerability can, in principle, let one container escape to affect others sharing that kernel, exactly the risk VMs are chosen over containers to avoid in strongly multi-tenant settings.
How do Linux Namespaces (PID, NET, MNT, UTS, IPC, USER) and cgroups (Control Groups) power Docker containers?
Concept: Namespaces and cgroups are the two Linux kernel features that together make containers possible, one controlling what a process can see, the other controlling how much it can use. Kernel Mechanics: Each namespace type isolates a different kind of global kernel resource for the processes inside it, a PID namespace gives a process its own isolated view of process IDs starting from 1, a NET namespace gives it its own network interfaces and routing table, MNT gives it its own filesystem mount points, UTS its own hostname, IPC its own isolated message queues and shared memory, and USER its own user and group ID mapping; cgroups separately limit and account for how much CPU, memory, and I/O bandwidth a group of processes can consume, regardless of what they can see. Trade-off: Because containers share the single host kernel underneath all these namespaces, the isolation is strong for normal workloads but weaker than a full VM's separate kernel, a kernel-level vulnerability can in principle let a process escape its namespace boundaries. Production Impact: Running docker stats and seeing per-container CPU and memory limits enforced is cgroups at work directly; running ps inside a container and seeing only that container's own processes starting from PID 1 is the PID namespace at work.
Level 12: Linux in practice and interview prep
Linux in practice and interview prep — interview questions
Answer in your head first, then open it.
Explain the Load Average numbers (1m, 5m, 15m) in Linux top/uptime—why can load average be high even when CPU usage is low?
Concept: Load average is a measure of system demand, not just CPU usage, it's a running average of the number of processes that are either actively running on a CPU or waiting for a resource, averaged over the last 1, 5, and 15 minutes. Kernel Mechanics: Critically, Linux's load average counts processes in an uninterruptible sleep state, typically waiting on disk I/O, as part of the load figure, not just processes waiting for CPU time, so the kernel is counting demand for the system broadly, including storage, not demand for the CPU specifically. Trade-off: This makes load average a convenient single number for overall system busyness, but it conflates two very different problems, CPU contention and I/O contention, into one figure, so a high load average doesn't by itself tell you which resource is actually the bottleneck. Production Impact: A load average of 20 on an 8-core machine with CPU usage sitting at only 10% almost always means many processes are stuck waiting on slow or stalled disk I/O, not that the CPU is overloaded, which is exactly why you check iostat or iotop alongside load average, never load average alone.
Differentiate between SIGKILL (-9), SIGTERM (-15), SIGINT, and SIGHUP. Why can't SIGKILL be caught?
Concept: These are all signals used to influence or stop a running process, but they differ in whether the process gets any chance to react before it's affected. Kernel Mechanics: SIGTERM asks a process to terminate gracefully and can be caught by a signal handler to clean up resources first; SIGINT is sent when a user presses Ctrl+C in a terminal and is also catchable; SIGHUP was originally sent when a controlling terminal closed, and is commonly repurposed by daemons today to mean reload your configuration; SIGKILL is handled entirely at the kernel level and never delivered to the process's own signal-handling code at all, the kernel simply terminates the process directly, which is exactly why no handler, no cleanup code, no matter how the process is written, can intercept or ignore it. Trade-off: Catchable signals let a well-behaved process shut down cleanly, flushing buffers, closing connections, but a process that's hung or refuses to respond to SIGTERM needs SIGKILL as the guaranteed last resort, at the cost of skipping any cleanup entirely. Production Impact: Graceful service shutdowns always send SIGTERM first and wait a grace period before escalating to SIGKILL, exactly so well-behaved processes get a chance to clean up, while unresponsive ones are still guaranteed to actually stop.
Explain the /proc and /sys virtual filesystems in Linux.
Concept: /proc and /sys are filesystems that don't represent real files on disk at all, they're a live, file-like window into the kernel's own in-memory data structures, generated on the fly when read. Kernel Mechanics: /proc exposes per-process information, like /proc/
How do Unix Domain Sockets differ from TCP Loopback (127.0.0.1) Sockets in performance and kernel overhead?
Concept: Both let two processes on the same machine communicate over a socket-style API, but a Unix domain socket is a purely local, filesystem-addressed IPC mechanism, while a TCP loopback socket goes through the full networking stack even though the data never leaves the machine. Kernel Mechanics: A Unix domain socket's data goes directly through the kernel's socket buffers between the two processes with no network-protocol processing at all, no IP headers, no TCP sequence numbers, no checksum calculation; a TCP loopback connection, even though it stays entirely inside the kernel and never touches a physical NIC, still runs through the real TCP/IP stack, constructing and parsing packet headers, tracking connection state, and computing checksums exactly as it would for a remote connection. Trade-off: The Unix domain socket's lack of networking overhead makes it measurably faster and lighter for strictly local communication, but it only works between processes on the same machine, since it's addressed by a filesystem path rather than an IP address and port, so it can't be used if the two sides might ever need to be on different machines. Production Impact: Local database clients and reverse proxies often default to Unix domain sockets for same-machine connections specifically to shave off this unnecessary TCP/IP stack overhead, falling back to TCP sockets only when true network reachability is required.
What is Zero-Copy Networking (sendfile()), and how does it avoid copying data between kernel space and user space?
Concept: Zero-copy networking lets a process send a file's contents out over a socket without the data ever actually passing through the process's own user-space memory. Kernel Mechanics: A traditional send would read a file from disk into a user-space buffer via read(), then write that buffer out to a socket via write(), a path that copies the data from kernel page cache to user space and then back into kernel socket buffers, twice through the user/kernel boundary; sendfile() instead tells the kernel directly to transfer data from a file descriptor straight to a socket descriptor, entirely inside kernel space, with the data touching user space zero times. Trade-off: This eliminates real CPU time and memory bandwidth spent on unnecessary copying, which matters enormously at high throughput, but it only works for the specific case of moving data from one kernel-visible source straight to a socket, it's not a general-purpose I/O primitive the application can arbitrarily process data through along the way. Production Impact: Web servers serving large static files, and systems like Kafka moving data between disk and network at high throughput, use sendfile()-style zero-copy specifically because it measurably reduces CPU usage per byte transferred compared to the traditional read-then-write path.
How do eBPF programs safely run custom logic inside the Linux kernel without recompiling it?
Concept: eBPF lets a developer load small, custom programs into the kernel at runtime, to observe or modify kernel behavior, without writing a kernel module or rebuilding the kernel itself. Kernel Mechanics: An eBPF program is compiled to a restricted bytecode, loaded into the kernel through a dedicated syscall, and before it's allowed to run at all, the kernel's eBPF verifier statically analyzes it to guarantee it will terminate, won't access memory outside what it's permitted to, and can't crash the kernel, only after passing that verification is the bytecode actually JIT-compiled to native instructions and attached to whatever kernel hook point it targets, a network event, a syscall, a kernel function entry. Trade-off: The verifier's strict safety guarantees are what make it safe to let arbitrary user-supplied programs run inside the kernel at all, but they also restrict what an eBPF program is allowed to express, no unbounded loops, limited instruction count, which rules out some logic that would be trivial to write as a traditional kernel module. Production Impact: Tools like Cilium for container networking and bpftrace for live kernel tracing are both built on eBPF specifically because it lets them observe and influence kernel behavior in production with safety guarantees that a traditional, unverified kernel module simply doesn't have.
Scenario: Your Linux server has 30% free RAM, yet a process crashes with Out of Memory or Cannot allocate memory. What could cause this?
Concept: Having free RAM reported overall doesn't guarantee a specific allocation request can actually succeed, since several other limits can block an allocation well before physical memory is truly exhausted. Kernel Mechanics: A cgroup memory limit on that specific process or its container can cap it far below the system-wide free memory total, so a container with a 512MB limit fails long before the host's actual free RAM runs out; vm.max_map_count can cap the number of distinct memory mappings a process is allowed, failing allocations like mmap() calls even with plenty of RAM; external memory fragmentation can mean no single contiguous block large enough for a specific request exists, even though the sum of smaller free blocks is large; and the kernel's own overcommit settings can, in stricter modes, refuse an allocation it estimates the system can't ultimately back. Trade-off: Each of these limits exists for a real reason, cgroups for container isolation, map count caps for stability, but together they mean free RAM alone is never a sufficient diagnostic; the actual limiting factor has to be identified specifically. Production Impact: Checking cgroup limits (cat /sys/fs/cgroup/memory/.../memory.limit_in_bytes or the equivalent for the process's container), sysctl vm.max_map_count, and dmesg for OOM Killer activity, in that order, is the standard path to finding which specific constraint actually triggered the failure.
Scenario: df -h shows a disk is 100% full, but du -sh /
shows only 40% of the space is used. How do you find and fix the missing 60%?* Concept: This is the signature symptom of one or more processes still holding open file descriptors to files that have already been deleted from the directory tree, so du, which only walks the visible directory structure, can't see or count them. Kernel Mechanics: rm only removes a file's directory entry and decrements its inode link count; the kernel only actually frees the underlying disk blocks once both the link count and the count of open file descriptors referencing that inode reach zero, so a deleted file still held open by a running process continues consuming its full disk space invisibly. Trade-off: This reference-counting behavior is what protects a running process from crashing just because another user deleted a file it was using, but it means standard directory-walking tools like du are structurally blind to this category of space usage, since by definition the file no longer appears anywhere in the directory tree they're walking. Production Impact: Running lsof +L1, which specifically lists open files with a link count of zero, or lsof | grep deleted, finds the exact process and file descriptor responsible; the space is reclaimed either by restarting that process gracefully or, without downtime, by truncating the file directly through its /proc/
Scenario: CPU utilization is at 100%, with 80% spent in %sy (system/kernel time) and only 20% in %us (user time). What is likely happening?
Concept: A CPU spending the overwhelming majority of its busy time in kernel mode rather than user mode means the bottleneck is in how the kernel is servicing the workload, not in the application's own computation. Kernel Mechanics: High %sy time is typically caused by excessive context switching from too many competing threads or processes, a very high rate of system calls being made by the application, lock contention inside the kernel itself (common with heavy multi-threaded syscall use), or a high rate of page faults forcing constant kernel-level memory management work; in every one of these cases, the CPU is genuinely busy, but almost entirely on kernel bookkeeping around the actual work rather than the work itself. Trade-off: Diagnosing which specific cause is responsible requires different tools for each, so there's no single fix, but all of them share the same symptom profile of busy CPU with little application-level throughput to show for it. Production Impact: vmstat's cs (context switches) and in (interrupts) columns, combined with strace -c to see which syscall is actually dominating, is the standard diagnostic path; a server spawning far too many short-lived threads per request is one of the most common real-world causes of exactly this pattern.
Scenario: A server becomes completely unresponsive, iowait is at 90%, and disk read/write throughput is maxed out. How do you diagnose which process is causing it?
Concept: High iowait specifically means CPUs are sitting idle, not busy, waiting on outstanding disk I/O to complete, which is a fundamentally different bottleneck from high CPU usage and needs different tools to pin down. Kernel Mechanics: The kernel reports a CPU as being in iowait when it has no runnable process to schedule because the processes that would otherwise run are all blocked waiting on disk I/O; this is visible system-wide in vmstat or top, but identifying which specific process is actually generating that I/O load requires per-process I/O accounting, which iotop surfaces directly, and pidstat -d supplements with historical per-process read and write rates over time. Trade-off: The system-wide iowait percentage alone tells you there's a disk bottleneck but nothing about its source, so jumping straight to a fix without identifying the actual offending process risks fixing the wrong thing entirely. Production Impact: Running iotop in accumulated mode during the incident, or pidstat -d 1, directly surfaces which process is responsible for the bulk of the disk throughput, at which point the fix is usually either throttling, rescheduling, or investigating why that specific process suddenly needs that much I/O.
Scenario: You see thousands of processes in the D (Uninterruptible Sleep) state in ps aux. Why can't you kill them with kill -9, and what does it indicate?
Concept: A process in uninterruptible sleep is blocked inside a kernel-level I/O operation that the kernel has decided cannot be safely interrupted partway through, which is exactly what makes it immune to being killed while stuck there. Kernel Mechanics: When a process issues certain low-level I/O requests, often to a hung or failing disk, or a stalled NFS mount, the kernel puts it into state D specifically because the operation is in a critical section of kernel code where partially unwinding it could leave kernel data structures in an inconsistent state; signals, including SIGKILL, are only actually delivered and acted on when a process next returns to a point where the kernel can safely check for them, which a process stuck in D state structurally never reaches until the underlying I/O operation itself finally completes or times out. Trade-off: This design protects kernel-level data consistency at the cost of making an application-level kill command powerless against processes stuck this way, no matter how urgently you need them gone. Production Impact: A large number of processes stuck in D state is a strong signal of a failing storage device or a hung network filesystem mount underneath them, and the real fix is addressing that underlying device or mount, often by forcibly unmounting or resetting it, rather than trying to kill the individual stuck processes directly.