Computer Fundamentals

A modern core does not execute your instructions one at a time in order. It runs a deep pipeline — typically 14–20 stages — with several instructions in flight at once, issues them out of order, executes speculatively past branches, and retires them back in program order so the result looks sequential.

That matters operationally because it decouples clock speed from work done. The number you actually care about is IPC — instructions per cycle. A core at 3 GHz with IPC 0.4 is doing less work than one at 2 GHz with IPC 1.8. When a service gets slower after a deploy and CPU utilisation looks identical, IPC is usually where the answer is: the code started missing cache, and the core is spending its cycles stalled rather than retiring work.

What actually stalls a core

• Cache misses — the biggest one. A last-level miss costs ~200–300 cycles of doing nothing useful.
• Branch mispredictions — the pipeline is flushed and refilled, ~15–20 cycles. Unpredictable branches in a hot loop are expensive.
• Dependency chains — instruction N+1 needs N's result, so out-of-order execution has nothing else to run.

The CPU never reads one byte from RAM. It reads a cache line — 64 bytes on every x86-64 and most ARM64 parts — and everything in that line comes along for free. This single fact drives most of the performance difference between two implementations of the same algorithm.

Walking an array of structs sequentially is fast because each miss pulls in the next several elements. Chasing pointers through a linked list is slow because every hop is a fresh miss with nothing useful alongside it — the same O(n) traversal can differ by 10× in wall time.

The two localities

• Temporal — you touched it recently, so it's probably still cached. Loop counters, hot config, the top of a call stack.
• Spatial — you touched the neighbour, so it's already in the line. Arrays, struct fields accessed together, contiguous file reads.

Your process sees a flat virtual address space. The MMU translates each virtual address to a physical one by walking page tables — four levels on x86-64, so an untranslated access could cost four extra memory reads. The TLB caches recent translations to avoid that walk; it holds only a few thousand entries.

With a 4 KiB page and ~1,500 TLB entries, a core can cover roughly 6 MB of memory before it starts missing the TLB on every access. A process with a 40 GB working set that jumps around randomly will spend a startling fraction of its time walking page tables.

Huge pages

A 2 MiB page covers 512× more memory per TLB entry. For databases, JVMs with large heaps, and anything with a big random-access working set, huge pages can be a double-digit-percent win. Transparent Huge Pages (THP) does it automatically — and is also a classic latency culprit, because the compaction it does to find contiguous memory stalls the process that triggered it. Most database vendors tell you to turn THP off and use explicit hugepages instead. They are right, for that workload.

Storage is where the latency pyramid gets steep. An NVMe read is roughly a thousand times slower than DRAM; a spinning disk seek is a hundred thousand times slower. Any design decision that turns a memory access into a disk access is worth a hundred micro-optimisations elsewhere.

IOPS and latency are not the same problem

An NVMe device advertising 800k IOPS achieves that at high queue depth — many requests in flight at once. A single-threaded process issuing one synchronous read at a time gets device latency, not device throughput: maybe 12k IOPS from the same hardware. If your benchmark says the disk is fine and your application says it isn't, queue depth is usually the gap.

On a multi-socket server, each CPU package has its own memory controller and its own directly attached DRAM. Accessing memory on the other socket goes across the interconnect and costs roughly 1.5× the latency and less bandwidth. The kernel tries to allocate memory on the node where the allocating thread runs — but if the scheduler later migrates that thread, every access becomes remote.

This is why large single-process databases are usually pinned. It's also why a container without CPU affinity can show 30% variance run to run on the same hardware for no visible reason.

A NIC receiving a packet does not interrupt the CPU for each byte. It DMAs the frame straight into a ring buffer in RAM, then raises one interrupt to say "there is work". The kernel's top half acknowledges it fast and defers the real processing to a softirq, which is where most of the network stack actually runs.

Under load the kernel switches to NAPI polling — interrupts off, poll the ring — because at a million packets per second, interrupt overhead alone would consume the machine.

Where this bites in production

By default all NIC interrupts may land on CPU 0. One core saturates handling softirqs while 47 others idle, and your throughput ceiling has nothing to do with your application. RSS (receive-side scaling) spreads flows across multiple queues, and IRQ affinity pins each queue to a core — ideally one on the same NUMA node as the NIC.

Saving registers and swapping page tables takes roughly 1–5 µs. That number is misleading, because the expensive part is what happens afterwards: the incoming process finds the L1 and L2 caches full of the outgoing process's data, and the TLB partly flushed. It runs slowly for tens of microseconds while it re-warms.

A machine doing 200k context switches per second is not spending 20% of its time in the switch code — it is spending far more than that running cold. This is the real argument for CPU pinning on latency-sensitive services, and the reason thread-per-request models fall over at high concurrency while event loops don't.

You will make a hundred design decisions before you ever profile anything. Rough magnitudes are what keep those decisions sane — and they are stable, because they are set by physics and by hardware generations, not by your code.

• A cache miss to DRAM is ~250 cycles. If a hot loop misses every iteration, you have a memory problem, not a CPU problem.
• An NVMe read is ~1,000× a DRAM read. Caching a value that costs a disk read is worth doing even if the cache hit rate is only 50%.
• A same-region network round trip is cheaper than an HDD seek. A remote cache can genuinely be faster than local spinning disk.
• Cross-region is ~100 ms and unfixable. Any design with N sequential cross-region calls has an N × 100 ms floor. Batch them or move the compute.
• 1 Gbps is 125 MB/s. Divide bits by eight before promising anyone a transfer window.