Hardware for AI Builders: How the Machine Your Code Runs On Actually Works

Quick answer

• A computer is four components cooperating: a CPU that executes instructions, RAM that holds what is in use, storage that remembers things permanently, and usually a GPU for bulk arithmetic.
• The performance of all of it is governed by one idea — a memory hierarchy where each tier is roughly a thousand times slower than the one above it. Caching and batching exist to avoid the slower tiers.
• Spec sheets are readable once you know the units. Gigahertz is cycles, not work; bits and bytes differ by eight; a vCPU is a share of a processor, not a whole one.
• For AI work the constraint is memory, not speed. Whether a model runs is a capacity question, and a card with enough memory beats a faster card that cannot hold it.
• The machine you rent is not the machine on your desk. It is a share of a busier, slower, differently built computer.
• Every concept below links to a guide written for someone who has never opened a case.

Why hardware is the blind spot for AI builders

AI coding assistants are extremely good at the layer above the machine. They will produce a working HTTP handler, a schema, a test suite, and a Dockerfile in one response. What they cannot see is what the code will run on: how much memory the container is allowed, whether the disk is local or network-attached, how many real cores exist behind the two vCPUs on the invoice, or whether the person asking has a graphics card at all.

That gap is where a specific kind of bug lives. A generated cache that grows with traffic is correct code that exhausts memory. A generated loop that queries the database once per row is correct code that makes the whole page slow. A generated deployment that recommends a local model is correct code that cannot load on the target machine. None of these are syntax errors, and none of them are caught by tests — they are caught by knowing what the machine is doing.

This page is the map. Read the section that matches what just broke, follow the linked guide, and come back for the next one.

The five-minute model of a computer

Four components, each with one job.

The CPU executes instructions. It is the only part that “thinks”. Every program, in every language, becomes a list of small operations this chip performs one after another, per core. Modern chips have several cores so that independent work can happen at the same time, and small amounts of very fast memory called cache to avoid waiting on RAM. What Is a CPU (and Why More Cores Isn’t Always Faster)? covers cores, threads, clock speed, and the x86-versus-ARM split you will meet the moment you build a container for a different machine.

RAM holds everything currently in use. It is fast and temporary — power off and it is empty — and it is the resource your app runs out of first. Running out of memory does not slow an application down politely; the operating system kills something. What Is RAM (and Why Does Your App Run Out of Memory)? explains capacity, bandwidth, ECC memory, and the swap file that turns a memory problem into a performance problem. In containers, the same failure arrives as Why Do My Containers Keep Getting Killed (OOMKilled)?, and it is what Kubernetes Requests vs Limits: Why Your Pod Gets OOMKilled is really about.

Storage remembers things when the power is off. It is permanent and far slower, which is why the operating system works hard to avoid touching it. SSD vs HDD vs NVMe: What Storage Speed Actually Means sorts out what the numbers on a drive mean, why random access is the figure that matters for a database, and why cloud storage is quietly slower than the drive in your laptop.

The GPU does the same thing to millions of numbers at once. It is not a faster CPU; it is a differently shaped one, and that shape happens to be exactly what neural networks are made of. What Is a GPU (and Why Is It Different From a CPU)? and What Is VRAM (and Why Does a Model Not Fit)? cover it, and GPU vs CPU for AI: When Is a GPU Actually Worth It? is the buying decision, which is usually “rent or call an API”.

Everything else in a computer — the motherboard connecting them, the network card, the power supply — exists to feed these four.

The one idea that explains performance

If you remember one thing from this page, remember the ladder. Accessing a value in a CPU register takes about a third of a nanosecond. From cache, a few nanoseconds. From RAM, around a hundred nanoseconds. From a fast NVMe drive, tens of microseconds. From a spinning disk, around ten milliseconds. Across a network to another data centre, tens of milliseconds and up.

Each step down that ladder is roughly a thousand times slower than the one above it. Almost every performance technique you will ever meet is a way of avoiding a step down: cache the query result, batch the writes, stream instead of loading, index the column, keep the connection open. And almost every performance problem is a hangover from a step you took without noticing. When What Is Database Indexing (and Why Is My Query Slow)? says an index turns a scan into a lookup, it means one step down the ladder becomes one step near the top.

This is also why “the CPU is only at 10%” and “the app is slow” are not contradictory. Low CPU utilisation is the signature of a program that is waiting — on memory, on disk, on a network call — and How a Program Actually Runs: From Your Python to the CPU walks through what happens between pressing enter and the work finishing, including the system calls that make waiting expensive.

Reading a spec sheet without being fooled

Four unit traps, and they account for most of the confusion.

Bits versus bytes. Storage and memory are measured in bytes; network speeds are measured in bits per second. A 1 Gbps link carries about 125 MB/s, not a thousand. Decimal versus binary. Drive vendors count in powers of ten and operating systems count in powers of two, which is why a 512 GB drive displays as roughly 476 GB. Gigahertz is not work. It counts cycles per second, and the amount of work per cycle varies by design, which is why a newer 3 GHz processor can beat an older 4 GHz one. Throughput is not latency. Bandwidth is how wide the pipe is, latency is how long one trip takes, and widening a pipe does nothing for a round trip.

Bits, Bytes, and Bandwidth: The Units Behind Every Spec Sheet works through all four with the arithmetic, and it is worth ten minutes because it stops you from accepting an estimate that is wrong by a factor of eight — which is exactly the kind of error AI-generated capacity planning makes.

What actually limits your app

When an app is slow, the culprit is almost always one of four resources, in this order of likelihood.

Memory comes first, because running out of it is fatal rather than gradual. Watch available memory under peak load, not the figure at idle. Storage comes second, and specifically the number of small random operations: an unindexed query on local NVMe is fast, and the same query against network-attached cloud storage is unusable. Network comes third — calls to third-party APIs, chatty service-to-service traffic, and large payloads moved unnecessarily. CPU is last for request-driven applications, because most of their time is spent waiting, and adding cores helps only when the work genuinely parallelises.

Databases for AI Builders: Why Your AI-Generated App Gets Slow and Loses Data is the companion guide for the most common source of all four, and How to Add Redis Caching to Your App is the cheapest way to buy headroom without buying hardware. How to Fix ‘Too Many Connections’ in Postgres covers the failure that looks like a hardware limit and is not.

AI hardware, in one section

Training a model needs a GPU. Fine-tuning usually does. Running one — inference — is a spectrum where a CPU is often good enough, and where the deciding constraint is memory capacity rather than speed. A model’s weights take roughly its parameter count times the bytes per parameter: a 7-billion-parameter model is about 14 GB at half precision and around 4 GB quantized to four bits, before you add the cache that grows with conversation length. If that does not fit in the card, the model does not run, however fast the card is.

What Is an NPU (and Do You Need an ‘AI PC’)? covers the newest tier, whose real benefit is battery life for small always-on models rather than any new capability. And the honest default for most projects is an API call: you pay per token instead of per month, and someone else owns the hardware.

The machine you rent is not the machine on your desk

This is the part that produces the most confusing bug reports. A rented instance is a virtual machine on a shared host, and the provider oversubscribes: more vCPUs are sold than there are physical threads, on the assumption that most customers are idle most of the time. Your instance reports four vCPUs; what that means in practice depends on what your neighbours are doing. What Is a vCPU (and Why Isn’t It a Real Core)? explains steal time, CPU credits on burstable instances, and why a benchmark on a quiet host flatters everything.

Rented machines are also server-class hardware, which is a different design from a PC: ECC memory, redundant power, hot-swap disks, and out-of-band management for a machine with no keyboard attached. PC vs Data Center Server: Why a Server Looks Slower and Costs More covers why a server can cost five times a gaming PC and lose a single-threaded benchmark, and ARM vs x86: Why Your Laptop, Your Server, and Your Cloud Disagree explains the mismatch that produces “exec format error” when an image built on a laptop is deployed to a server of another architecture.

Once the model is clear, sizing is a procedure rather than a guess: How to Choose a VPS Size for an AI-Built App starts from the shape of the workload, names the three numbers to watch, and argues for starting small and resizing on evidence. How to Choose a Cloud Provider for Your AI-Generated App, Budget Cloud and PaaS Compared: Hetzner, DigitalOcean, Railway, Fly.io, Render, Vercel, Netlify, and Ultra-Budget Cloud Providers Compared: Contabo, UpCloud, Kamatera, IONOS cover where to rent it, and How to Reduce Your Cloud Bill Without Breaking Production covers paying less for it. If you would rather run the machine yourself, How to Deploy an App to a Cheap VPS With k3s and What Is k3s (and When Should You Use It)? are the next step.

Where this bites vibecoders

The canonical failure is a class of bug that has no signature in the code. An assistant generates an in-memory cache, an unbounded queue, a query inside a loop, a script that loads a whole table, or a deployment that builds a model locally. Every one of those is idiomatic code, every one passes review, and every one is a decision about resources the model cannot see. The vibecoder’s experience is that the app “just died” — a container killed with no stack trace, a page that is slow everywhere and fast on a laptop, a model that fails to load on the machine it was planned for. The diagnosis is always the same question, asked in order: is it out of memory, is it waiting on storage, is it waiting on the network, or is it actually short of CPU? Answering it in that order finds the problem in minutes instead of days, and it requires nothing more than the model of the machine above.

Where AI coding assistants get this wrong

• Writing code that assumes unlimited memory, then running it in a container with a limit.
• Doing one file, network, or database operation per loop iteration instead of batching.
• Sizing a recommendation by the example architecture instead of the actual workload.
• Assuming production hardware matches the development machine’s speed and architecture.
• Suggesting a local model without checking available memory or the presence of a GPU.
• Treating a vCPU as a dedicated core and setting worker counts far above it.
• Benchmarking sequential disk throughput when the workload is random access.
• Building a container image for one processor architecture and deploying to another.
• Confusing disk space with memory, so a full volume is read as a memory leak.
• Ignoring a burstable instance’s baseline until the CPU credits run out.

Checklist

• Learn the latency ladder: register, cache, RAM, SSD, disk, network — each roughly a thousand times slower than the last.
• Know which resource your workload actually waits on: memory, storage, network, or CPU.
• Set memory limits on containers and check the app fits with headroom at peak.
• Batch I/O instead of performing one operation per item in a loop.
• Confirm whether production storage is local or network-attached before trusting a local benchmark.
• Check the architecture of both the build machine and the deployment target.
• Compute a model’s memory requirement before choosing any hardware for it.
• Remember that a vCPU is a share of a shared machine, and monitor steal time.
• Prefer an API or a rented GPU over buying hardware until usage is continuous.
• Size machines from measurement, and fix the inefficient query before the bigger instance.

FAQ

Do I need to learn hardware to build software with AI?

Not to write code — but you do need enough of it to catch the failures that hardware causes. Most of them are invisible in the source: an unbounded cache that exhausts memory, a full disk that stops a database writing, a container image built for the wrong processor, or a model whose weights do not fit in a graphics card. A mental model of four components and one table of latency magnitudes prevents nearly all of them.

What is the single most useful thing to know about computer performance?

That accessing memory is roughly a thousand times slower than a CPU cycle, and reading from disk is another thousand times slower than that. Almost every performance problem is a variant of waiting for data to arrive, and almost every optimisation is a way of avoiding that wait by caching, batching, or keeping data close to the processor.

Why does my app run fine locally and fall over in production?

Usually because the two machines are different in ways no one wrote down. Production often has fewer cores, less memory, network-attached storage instead of local NVMe, and a shared processor that other customers are also using. Your development machine is a quiet, generously provisioned machine; the production instance is a slice of a busy one.

Should I learn assembly or C to understand hardware?

No. Assembly tells you how a specific processor encodes instructions, and C teaches you about memory layout — both are useful later and unnecessary at the start. What pays off immediately is knowing what each component does, how fast each tier of memory is relative to the others, and which resource your workload is actually waiting on.

Related topics

• What Is a CPU (and Why More Cores Isn’t Always Faster)?
• What Is RAM (and Why Does Your App Run Out of Memory)?
• SSD vs HDD vs NVMe: What Storage Speed Actually Means
• What Is a GPU (and Why Is It Different From a CPU)?
• What Is VRAM (and Why Does a Model Not Fit)?
• PC vs Data Center Server: Why a Server Looks Slower and Costs More
• What Is a vCPU (and Why Isn’t It a Real Core)?
• How to Choose a VPS Size for an AI-Built App

Sources

• Linux Kernel Documentation
• NVM Express
• NVIDIA CUDA programming guide
• Arm architecture