A cloud AI chat feels weightless. Type a question, get an answer, close the tab. Nothing about the experience suggests machinery.
I spent the last few months building the opposite: a sovereign AI knowledge system for a client, designed to answer questions from a fixed set of a vessel’s technical documentation, with no cloud dependency in the final deployment. Everything has to run on a single workstation-class machine that the client owns and controls.
I expected the project to teach me about small models. It did. What I didn’t expect was how much it would teach me to respect the big ones, because once you’ve fought for every gigabyte on one machine, you start to see what that weightless chat window is actually resting on.
What the system does, and what it won’t
The system’s job sounds simple: help a crew work faster with their own manuals. Ask it how to isolate a pump for maintenance, what the shutdown sequence is for a piece of machinery, or what protective equipment a procedure requires, and it answers directly from the approved documents, with a citation to the page.
Ask it why a pump keeps tripping, or whether it’s acceptable to keep operating with a known fault, and it declines. It shows what the documents say and hands the judgement back to a qualified person. That’s not a gap in the system. It’s the design. Diagnosis and safety calls need experience and authority, not document lookup, and no amount of compute changes that.
The same boundary applies anywhere the documents are approved and the stakes are real. Picture a hospital’s equipment procedures, a factory’s safety SOPs, an airline’s maintenance manuals: in every case, the useful system answers how do I do this and refuses is this safe.
So the first “can’t do” in this post’s title is deliberate. The system can’t answer beyond its documents because I built it not to. The second “can’t do” is the one the hardware imposed, and that’s where the education started.
The first lesson: memory is the wall
On a single machine, GPU memory is not a spec you shop for. It’s the wall you design against.
Most people never meet this wall, because most people are calling an API and letting someone else worry about the hardware underneath it. Build offline and you meet it on day one. The model weights, the working memory for the conversation (the KV cache), the activations, the runtime overhead: all of it has to fit inside one card’s VRAM, and the moment it doesn’t, you’re either slow or you’re out.
Here’s what models actually demand, roughly, at full FP16 precision:
| Model | Parameters | VRAM needed |
|---|---|---|
| Llama 3.2 3B | 3B | ~6 GB |
| Llama 3.1 8B | 8B | ~16 GB |
| Llama 3.3 70B | 70B | ~140 GB |
Now hold that against the hardware. The best consumer card you can buy today carries 32 GB. Top workstation cards reach 96 GB. A 70B model at full precision fits on none of them.
So how does anyone run the models behind the famous chat services, which are far larger still?
They don’t run on a GPU. They run on a building.
The data centre answer to the memory wall isn’t a bigger card. It’s abandoning the idea that a model lives on one card at all.
Frontier models are sharded: weights split across many accelerators, layers distributed, experts routed to different chips, the KV cache managed across all of it, stitched together by interconnects fast enough that the seams don’t show. NVIDIA’s DGX B200 links eight GPUs into about 1.4 TB of combined memory, and frontier deployments chain many such systems together. A 70B model, the kind that won’t fit on any single card you can buy, is commonly spread across two to four data-centre accelerators. The trillion-parameter class lives across racks.
This is why AI companies buy racks, not GPUs. And it’s the first thing offline building recalibrates in your head: the gap between what one machine holds and what the frontier requires isn’t a factor of two. It’s orders of magnitude, bridged by an engineering discipline that has almost nothing in common with plugging in a graphics card.
Squeezing, and what squeezing costs
Back on my one machine, the response to the wall is compression. Quantisation shrinks a model’s numerical precision to fit more capability into less memory: a 32B model that would want around 64 GB at full precision runs in roughly 20 GB at 4-bit.
But compression is a negotiation, not a free lunch. Squeeze moderately and the model barely changes. Squeeze hard and you can damage the exact behaviours a grounded system depends on: following the retrieved pages faithfully, and admitting when the documents don’t contain the answer. For a system whose whole value is the boundary I described above, that second behaviour is everything. A model that fills gaps with confident invention isn’t smaller. It’s broken.
Whether a squeezed model still behaves is something you test, question by question, against the behaviour you proved with a capable model first. Months of this negotiation changes how you see the cloud services. Every effortless answer in that weightless chat window is a model nobody had to squeeze, running at a precision nobody had to ration, on memory nobody had to fight for. The weightlessness is purchased elsewhere.
The twist: my one machine matched the cloud
Here’s where the story stops being a lament about small hardware, because my benchmark results surprised me in the other direction.
When I tested the full pipeline on a rented 32 GB GPU against a hosted cloud API, same questions, same knowledge base, same day, the local card was marginally faster: under five seconds per answer on average, with a lower worst case than the cloud service.
For a moment that looks like a paradox. If one rented card matches the cloud, what are all those racks for?
The answer is the honest heart of this post, and of its title: the racks are for everything my offline system can’t do, and was deliberately designed not to attempt.
My system matches cloud speed because it only ever attempts a sliver of what the cloud attempts. Retrieval hands the model a few pages, the model reasons inside them, one inference pass, done. No open-ended conversation stretching across a whole session’s history. No agent loops planning, acting, checking and re-planning, each loop another pass through the model. No tool calls fanning out mid-answer. No million concurrent users arriving at once. And no general knowledge, because the design is to have none: ask my system about equipment that isn’t in its manuals and the correct response, the one I tested for, is that it tells you so.
Strip all of that away and a single well-fed card keeps up. Add any of it back and the workstation falls behind immediately, not because the card is weak, but because those workloads are shaped for distributed infrastructure. The racks aren’t there to answer bounded questions faster. They’re there to make the unbounded ones possible at all.
What the frontier actually is
That’s the appreciation I didn’t have before building offline.
A frontier AI service isn’t a bigger chatbot. It’s the sharded model, plus the orchestration deciding which hardware serves which request, plus the memory management keeping thousands of simultaneous conversations resident, plus the routing, monitoring, safety systems and specialised models working around the core one, all engineered so that you, in the chat window, feel none of it. The product isn’t just intelligence. It’s intelligence with the machinery hidden.
I used cloud AI for years without seeing any of that. It took failing to fit a model onto one very good graphics card, repeatedly, to understand what I’d been leaning on.
Both, deliberately
None of this sent me back to the cloud for the client’s system. The offline build is the right call for it, precisely because the work is bounded: private documents, grounded answers, hard limits, hardware the client owns. My own benchmark says that within those limits, local gives up nothing.
But I’ve stopped thinking of offline as the principled option and cloud as the lazy one. They’re different machines for different shapes of work. One machine, close to the knowledge, for the questions you can draw a boundary around. The building full of racks for the questions you can’t.
I still build for the single machine. I just no longer mistake the chatbox for the computer behind it.
