Open a desktop computer and the RAM slides out. Two clips, a firm push, done. Any competent teenager can upgrade it.
Open the same machine and look at the graphics card. The memory there is soldered permanently to the board. On the newest AI hardware it sits beside the processor on the same package, a few millimetres away, joined by an interface thousands of lanes wide. You cannot upgrade it. You cannot move it. If you want more, you buy a different card.
Someone made that decision deliberately, and it cost a great deal of money to implement.
What the machine is actually doing
A language model is an enormous pile of numbers, learned during training. A small one has a few billion of them. A large one has hundreds of billions.
To produce even a fragment of its answer, the machine runs through almost all of them. Then it does the same thing again for the next fragment, and again for the one after that.
So the machine is doing less thinking than the word suggests, and a great deal of fetching and carrying. That changes what the memory has to be. Big enough to hold the numbers, which everyone expects. Close enough to hand them over fast, which almost nobody does.
Why closeness is the thing that matters
Here is the step that usually gets skipped. Distance does not slow things down mainly by making the signal travel further. Inside a single machine, the trip time is not the headline cost. Distance matters because of what it permits.
A very wide connection can only be built across a very short gap. Current high-bandwidth memory talks to the processor over an interface 2,048 lanes wide. You can etch that many parallel connections across a few millimetres of silicon. What you cannot do is run that same interface down a metre of cable and expect the signals to arrive clean, in step, and without consuming absurd amounts of power.
So proximity buys width. Width buys bandwidth. Bandwidth is what the model actually needs, because the model is moving numbers rather than pondering them.
That is why the memory is glued down.
You can watch this happen
Anyone who has run a model on their own machine has met this rule without being introduced to it. Load a model that fits comfortably in the graphics card’s memory and it answers at conversational speed. Load one slightly too large, with system-memory offloading enabled, and something strange happens. It still works. It just crawls.
Nothing about the model changed. Part of it now lives in system RAM instead of graphics memory. A few centimetres further away, and on a completely different road.
The extra centimetres are not the cost in themselves. The traffic has moved onto a much narrower path. The centimetres are simply the reason the wide road stopped being available.
The same rule, all the way up
It does not stop at the edge of your graphics card. Beside the processor, then a few centimetres away in system memory, then across a cabinet of machines wired together with dedicated high-speed links, then between cabinets over ordinary networking, then between buildings, then between cities. At every step the road narrows and the cost rises.
Between cities you eventually meet something that does not negotiate. Light in glass fibre covers about a thousand kilometres in five milliseconds, before any switching or routing is counted. Memory beside the processor answers in a tiny fraction of that.
The further the numbers travel from the processor, the narrower, slower and more energy-expensive the path becomes. Not because length is expensive in itself, but because length forecloses the options that made the short path fast.
Which makes this a geometry problem rather than a shopping problem. That distinction is easy to miss when the industry is discussed purely in terms of how many chips someone has bought.
The strange part
The obvious question is what biology does about all this, given that brains are the only working example of general intelligence anyone has. The interesting answer is that the brain never built our problem in the first place.
In a computer, memory and calculation live in different places. Stored numbers sit in memory, arithmetic happens in the processor, and every operation requires shipping data between them. That shipping is the bottleneck this whole article has been describing.
In a brain, storage and processing are deeply interwoven rather than assigned to separate boxes. The same synapses that carry and transform signals also undergo the changes associated with learning and memory. There is no direct equivalent of loading the weights, because there is no separate place to load them from.
The brain still pays for distance, and its structure carries the evidence. A great deal of its volume is long-range connective fibre. And the cortex is folded, crumpling a sheet far larger than the skull into the space available, which increases surface area enormously without every connection becoming proportionally longer.
And the whole arrangement runs on roughly twenty watts. A data centre training a frontier model draws enough electricity for a small city.
The comparison is directional rather than exact. The substrates differ, and biological learning is not the same operation as training a model from scratch. It is still the most striking number in computing.
So
We did not glue the memory down to save space. We did it because moving numbers has a cost, and the cost is set by how far they have to go.
That is a small fact about a graphics card. It is also, it turns out, a fact about data centres, about where the industry’s money actually goes, and about what all that concrete is really buying.
More on that soon.
