Imagine an assistant on a ship at sea. The crew can ask it questions about the vessel’s technical manuals: how to service a pump, what a warning light means, which checklist applies. There’s no internet connection for days at a time, and even in port, the client doesn’t want sensitive documentation leaving their control.
That means no ChatGPT, no Claude, no cloud AI of any kind. The entire system has to live on one machine, on board.
I’ve spent months building exactly this kind of system for a client, and the ship is just one version of the setting. Swap it for a hospital basement, a remote mine site, or a factory that keeps its processes off the internet, and the constraint is identical: the AI has to work where the cloud can’t reach, on documents that can’t leave the building. The experience taught me that “offline AI” is nothing like the cloud AI most people have used, in ways that are mostly invisible until you try. This post is the whole journey in plain English: what offline AI can realistically do, why smaller AI models quietly make things up, how you find the right model, and what happened when I finally tested the system on real hardware.
First, “AI” isn’t one thing
The most common question I get is, “Can you run AI without the internet?”
The answer is: it depends entirely on what kind of AI work you mean, because the label covers several very different jobs.
| Type of work | What it looks like | Runs offline? |
|---|---|---|
| Chat | Ask a question, get an answer | Yes. Small models can run even on a laptop, although quality and speed depend heavily on the hardware |
| Summarising | Condense a long document | Yes, if the machine can handle the model and the required context |
| Querying your documents | Ask questions of your own manuals and files | Yes. This is arguably the sweet spot for offline AI |
| Multi-step workflows | Extract, then sort, then draft, in sequence | Yes, but every model call adds more waiting time |
| AI agents | AI that plans, acts, checks results and loops | Technically yes. This is where offline hardware starts to struggle, because agentic loops multiply model calls |
Notice that the dividing line isn’t really “offline AI works” versus “offline AI doesn’t work.” It’s this: the more times the system needs to think, the more the hardware matters.
A chat answer is one thought. An agent is a chain of them: plan, act, check, correct, repeat. On a single machine you feel every link in that chain, while cloud services hide the cost by spreading the work across entire data centres. This is why answering questions from a fixed set of documents is a sensible job for one machine, and why a general autonomous agent, for now, is not.
The good news: the job my client needed sits exactly in the sweet spot. Here’s why.
What “knowledge offline” actually means
A common misunderstanding is that an offline AI “knows” your documents the way ChatGPT seems to know everything. It doesn’t, and you wouldn’t want it to.
Instead, well-built offline systems work like a librarian, not an oracle. When a crew member asks a question, the system first searches the document library and pulls out the handful of pages that actually relate to the question. Only then does the AI read those pages and compose an answer, citing where it came from. If the pages don’t contain the answer, the system says so and stops.
This design has a name in the industry (retrieval-augmented generation, if you want to look it up), but the plain version is: look it up first, then answer only from what you found.
Two things follow from this, and they’re the heart of why offline AI works at all.
First, the AI doesn’t need to be an expert in the subject. It needs a much narrower skill: read a few retrieved pages carefully, answer from them accurately, and admit when they don’t contain the answer. That’s a far smaller job than knowing everything, which is what makes it possible on one machine.
Second, refusing to answer is a feature, not a failure. An assistant that answers only from approved manuals, and declines everything else, is exactly what you want near expensive machinery. The system I built will not guess at a fault diagnosis or offer an opinion on safety. It shows what the documents say, then hands the judgement back to a qualified human. A system that pressed on anyway wouldn’t be more helpful. It would be more dangerous.
The catch: smaller models lose detail, then start inventing
Here’s where the offline constraint bites.
An AI model is essentially one very large file: the “brain” produced by training, made up of billions of internal values called parameters. As a rough rule, more parameters means more capacity for nuance, and a bigger file that needs a bigger machine. The models behind the famous cloud services are so large they run across racks of specialised computers. No single machine you can deploy in the field runs anything close to that.
So offline means choosing a smaller model. And smaller models fail in a specific, sneaky way: they don’t get visibly worse at English. They stay perfectly fluent, grammatical and confident, long after they’ve stopped being reliable.
The industry word for this is hallucination: an AI producing a confident, plausible answer that is simply wrong. All AI models do it sometimes. Smaller ones do it more, and in a document-grounded system it shows up in particular ways. The model paraphrases a procedure and quietly drops a step. It blends two similar-looking specifications into one. Or, worst of all, the retrieved pages don’t quite contain the answer, and instead of saying so, the model fills the gap from its general training and presents the result as if it came from the manual.
That last failure is the one that keeps me cautious. A wrong answer that sounds exactly like a right answer, delivered with a citation, is worse than no answer at all. And the ability to resist that temptation, to say “the documents don’t cover this,” turns out to be one of the first things that degrades as models get smaller.
Finding the right model: test, don’t guess
So how small can you go? Here’s the uncomfortable truth: you cannot answer that from a spec sheet, a leaderboard, or a review. Public benchmarks measure general ability. They don’t measure whether a model will faithfully follow your documents and refuse your out-of-scope questions.
The only way to know is to test. I built a suite of real questions with the client, spanning every major category of system the manuals cover. Some questions have clear answers in the documents. Some deliberately don’t, because a model that answers those is failing, however confident it sounds.
Then I proved the system’s behaviour with a highly capable model first, before worrying about hardware at all. That established what “correct” looks like: which questions get answered, which get refused, what a good citation looks like. Only with that baseline in hand does it make sense to ask the real deployment question: what’s the smallest model, on the smallest machine, that still passes the same test?
Guess the model first and you make one of two expensive mistakes. Overspend on hardware nobody needed, or underspend and ship a system that quietly invents answers.
Testing the actual hardware
This is the part most write-ups skip, and it’s where the surprises were.
The machine question comes down to one number: VRAM, the memory on a graphics card. Think of it as the workbench. The whole model has to fit on the workbench to run properly, along with working space for the documents being read. Too small a bench and things get slow, or stop fitting at all.
Models can be compressed to need less space, a technique called quantisation. It’s like reducing the resolution of a photograph: done moderately, you barely notice; done aggressively, the details you cared about vanish. Whether the compressed model still behaves is, again, something you test rather than assume.
Rather than buying machines to find out, I rented them. Cloud GPU services let you hire a specific graphics card by the hour, so for the price of a few hours I ran the same questions, against the same document library, on the same day, three ways: through a cloud AI service, on a rented card with 32GB of VRAM, and on a rented card with 24GB.
Three results, in plain terms.
The bigger card didn’t give better answers. It gave faster ones. The two rented cards, one with a third more memory than the other, agreed on 96% of the questions. Near-identical answers. The difference was speed: the smaller card took roughly five times longer per question, with its worst cases approaching two minutes, because the model barely fit on its workbench and had almost no room to move.
Offline matched the cloud for speed. On the 32GB card, answers came back in about five seconds on average, fractionally faster than the cloud service itself. The belief that local AI is always sluggish isn’t a law of nature. It’s what happens when the model doesn’t comfortably fit.
The remaining differences were about judgement, not power. The cloud service and the local model occasionally disagreed about whether the retrieved pages justified an answer, sometimes while looking at exactly the same pages, and not always in the direction you’d expect. Twice, the local model answered where the cloud declined. For a system whose whole safety story is knowing when not to answer, that borderline judgement is the thing to watch, and it has nothing to do with buying a bigger machine.
What this all adds up to
Six months ago I’d have told you offline AI was mainly a hardware problem: buy a big enough machine and you’re done. I now think that’s exactly backwards.
The sequence that works is this. Understand your documents and what people will actually ask of them. Decide, deliberately, what the system must refuse to do. Prove that behaviour with a capable model. Then, and only then, test how small a model and machine can reproduce it, with evidence rather than a guess behind the purchase.
And if you take nothing else from this post, take the two rules the whole project kept returning to.
How often the system needs to think determines the hardware. One thought per question, and a single machine can match the cloud. Chains of thoughts, looping and self-correcting, and you’re back in data-centre territory, whatever the marketing says.
What the system refuses to answer determines the safety. Not the model size, not the card, not the cloud. The most important line in the whole build isn’t in the hardware budget. It’s the line between what the documents support and what they don’t.
An assistant that thinks once, answers from your own documents, and knows exactly what it’s not allowed to say. On a ship with no internet, and honestly in most places, that’s not a limitation. That’s the point.
