Two things push people toward running AI locally. The API bill, and the moment a client asks whether their documents are being sent to an American company.
Both are legitimate. Neither means a local model is a drop-in replacement for the hosted one, and treating it as one is how people conclude local AI is useless after an afternoon.
What actually changes
Running a model locally changes four things at once, and only two of them are in your favour.
| Hosted API | Local model | |
|---|---|---|
| Cost | Per token, scales with use | Free per token, you pay for hardware |
| Data | Sent to the provider | Never leaves your machine |
| Capability | Frontier quality | Meaningfully weaker |
| Speed | Fast, no local hardware needed | Depends entirely on your machine |
The first two columns are why people try it. The second two are why most go back. The useful question is not which is better, it is which jobs fall on which side.
What local models are genuinely good at
- Classification. Sorting messages into categories, tagging tickets, routing enquiries. Narrow, repetitive, and a smaller model handles it perfectly well.
- Extraction. Pulling fields out of invoices, forms and emails into structured data. High volume, well defined, and often the exact work that must not leave the building.
- Confidential documents. Contracts, legal files, medical records, anything under an NDA. Here local is not the cheaper option, it is the only permitted one.
- Drafting at volume. First-pass summaries and rewrites where a human edits afterwards and quality per item matters less than throughput.
What they are still bad at
- Complex multi-step reasoning. The gap between a local model and a frontier one widens sharply as a task needs more chained inference.
- Long context. Large documents need memory you probably do not have, and quality degrades well before the stated context limit.
- Anything a client reads unedited. The quality difference is visible in the output, and your name is on it.
- Code generation of any real difficulty. This is where the gap is widest and most obvious.
What you need to run one
Hardware is the honest gate, and it is worth being blunt about it before anyone spends an evening on this.
- 01RAM is the binding constraint. Eight gigabytes runs only the smallest models and runs them poorly. Sixteen is a workable floor. Thirty-two is comfortable.
- 02A GPU changes the experience more than anything else. Without one you are on CPU, and responses arrive slowly enough to break any interactive use.
- 03Disk. Models are measured in gigabytes each, and you will download several before settling on one.
- 04A runner. Ollama is the usual starting point because it handles downloading and serving in one tool.
- 05An interface. A local chat interface makes the whole thing usable and lets you point at both local and hosted models from one place.
The privacy claim, precisely
This is worth stating carefully, because it is the part most often overclaimed to clients.
A local model running on your hardware genuinely does not send data anywhere. That claim holds exactly as long as everything in the pipeline is local. The moment you connect a hosted API for one step, that step's data goes to that provider like any other request.
So a mixed pipeline is not private just because part of it is local. If you tell a client their data stays in-house, make sure every step in the chain is actually local, and be able to show which.
Where to start
Install a runner, pull one small model, and point a local interface at it. Then take one real task you already do with a hosted API, run it locally, and compare the output honestly.
That single comparison tells you more than any benchmark, because it is your task, your quality bar and your hardware. Most people find one job that local handles fine and several it does not. That is the correct outcome, and it is the map you actually needed.
