Running a capable AI model on your own computer used to be a weekend project involving a command line, quantisation formats and at least one dependency error. It now takes about five minutes.
That change is real and worth knowing about. It is also why a lot of people are now running local models for no particular reason, having accepted a meaningfully worse tool in exchange for a benefit they did not need.
The three reasons that genuinely justify it
The material cannot leave your machine
This is the strongest reason by far, and for some people it is not a preference but a rule.
Client work under a confidentiality agreement. Medical or legal notes. Anything covered by a policy that forbids sending data to a third party. In those situations a cloud assistant is not a slightly worse option, it is not an option, and the real comparison is between a local model and no AI at all.
Judged that way, a local model is excellent.
Superwhisper is the clearest example: dictation that runs entirely on a Mac, so nothing you say is transmitted anywhere. For anyone dictating case notes or client material, that distinction is the whole product.
There is no connection
A train, a plane, a site with no signal, a country where the service is blocked. Local models work identically offline, because there is nothing to reach.
The volume makes per-token pricing absurd
If you are processing large amounts of text repeatedly, a model running on hardware you already own has no marginal cost. At small scale this argument is imaginary. At genuine volume it is decisive.
The reason that does not justify it
"I do not want a company training on my conversations."
This is a legitimate concern, but it is now largely addressed by paid tiers and business plans at the major providers, which contractually exclude your data from training. If that is the only thing pushing you local, read the terms of what you already pay for first — you may have already solved it.
Choose local because the data cannot move, not because you would prefer it did not.
What it actually costs
Two things, and both are usually understated.
Hardware. The useful models want a lot of memory. A machine with 8GB will run something, and that something will be noticeably poor. 16GB is workable, 32GB and above is where local starts feeling like a real tool. GPT4All is the exception worth knowing about — it is built to run on an ordinary processor with no graphics card at all, which makes it the realistic option on a standard work laptop, at the cost of speed.
Quality. This is the part people are least honest about. A model small enough to run on your laptop is meaningfully weaker than a frontier model, and the gap is widest exactly where you would notice: long documents, careful reasoning, code across several files. It has narrowed a great deal and it has not closed.
The practical rule: run the largest model your memory will hold, not the smallest one that starts. The gap between local and hosted narrows considerably at the top of what your machine can manage, and most disappointment comes from people judging local AI by a tiny model they picked for speed.
Where to start
Ollama reduced this to a single command, and it is the reason local models became usable for ordinary developers. It also exposes a local endpoint that a great deal of existing software will talk to with only the address changed.
LM Studio is the better starting point if you would rather not touch a terminal. It downloads models for you, gives you a chat window, and can serve them to other applications.
Secret Llama is the curiosity worth trying once: a model running entirely inside a browser tab, with nothing installed. Disconnect from the internet after it loads and it keeps working. As a daily tool it is limited; as a demonstration that the privacy claim is real, it is unusually convincing.
For models themselves, Hugging Face is where the ecosystem lives, Llama is the family with the largest tooling around it, and Qwen is among the strongest at the smaller sizes that actually fit on a normal machine.
Two things worth reading before you ship anything
Open does not mean unrestricted. Llama's licence is a custom community licence with real conditions, not a standard open-source one. Several popular models on Hugging Face are not free for commercial use. Read the licence of the specific version you are using rather than assuming the word open covers it.
Local does not mean private by itself. A local model keeps your text on your machine. It does not stop the application around it from sending telemetry, and it does not make a badly configured server safe.
For code specifically
Tabby is self-hosted and exists for exactly the case where source code cannot leave the network — defence, regulated finance, strict client agreements. Judge it against having no assistant at all, which is the actual alternative in those environments, rather than against the market leaders.
Continue is the more flexible option: it runs in VS Code and JetBrains and lets you point completion at a local model while sending chat somewhere stronger, which is a sensible compromise if your constraint is partial rather than absolute.
The honest summary
Local AI is now genuinely easy, genuinely private, and genuinely worse. If you have a real reason for the second, the third is a fair price. If you do not, you are accepting a worse tool for a benefit you were already getting.
For image generation the same logic applies through Stable Diffusion, which runs offline, costs nothing per image once it does, and is the only realistic route to a model trained on your own material.
If you are earlier than this and still working out which tasks are worth handing over at all, what AI can actually do for you is the better place to start.