Keeping AI on your own machine: when it matters and how to do it
Most people meet AI through a website or an app: you type, the text travels to a company's servers, a model answers, and the answer comes back. That is convenient, and for many tasks it is fine. But it means every question you ask, and every document you paste, leaves your computer.
For some work that is not acceptable: a client's contract, a patient's notes, source code that is not public yet, a company's finances. For that work there is another option: run the AI model on hardware you control. This article explains when that is worth it, what it costs, and how to start.
What "local AI" actually means
An AI language model is a large file of numbers (the "weights") plus a program that runs it. Open-weight models publish those files so that anyone can download and run them. When you run one on your own computer or server:
- your prompts and documents never leave the machine;
- it keeps working without an internet connection;
- there is no per-message bill, only the electricity and the hardware;
- nobody can change the model under you without your knowing, because you choose when to update it.
When it is worth it
Local AI makes the most sense when one of these is true:
- The data must not leave. Contracts, health information, unreleased product plans, anything covered by a confidentiality agreement or a regulation.
- The work is repetitive and high volume. Sorting, summarizing or tagging thousands of documents a day can cost more through a paid service than a one-time hardware purchase.
- It has to work offline. A field laptop, a workshop, a site with a poor connection.
- You need the same answer next year. A model you keep does not change; a hosted one may be updated without notice.
It is usually NOT worth it when you need the very strongest reasoning available, or when your volume is small enough that a hosted service costs a few dollars a month.
What it costs
The main cost is memory on the graphics card (GPU). As a rough guide, a model with about 7 to 8 billion parameters, compressed ("quantized") to 4 bits, needs around 5 to 6 GB of GPU memory and runs well on many consumer graphics cards. Larger models need more memory and give better answers. A computer without a suitable GPU can still run small models on its processor, only more slowly.
Plan for:
- Hardware: a graphics card with enough memory for the model size you need.
- Power and heat: a GPU under load draws real power for as long as it is working.
- Your time: installing, updating and checking the setup is ongoing work, not a one-time job.
How to start
- Pick a runner. Free, open-source programs download and run open-weight models with a single command and offer a local address that other programs can talk to.
- Pick a model sized to your card. Start small (7 to 8 billion parameters) and move up only if the answers are not good enough for your task.
- Keep it on your machine. Make the runner listen only on your own computer, not on your whole network, unless you have a reason and a firewall rule for it. An AI server open to the network is open to anyone on that network.
- Test it on your real work. Take twenty real examples, compare the local answers with what you need, and decide with evidence.
- Write down what you run. The model name and version, where it came from, and its license. Some open-weight licenses limit commercial use; read it before you build a product on it.
Two traps to avoid
- Sharing one GPU between many jobs. Two large models loaded at once can run out of memory and slow everything down. Queue the work, or give each job its turn.
- Assuming "local" means "safe." A local model can still be tricked by the text you feed it (for example a document that contains hidden instructions). Treat its output like any other draft: check it before you act on it.
The short version
If your data must stay with you, or your volume is high and steady, running AI on your own hardware gives you privacy, predictable cost and control. Size the model to your hardware, keep it bound to your own machine, test it on real work, and record what you run.