How to Choose a Local AI Model (Step by Step)
A guide to Phase 1 — Define the Job and Measure Your Hardware, Phase 2 — Read the Specs and Shortlist, Phase 3 — Install, Run and Benchmark, Phase 4 — Decide, Integrate and Maintain
There is no single best local model, only the best fit for your tasks, your memory and your tolerance for trading a little quality for speed. Choosing well is mostly measurement: know your hardware, shortlist a few candidates, then run the same real prompts through each one. This guide takes you through 28 concrete steps, from writing a small test set to pinning the exact version you deployed. By the end you will have a model you can justify with numbers rather than one you picked from a leaderboard. Browse all illustrated guides or the AI Models & Technology category for more illustrated walkthroughs on Howcompass.
How do you choose a Local AI Model (Step by Step)?
Howcompass | How to Choose a Local AI Model (Step by Step)Phase 1 — Define the Job and Measure Your Hardware
HowcompassList the exact tasks it will handle
List the exact tasks it will handle. Name the concrete jobs — summarising notes, question answering, coding, classification or searching documents. Tasks decide the model family and size; a small model that is excellent at classification may be poor at long reasoning. For the primary source behind this phase, see Anthropic — AI research and documentation.
HowcompassPrepare five to ten real test prompts
Prepare five to ten real test prompts. Collect prompts you actually intend to use and keep them as a fixed test set for comparing candidates. Judging models on your own inputs beats any general leaderboard, which rarely reflects your workload.
HowcompassDecide where the model will run
Decide where the model will run. State whether it is a laptop, a desktop with a discrete GPU, a server or a phone. The device sets the ceiling on memory, file format and acceptable model size before you look at any model at all.
HowcompassFix your privacy and offline requirements
Fix your privacy and offline requirements. If source material cannot leave the machine, require fully offline operation with no telemetry and no cloud fallback. Write this down now, because it rules out convenient hosted options later.
HowcompassRead the available memory
Read the available memory. Check usable unified memory or VRAM rather than the advertised total, since the system and other apps reserve a share. Memory capacity and bandwidth, not raw processor speed, usually decide whether a model runs smoothly.
HowcompassCheck CPU, GPU and supported backends
Check CPU, GPU and supported backends. Confirm whether your hardware supports Metal, CUDA, ROCm or Vulkan, because the runtime backend determines how fast inference can be. A model that loads on an unsupported backend may run, but painfully slowly.
HowcompassSet a realistic size budget
Set a realistic size budget. As a rule of thumb, the quantised model file should fit well inside available memory with room left for context and the operating system. Filling every last gigabyte leads to sluggish swapping, not a bigger model for free. Related on Howcompass: make a monthly budget.
Phase 2 — Read the Specs and Shortlist
HowcompassDecode parameter count and quantisation
Decode parameter count and quantisation. Parameter labels such as 7B or 70B indicate capacity and resource hunger, while quantisation like Q4 or Q8 trades precision for a smaller file. Bigger is not automatically better for a narrow task, and a well-chosen quant is often indistinguishable from full precision. For the primary source behind this phase, see NIST — AI Risk Management Framework.
HowcompassMatch quantisation to available memory
Match quantisation to available memory. A Q4 quant uses roughly half a byte per parameter and is the practical default on most consumer machines. If memory allows, Q5 or Q6 can recover a little quality, so spend the budget where it benefits your tests.
HowcompassChoose a file format for your runtime
Choose a file format for your runtime. GGUF suits CPU and Apple-silicon inference through llama.cpp or Ollama, while formats like EXL2 or AWQ target GPU runtimes. The wrong format will fail to load or run far slower than the same weights in the right one.
HowcompassCheck context length and special abilities
Check context length and special abilities. Long documents need a larger context window, and coding, maths or multilingual work have their own benchmarks. Do not pick by parameter count alone when a smaller specialised model may serve you better. Browse all illustrated guides or the AI Models & Technology category for more.
HowcompassRead the licence before downloading
Read the licence before downloading. Check the terms for commercial use, redistribution and limits on derivative models, since weights can carry restrictions that code does not. A model you cannot legally deploy is a poor bargain no matter how well it scores.
HowcompassCheck provenance and maintenance
Check provenance and maintenance. Prefer models with a clear source, official documentation and recent updates over anonymous weight files. A maintained model has a place to report bugs and a path to security fixes.
HowcompassShortlist two or three candidates
Shortlist two or three candidates. Filter by task, memory budget and licence until only two or three remain. A longer list turns selection into endless comparison, when a few well-chosen candidates are enough to decide. Related on Howcompass: protect your privacy when using AI.
Phase 3 — Install, Run and Benchmark
HowcompassInstall a local inference runtime
Install a local inference runtime. Set up a runtime such as llama.cpp, Ollama or a GPU-oriented server that matches your backend, and confirm its command line lists available models. A working runtime gives you a stable base for fair comparisons. For the primary source behind this phase, see Mozilla — web and browser resources.
HowcompassDownload weights from a trusted source
Download weights from a trusted source. Pull the model files from an official repository or a well-known hub and verify the size and checksum before use. A corrupted or tampered file can fail mysteriously or behave in ways you did not intend.
HowcompassLoad the model and confirm it answers
Load the model and confirm it answers. Start with one simple prompt and read the load log for how many layers were placed on the GPU and how much memory was used. If this baseline is unhealthy, no benchmark that follows will be meaningful.
HowcompassTune how many layers go to the GPU
Tune how many layers go to the GPU. If memory headroom remains, increase GPU-offloaded layers in small steps to gain speed, stopping before usage reaches the limit. Pushing one layer too far trades a fast model for constant, slow swapping.
HowcompassRun the same prompt set on every model
Run the same prompt set on every model. Feed the fixed test prompts prepared earlier to each candidate and record the answers side by side. Identical inputs are what make a quality comparison fair rather than a matter of impression.
HowcompassMeasure speed and time to first token
Measure speed and time to first token. Log tokens per second and the delay before the first word appears, since responsiveness shapes everyday use as much as raw answer quality. A slightly weaker model that answers instantly often feels better in practice.
HowcompassSave the hardest real task for last
Save the hardest real task for last. Finish by running your most demanding prompt, such as a long document or a complex code change, on the leading candidate. Models that look strong on short questions can fall apart once the context grows. Related on Howcompass: make a monthly budget.
Phase 4 — Decide, Integrate and Maintain
HowcompassScore candidates on quality, speed and fit
Score candidates on quality, speed and fit. Put quality, speed, memory use and licence into one weighted table, then pick the best total rather than the winner of a single column. This makes the decision defensible to yourself and to anyone who repeats it. For the primary source behind this phase, see Microsoft Learn.
HowcompassSet a default model and a fallback
Set a default model and a fallback. Choose one everyday default and name a second option for demanding tasks or when the first fails. Deciding this once avoids re-litigating the choice every time you open an application.
HowcompassConnect it through a local API
Connect it through a local API. Expose the runtime through a local, OpenAI-compatible endpoint so applications can switch models without rewriting code. Keeping the endpoint on your machine means prompts and keys never need to leave it.
HowcompassPin the exact version and quantisation
Pin the exact version and quantisation. Record the precise model revision and quant tag you deployed, and change only one variable at a time when upgrading. When a later update changes behaviour, a pinned version gives you something reliable to roll back to.
HowcompassWatch memory and thermal headroom
Watch memory and thermal headroom. Monitor memory use and temperature during long sessions, since sustained full load throttles speed and stresses hardware. Leaving a margin keeps responses consistent when a long conversation grows.
HowcompassKeep a small regression prompt set
Keep a small regression prompt set. Re-run your test prompts after every model or runtime update and compare against saved answers. Silent quality regressions are common, and they are invisible unless you check deliberately.
HowcompassRe-evaluate every few months
Re-evaluate every few months. Local models and quantisation methods improve quickly, so revisit the same benchmark on a loose schedule to see whether a newer option is worth switching to. A dated comparison keeps the decision current without chasing every release. Related on Howcompass: protect your privacy when using AI.
Frequently Asked Questions
Question: How much memory do I actually need?
Enough to hold the quantised model file plus room for context and the system, typically leaving at least a few gigabytes spare. Check usable unified memory or VRAM rather than the advertised total, and match the quantisation to that figure.
Question: What is quantisation, in plain terms?
It stores the model's numbers at lower precision, shrinking the file and speeding up inference with a small quality cost. Q4 is a common sweet spot on consumer hardware, while Q5 or Q8 use more memory for finer answers.
Question: Is a local model as good as a cloud model?
Top-tier cloud models still lead on the hardest reasoning tasks, but a well-matched local model handles many everyday jobs well, keeps data private and costs nothing per call. Benchmark on your own tasks to see where the gap actually matters.
Warnings
- Choosing by parameter count alone: A larger model is not automatically better for your workload and will demand more memory and bandwidth; compare candidates on a fixed set of your own prompts instead of a leaderboard.
- Ignoring the licence: Weight files can restrict commercial use, redistribution or derivatives, so read and record the terms before investing time in a model you may not be able to deploy.
- Downloading an oversized quantisation: A model that barely fits in memory will swap constantly and feel unusable; leave headroom for context and the operating system, then raise quantisation only if your tests show a benefit.
Related Howcompass Guides
References (4)
- Anthropic — AI research and documentationanthropic.com
- NIST — AI Risk Management Frameworknist.gov
- Mozilla — web and browser resourcesmozilla.org
- Microsoft Learnlearn.microsoft.com
Sources are listed so you can read the underlying guidance yourself. Our editorial policy explains how guides are reviewed.
Did this article help you?
0 readers found this helpful — add your vote.
No comments yet — be the first to share your results.