LLM Best Practices

Ollama: Model Selection

4 min read

View markdown source

Overview

The Ollama model registry lists hundreds of models; most of them are wrong for your hardware or task. The selection decision has two axes: what fits in memory, and what performs best on the task. The right process is to filter by hardware first, then benchmark on your actual task. Benchmark tables from model releases are a starting point, not the answer.

Filter by VRAM before anything else

A model that does not fit in VRAM spills to CPU RAM, which reduces throughput by 10x or more. Know your memory budget before opening the model registry.

HardwareVRAM / Unified MemoryPractical max model
M2/M3 MacBook Pro 16 GB16 GB unified8B at Q4, or 7B at Q8
M2/M3 MacBook Pro 32 GB32 GB unified27B to 30B at Q4
M3 Max / M4 Max 64 GB64 GB unifiedgpt-oss-120b at MXFP4, or 70B-class at Q4
Single A100 80 GB80 GB VRAMgpt-oss-120b, or Qwen3 235B-A22B MoE at low bit-width
Dual A100 / H100 80 GB160 GB VRAMQwen3 235B-A22B at Q4 with headroom

Add 10 to 20 percent headroom above the model size for the KV cache at your target context length. A 30B-class Q4 model is roughly 18 to 20 GB of weights; a 32K context adds several more GB depending on architecture.

Use Qwen3 for the best open-weight general reasoning

Qwen3 235B-A22B (MoE) tops most open-weight leaderboards for all-around reasoning and coding as of mid-2026, with the dense Qwen3.6-27B close behind at a fraction of the memory. Pick the tier that fits your host.

ollama pull qwen3.6:27b-instruct-q4_K_M
  • Qwen3.6-27B at Q4_K_M is about 17 GB and runs comfortably on a single consumer GPU or 32 GB unified memory.
  • Qwen3 235B-A22B needs multi-GPU or a high-memory Mac (128 GB+ unified); reach for it only when the dense tier misses your eval bar.
  • Q4_K_M is the right quantization for most tasks; Q8_0 if quality is the constraint and memory allows. See ollama-quantization.

Use qwen3-coder:30b for the strongest open-weight code model

qwen3-coder:30b is a 30B mixture-of-experts model with 3.3B active parameters, purpose-built for code generation and review.

ollama pull qwen3-coder:30b-q4_K_M
  • Fits in roughly 19 GB VRAM at Q4_K_M with a 256K context window.
  • Follows structured output instructions reliably, making it a good choice for extraction and refactoring pipelines.
  • Step up to Q8_0 (roughly 32 GB) when quality is the constraint and memory allows.

Use gpt-oss for a single-GPU-slot MoE alternative

gpt-oss-20b and gpt-oss-120b are OpenAI’s open-weight models, released under Apache 2.0 and shipped natively in MXFP4.

ollama pull gpt-oss:20b
ollama pull gpt-oss:120b
  • gpt-oss-20b fits in about 16 GB VRAM; a solid pick for a single consumer GPU or a high-memory laptop.
  • gpt-oss-120b was built for a single 80 GB GPU slot; budget 60 to 96 GB depending on quantization.

Use Gemma 4 or Llama 3.2 8B for low-memory and high-throughput workloads

For laptops under 16 GB, edge deployments, or batch jobs where throughput matters more than quality, smaller models are the correct pick.

ollama pull gemma4:12b-instruct-q4_K_M   # multimodal, 16 GB sweet spot
ollama pull llama3.2:8b-instruct-q4_K_M  # fits in 8 GB
  • Gemma 4’s small variants (E2B/E4B) add on-device image and audio input alongside text, with 140+ language coverage and up to 256K context.
  • Llama 3.2 8B runs in 8 GB of VRAM or unified memory. Use it for classification, short extraction, or high-volume preprocessing where errors are filtered downstream.
  • At 8B, quality drops are real. Compensate with few-shot examples in the prompt (see prompt-design) and tighter output constraints via ollama-modelfile.

For embeddings, use a dedicated embedding model

Generative models (Qwen, gpt-oss, Gemma) produce poor embeddings. Use a dedicated model.

ollama pull nomic-embed-text
ollama pull bge-large       # 1024 dim, strongest open-weight English retrieval

Run the embedding model as a separate Ollama instance or on a separate process to avoid contention with the generative model’s memory. See embeddings for model selection rules and rag for how embeddings plug into a retrieval pipeline.

Benchmark on your own task before committing to a model

Public benchmarks (MMLU, HumanEval, GSM8K) measure general capability. They do not predict performance on your specific task, domain vocabulary, or output format.

  • Build a 50- to 100-question golden set with known correct answers.
  • Run each candidate model on the golden set with the same prompt template.
  • Score correctness, format adherence, and latency separately.
  • Pick the smallest model that meets the quality bar on your golden set.

The cheapest model that passes your eval is the right model. Reserve larger models for subtasks where smaller ones fail.

Backlinks