---
title: "Ollama: Model Selection"
slug: "ollama-model-selection"
category: "ai-agents"
tags: ["ai-agents", "ollama", "local-llm", "model-selection", "llama", "qwen", "mistral"]
status: "stable"
last_updated: 2026-08-14
summary: "How to pick the right Ollama model for your hardware and task: Qwen3, gpt-oss, and Gemma 4 compared by VRAM, quality, and use case."
related: ["[[ai-agents/ollama]]", "[[ai-agents/ollama-quantization]]", "[[ai-agents/ollama-serving]]", "[[ai-agents/ollama-modelfile]]", "[[ai-agents/embeddings]]", "[[prompt-engineering/prompt-design]]", "[[ai-agents/rag]]"]
---

## Overview

The Ollama model registry lists hundreds of models; most of them are wrong for your hardware or task. The selection decision has two axes: what fits in memory, and what performs best on the task. The right process is to filter by hardware first, then benchmark on your actual task. Benchmark tables from model releases are a starting point, not the answer.

## Filter by VRAM before anything else

A model that does not fit in VRAM spills to CPU RAM, which reduces throughput by 10x or more. Know your memory budget before opening the model registry.

| Hardware | VRAM / Unified Memory | Practical max model |
|---|---|---|
| M2/M3 MacBook Pro 16 GB | 16 GB unified | 8B at Q4, or 7B at Q8 |
| M2/M3 MacBook Pro 32 GB | 32 GB unified | 27B to 30B at Q4 |
| M3 Max / M4 Max 64 GB | 64 GB unified | gpt-oss-120b at MXFP4, or 70B-class at Q4 |
| Single A100 80 GB | 80 GB VRAM | gpt-oss-120b, or Qwen3 235B-A22B MoE at low bit-width |
| Dual A100 / H100 80 GB | 160 GB VRAM | Qwen3 235B-A22B at Q4 with headroom |

Add 10 to 20 percent headroom above the model size for the KV cache at your target context length. A 30B-class Q4 model is roughly 18 to 20 GB of weights; a 32K context adds several more GB depending on architecture.

## Use Qwen3 for the best open-weight general reasoning

Qwen3 235B-A22B (MoE) tops most open-weight leaderboards for all-around reasoning and coding as of mid-2026, with the dense Qwen3.6-27B close behind at a fraction of the memory. Pick the tier that fits your host.

```bash
ollama pull qwen3.6:27b-instruct-q4_K_M
```

- Qwen3.6-27B at Q4_K_M is about 17 GB and runs comfortably on a single consumer GPU or 32 GB unified memory.
- Qwen3 235B-A22B needs multi-GPU or a high-memory Mac (128 GB+ unified); reach for it only when the dense tier misses your eval bar.
- Q4_K_M is the right quantization for most tasks; Q8_0 if quality is the constraint and memory allows. See [[ai-agents/ollama-quantization]].

## Use qwen3-coder:30b for the strongest open-weight code model

`qwen3-coder:30b` is a 30B mixture-of-experts model with 3.3B active parameters, purpose-built for code generation and review.

```bash
ollama pull qwen3-coder:30b-q4_K_M
```

- Fits in roughly 19 GB VRAM at Q4_K_M with a 256K context window.
- Follows structured output instructions reliably, making it a good choice for extraction and refactoring pipelines.
- Step up to Q8_0 (roughly 32 GB) when quality is the constraint and memory allows.

## Use gpt-oss for a single-GPU-slot MoE alternative

`gpt-oss-20b` and `gpt-oss-120b` are OpenAI's open-weight models, released under Apache 2.0 and shipped natively in MXFP4.

```bash
ollama pull gpt-oss:20b
ollama pull gpt-oss:120b
```

- `gpt-oss-20b` fits in about 16 GB VRAM; a solid pick for a single consumer GPU or a high-memory laptop.
- `gpt-oss-120b` was built for a single 80 GB GPU slot; budget 60 to 96 GB depending on quantization.

## Use Gemma 4 or Llama 3.2 8B for low-memory and high-throughput workloads

For laptops under 16 GB, edge deployments, or batch jobs where throughput matters more than quality, smaller models are the correct pick.

```bash
ollama pull gemma4:12b-instruct-q4_K_M   # multimodal, 16 GB sweet spot
ollama pull llama3.2:8b-instruct-q4_K_M  # fits in 8 GB
```

- Gemma 4's small variants (E2B/E4B) add on-device image and audio input alongside text, with 140+ language coverage and up to 256K context.
- Llama 3.2 8B runs in 8 GB of VRAM or unified memory. Use it for classification, short extraction, or high-volume preprocessing where errors are filtered downstream.
- At 8B, quality drops are real. Compensate with few-shot examples in the prompt (see [[prompt-engineering/prompt-design]]) and tighter output constraints via [[ai-agents/ollama-modelfile]].

## For embeddings, use a dedicated embedding model

Generative models (Qwen, gpt-oss, Gemma) produce poor embeddings. Use a dedicated model.

```bash
ollama pull nomic-embed-text
ollama pull bge-large       # 1024 dim, strongest open-weight English retrieval
```

Run the embedding model as a separate Ollama instance or on a separate process to avoid contention with the generative model's memory. See [[ai-agents/embeddings]] for model selection rules and [[ai-agents/rag]] for how embeddings plug into a retrieval pipeline.

## Benchmark on your own task before committing to a model

Public benchmarks (MMLU, HumanEval, GSM8K) measure general capability. They do not predict performance on your specific task, domain vocabulary, or output format.

- Build a 50- to 100-question golden set with known correct answers.
- Run each candidate model on the golden set with the same prompt template.
- Score correctness, format adherence, and latency separately.
- Pick the smallest model that meets the quality bar on your golden set.

The cheapest model that passes your eval is the right model. Reserve larger models for subtasks where smaller ones fail.

## Related

- [[ai-agents/ollama]]
- [[ai-agents/ollama-quantization]]
- [[ai-agents/ollama-serving]]
- [[ai-agents/ollama-modelfile]]
- [[ai-agents/embeddings]]
- [[prompt-engineering/prompt-design]]
- [[ai-agents/rag]]