---
title: "Ollama Best Practices"
slug: "ollama"
category: "ai-agents"
tags: ["ai-agents", "ollama", "local-llm", "quantization"]
status: "stable"
last_updated: 2026-08-14
summary: "When to use a local LLM via Ollama, which models and quantizations to pick, and how to wire it into agent workflows."
related: ["[[prompt-engineering/prompt-design]]", "[[ai-agents/embeddings]]", "[[ai-agents/claude-code]]"]
---

## Overview

Ollama runs open-weight LLMs locally with one command. Use it when the workload demands offline operation, data privacy, or a hard cost ceiling on bulk inference. For the production-quality reasoning path, frontier models (Claude, GPT-5.6) remain the default; Ollama is the fallback, the fast path, and the privacy lane.

## Reach for a local model when one of three conditions holds

Pick Ollama deliberately, not by default.

- Offline or air-gapped: no network to the API. Ollama runs entirely on the host.
- Privacy: data cannot leave the machine. Patient records, internal source code under restrictive licenses, customer PII under contract.
- Bulk cost: a million low-stakes classifications. The marginal token is free once the GPU is paid for.

If none of those apply, the frontier API is faster to set up, smarter per token, and cheaper at low volume. Hybrid is common: Ollama for bulk preprocessing, frontier model for the final reasoning step. See [[prompt-engineering/prompt-design]] for prompts that port across both.

## Pick the model by GPU memory, then by task

The model has to fit in VRAM (or unified memory on Apple Silicon) with room for the [[glossary/context-window|context window]]. As of mid-2026, four families cover most use.

- Qwen3.6 (dense, up to 27B) and Qwen3 235B-A22B (MoE): the strongest open-weight general reasoning tier; the 27B dense variant runs on 24 GB at Q4, the 235B MoE needs multi-GPU or a high-memory Mac.
- qwen3-coder:30b: the strongest open-weight code model, a 30B MoE with 3.3B active parameters; fits in 19 GB at Q4 with a 256K context.
- gpt-oss-20b and gpt-oss-120b: OpenAI's open-weight models, natively shipped in MXFP4. The 20B fits 16 GB VRAM; the 120B wants 60 to 80 GB or a multi-GPU host.
- Gemma 4 and Llama 3.2 8B: lower-end machines, edge devices, batch jobs. Gemma 4's small variants add on-device image and audio input; Llama 3.2 8B runs comfortably on 8 GB.

For embeddings, run `nomic-embed-text` or `bge-large` separately (see [[ai-agents/embeddings]]).

## Use Q4_K_M as the default quantization

Quantization trades quality for memory. The standard ladder.

- Q4_K_M: the default. ~4 bits per weight, ~1 to 2 percent quality loss vs. full precision on most benchmarks.
- Q5_K_M: small step up, ~25 percent more memory.
- Q8_0: near full quality, roughly 2x memory of Q4. Use when the host has the headroom and quality matters.
- Q3 or lower: only when memory forces it. Quality drops noticeably.

```bash
ollama pull qwen3.6:27b-instruct-q4_K_M
ollama pull qwen3-coder:30b-q8_0
```

Benchmark on your own eval set. The relative ranking between Q4 and Q8 varies by task.

## Match context length to actual need

A larger context window costs memory linearly and slows the first token.

- Set `num_ctx` to the largest input you actually send, rounded up to a power of two.
- Chat: 4K to 8K is plenty. Long-document RAG: 16K to 32K, with retrieval narrowing the input.

```bash
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
```

Long contexts also degrade attention quality in many open-weight models. Aggressive retrieval (see [[ai-agents/rag]]) beats a giant context window.

## Serve via the REST and OpenAI-compatible endpoints

Ollama exposes two HTTP surfaces on `localhost:11434`.

- Native: `POST /api/generate`, `POST /api/chat`, `POST /api/embeddings`. Streaming by default.
- OpenAI-compatible: `POST /v1/chat/completions`, `POST /v1/embeddings`. Lets any OpenAI client point at Ollama with one env var.

```bash
export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY="ollama"
```

The OpenAI-compatible mode is the fastest way to swap a frontier model into Ollama for local testing of an agent loop.

## Customize behavior with a Modelfile

A `Modelfile` pins a model with a [[ai-agents/system-prompts|system prompt]], parameters, and template into a named local image.

```text
FROM qwen3-coder:30b-q4_K_M
SYSTEM "You are a code reviewer for a TypeScript monorepo. Return JSON only."
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
```

```bash
ollama create code-reviewer -f Modelfile
ollama run code-reviewer
```

Modelfiles are version-controllable. Check them in alongside the agent code so the model and the prompt move together.

## Wire local models in as a fallback or fast path

Two common integration patterns with frontier-model agents like [[ai-agents/claude-code]].

- Fallback: try Claude first, fall back to Qwen3.6 or gpt-oss-120b when the API is down or quota is exhausted.
- Fast path: route low-stakes calls (classification, summarization, extraction) to Ollama; reserve the frontier model for the hard reasoning step.

Log both routes with the same schema so quality comparisons sit side by side.

## Related

- [[prompt-engineering/prompt-design]]
- [[ai-agents/embeddings]]
- [[ai-agents/claude-code]]