LLM Best Practices

Semantic Audit

3 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Some vault defects are invisible to a script: a title that parses but is a topic label, an atom under the byte threshold that carries two ideas, a pre-decision section rewritten without touching any watched field. These need a reader, and the reader is an LLM judge running a fixed rubric. Run it only after the deterministic pass in vault-audit is clean.

Escalate only the six defects code cannot check

DimensionApplies toFails when
Title as claimatom, claimThe title is a topic label, not an assertion
Atomicityatom, claimThe body holds two separable claims, with a nameable seam
Vapidityatom, synthesisNo load-bearing content: definitional restatement or hedged filler
Hindsight rewritedecisionThe pre-decision prose reads as outcome-aware
Naked link dumpatom, synthesisLinks listed with no sentence explaining each relationship
Source monoculturethe vaultOne or two sources dominate citations on a topic needing corroboration

Anything a regular expression or field comparison can decide stays in the deterministic pass.

Fix the rubric and the output schema

An open “review my vault” prompt gives a different review every run, so nothing is comparable over time. Fix both ends: the six dimensions and a strict output object.

{
  "judged": 40,
  "findings": [
    {
      "file": "20-Atoms/mece-decomposition.md",
      "dimension": "atomicity",
      "verdict": "fail",
      "evidence": "Body pivots at 'moreover, the same tree fails when...'",
      "suggested_action": "Split at the 'moreover' seam into [[claim-a]] and [[claim-b]]",
      "confidence": "high"
    }
  ],
  "clusters": [
    {"theme": "atoms mined from one book, all over-long", "files": ["..."], "action": "refactor session"}
  ]
}

Verdicts are pass, flag, fail, unknown, or na. The schema makes this run comparable to the last, the way prompt-evals makes prompt changes comparable.

Gate every failing verdict on quoted evidence

A fail or flag carries a quote from the note; with no quote, the verdict drops to a softer level. Prefer unknown to a guess. The clearest case is the hindsight rewrite: without the prior text, the judge cannot know a section was edited, and the honest output names the evidence it would need (decision-journals).

Let deterministic flags set the reading order

Do not ask a model to read the vault cold. Assemble a bundle: candidate synthesis-bearing notes ordered by the deterministic flags they already tripped, their bodies and needed frontmatter, and per-source citation counts for the monoculture check. The vault-architect tooling builds this as 99-Tools/semantic-bundle.json with semantic_prep.py --sample 40. A prioritized 40 finds more real defects than skimming 400 unordered, and stays affordable enough to run monthly. Sizing the sample against the context window is the budgeting problem in evaluation.

Cluster findings, then hand the fix to a human

Group by shared cause and propose one batch action per cluster. Splitting an atom, rewriting a title, and diagnosing a monoculture are domain calls: the judge recommends and a person applies. Suggestions must be specific; “split at the ‘moreover’ seam into these two titles” is actionable, “improve atomicity” is not.

Diff runs to catch drift

The fixed rubric exists for the comparison. Store each run and diff: which files cleared, which recurred, which dimension is trending worse. Vapidity findings that climb every month point to a capture problem, not a writing problem. Run it monthly at most; the deterministic auditor runs constantly, and the schedule is set in vault-maintenance.

Backlinks