LLM Best Practices

Semantic Audit

4 min read

View markdown source

Overview

Some vault defects are invisible to a script. A title can parse fine and still be a topic label. An atom can sit under the byte threshold and still carry two ideas. A decision’s pre-decision section can be rewritten without changing any field the auditor watches. Those need a reader, and the reader is an LLM judge running a fixed rubric. Run it only after the deterministic pass in vault-audit is clean.

Escalate to a judge only for what code cannot check

Six defects justify the cost, and nothing else does:

DimensionApplies toFails when
Title as claimatom, claimThe title is a topic label rather than an assertion
Atomicityatom, claimThe body holds two separable claims, with a nameable seam
Vapidityatom, synthesisNo load-bearing content: definitional restatement or hedged filler
Hindsight rewritedecisionThe pre-decision prose reads as outcome-aware
Naked link dumpatom, synthesisLinks listed with no sentence explaining each relationship
Source monoculturethe vaultOne or two sources dominate citations on a topic needing corroboration

Anything a regular expression or a field comparison can decide stays in the deterministic pass. Sending checkable rules to a model makes them slower, more expensive, and less reliable.

Fix the rubric and the output schema before judging

An unstructured “review my vault” prompt produces a different review every run, so nothing can be compared over time. Fix both ends: the six dimensions above, and a strict output object.

{
  "judged": 40,
  "findings": [
    {
      "file": "20-Atoms/mece-decomposition.md",
      "dimension": "atomicity",
      "verdict": "fail",
      "evidence": "Body pivots at 'moreover, the same tree fails when...'",
      "suggested_action": "Split at the 'moreover' seam into [[claim-a]] and [[claim-b]]",
      "confidence": "high"
    }
  ],
  "clusters": [
    {"theme": "atoms mined from one book, all over-long", "files": ["..."], "action": "refactor session"}
  ]
}

Verdicts are pass, flag, fail, unknown, or na. The schema is what makes this run comparable to the last one, the same way prompt-evals makes prompt changes comparable.

Gate every failing verdict on quoted evidence

A fail or flag must carry a quote from the note. No quote means the verdict drops to a softer level. This single rule is what stops a judge from producing confident-sounding findings it cannot support.

Require unknown over a guess. The clearest case is the hindsight rewrite: unless the judge can see the prior text, it cannot know whether a section was edited, and the honest output says what evidence it would need. Version history answers that question; the note alone does not. See decision-journals.

Let deterministic flags set the reading order

Do not ask a model to read the whole vault cold. Assemble a bundle: candidate notes ordered by the deterministic flags they already tripped, their bodies and the frontmatter the checks need, and the per-source citation counts the monoculture check requires.

Sampling 40 prioritized notes finds more real defects than skimming 400 unordered ones, and it keeps the pass affordable enough to actually run monthly. Sizing the sample against the context window is the same budgeting problem described in evaluation.

Cluster findings by root cause, then propose fixes

Judged findings get the same triage as deterministic ones. Group by shared cause, propose one batch action per cluster, and hand the semantic fixes back to a human: splitting an atom, rewriting a title, and diagnosing a monoculture are all domain calls a judge should recommend rather than apply.

Suggested actions must be specific. “Split at the ‘moreover’ seam into these two titles” is actionable; “improve atomicity” is not.

Diff runs to see whether quality is drifting

The point of a fixed rubric is the comparison. Store each run and diff the findings: which files cleared, which recurred, which dimension is trending worse. A vault where vapidity findings climb every month has a capture problem, not a writing problem, and the trend is the only thing that shows it.

Run this monthly at most. The deterministic auditor runs constantly; the judge is for depth, on a schedule set in vault-maintenance.

Backlinks