My Blog

Caveman mode for Claude Code: fewer output tokens, same facts

Muhammad Ahmad — Tue, 27 Oct 2026 06:49:48 GMT

Claude Code is polite. Ask why a component re-renders and you get a paragraph that restates your question, explains the cause, suggests a fix and offers more help. In a terminal, at the end of a long day, I want the fix and the reason. Nothing else.

Caveman is a Claude Code plugin (and a skill for many other agents) that does exactly that. Its tagline is "why many token when few do trick". Under the joke is a careful set of writing rules. This post covers what it is, how to install it, the modes, what its own repository measures, and the cases where I turn it off.

Everything below comes from the plugin files installed on my machine (version 3.1.0) and the public repository. Where I quote a number, it is the repository’s number, and I say so.

JuliusBrussee/caveman: The caveman repository: the skill, the Claude Code plugin, the optional proxy CLI, and the benchmarks behind its numbers. Apache-2.0.

What it actually changes

The core is one skill file. It tells the model to answer first and then give the reason and the next step, to drop greetings, hedging and recaps, and to use short words. Articles like "a" and "the" are optional. Words that carry meaning are not: the rules say never to drop not, never, no, only or except, and to keep numbers and units exact.

The rule I like most is "payload verbatim". Code blocks, commands, file paths, API names and error messages are copied character for character. Only the prose around them gets shorter. The README shows the same React answer three ways:

One answer, four voices. Example and token counts from the caveman README (tiktoken o200k).

normal reply

The reason your React component is re-rendering is likely because you're creating a new object reference on each render cycle. When you pass an inline object as a prop, React's shallow comparison sees it as a different object every time, which triggers a re-render. I'd recommend using useMemo to memoize the object.

/caveman reply

New object ref each render, so React re-renders. Wrap the prop in `useMemo`.

The facts are the same: a new reference each render, React sees a change, useMemo fixes it. What is gone is the ceremony. The skill also says caveman is "a voice, not broken grammar". There is no "me think" and no caveman prefix, and if the terse version is not actually shorter, plain wins.

Why fewer output tokens matter

The skill works on the second box only.

There are three reasons, and they are not equally strong.

Cost. Output tokens are billed, and on most model price lists they cost several times more per token than input. The skill file puts it simply: every output token is billed and read, so filler costs twice.

Speed. A model writes its answer one token at a time. A shorter answer finishes sooner. In an agent session with many turns, that adds up to less waiting.

Reading. This is the one I feel most. I read every reply in a terminal. An answer that starts with the cause, not with "Great question", saves my attention, which is the scarcer resource.

The important limit: the skill only shortens what the agent says. It does nothing to what the agent reads, which in a coding session is usually much bigger: the system prompt, tool definitions, files, test output and the conversation. The repository’s own Honest Numbers page lists the skill’s input reduction as 0%. For input, the project has a separate, optional proxy CLI, which I cover near the end.

Install it in Claude Code

Add the marketplace and install the plugin

This is the Claude Code route from the README and INSTALL.md. The plugin starts caveman automatically in every session.
```
claude plugin marketplace add JuliusBrussee/caveman && claude plugin install caveman@caveman
```
Or do the same from inside a session

The public README lists the slash-command form too.
```
/plugin marketplace add JuliusBrussee/caveman
/plugin install caveman@caveman
```
Restart and check the flag file

Hooks are read once at session start. INSTALL.md says the active mode is mirrored in a small flag file.
```
cat "${CLAUDE_CONFIG_DIR:-$HOME/.claude}/.caveman-active"
# expected output: caveman
```
Update with the full name

The plugin and its marketplace are both called caveman, so the short name fails with "Plugin caveman not found". Use plugin@marketplace.
```
claude plugin update caveman@caveman
```

If you only want the skill, without hooks, npx skills add JuliusBrussee/caveman -g installs it for Claude Code and other agents. Then type /caveman to start it. One version note: my machine runs 3.1.0, while the public README now pins its install scripts to v4.0.1. The plugin commands above are the same in both.

Modes and levels

Version 3.1 has three modes. Older level names still resolve.

There are three modes. /caveman is the default voice described above. /ultracave (alias /caveman ultra) strips grammar further: fragments, one word when one word is enough, each fact stated once. /megacave (alias /caveman wenyan) answers in Classical Chinese, with technical terms left in their original script; its own skill file says the token savings are small and noisy and must never be claimed.

Older releases talked about levels: lite, full, ultra and wenyan. The 3.1 config code still accepts those names and maps lite and full to caveman, ultra to ultracave, and any wenyan variant to megacave. A mode sticks for the whole session until you change it.

/caveman status reports the current mode and changes nothing.
/caveman off, "stop caveman" or "normal mode" switches it off.
/caveman-help prints the reference card.

You can also change what a new session starts in. The help card lists an environment variable, which wins, and a user config file. Setting the default to manual means sessions start off and /caveman turns it on, which is how I would roll it out to a team that has not asked for it.

terminal

# highest priority
export CAVEMAN_DEFAULT_MODE=ultracave

~/.config/caveman/config.json

{ "defaultMode": "manual" }

How the plugin wires itself in

The plugin is small. Its manifest registers two hooks, and a third script runs when you ask for stats.

Two hooks and a stats script. Each Claude Code window keeps its own mode.

Each window keeps its own mode in a file under .caveman-sessions/ in your Claude config folder, so two sessions can run different modes. /caveman-stats reads the session log and prints the tokens actually recorded. It reports savings as unknown, because a session has no measured twin without caveman to compare against. I like that honesty more than a made-up percentage.

The plugin also ships helpers I use more than I expected: /caveman-commit for one-line Conventional Commit messages, /caveman-review for one-line review findings, and /caveman-compress for shrinking memory files like CLAUDE.md while keeping headings, code blocks and paths. The repository reports about 46% smaller files across five fixtures for that last one.

What the numbers say, and what they do not

The repository’s own eval on claude-opus-5-5. Ten questions, one run, output length only.

The README’s eval asks ten developer questions on claude-opus-5-5. With no instruction, the answers totalled 6,983 output tokens. With "Answer concisely." they totalled 4,334. /caveman gave 4,119 and /ultracave gave 2,693. The repository makes the point itself: new models already know how to be concise, so that one line is the fair baseline. Against it, the Honest Numbers page reports caveman 3% shorter, ultracave 35% and megacave 9% at the median, from ten questions in a single run, measuring length only.

The README also cites outside work: a JetBrains A/B over 86 coding tasks that found 8.5% fewer output tokens and no measurable quality loss, an Adobe Research paper reporting 1.4 to 2.4 times lower cost for caveman-style output across eight models, and an Elastic adaptation with 63.6% fewer response tokens. Those are their numbers, in their setups. The honest summary for plain /caveman on a modern model is: a little shorter than asking for concise answers, and noticeably easier to scan.

When terse output hurts

Caveman has a cost side, and the project documents it better than most tools do.

The rules are extra input on every call. The 3.1 README estimates about 1,000 input tokens per call and the current one about 1,160; the Honest Numbers page says the real cost depends on caching and was not measured. In short coding Q&A, one user measured a net loss (issue #145).
Per-request billing gets nothing. If your plan charges per request rather than per token, as GitHub Copilot premium requests do, a shorter answer costs the same.
Learning needs full sentences. When I am new to a codebase or a library, I want the explanation, not the verdict. Fragments assume you already share the context.
Two readings of one fragment. "Migration drop column backup first" can be read two ways. The skill’s own fix is "Back up first. Then run migration: it drops the column." Ambiguity is worth more tokens than it costs.

Built-in exceptions. After them, it goes back to terse replies.

To its credit, the skill switches to plain prose by itself for the riskiest cases: security warnings, irreversible actions, step sequences that a fragment could scramble, a user who seems confused, and anything written outside the chat, such as code comments, commits, docs and pull requests. Those exceptions are the reason I trust it enough to leave on.

A note on the proxy

The bigger savings the README advertises come from a separate tool: the @caveman-ai/cli proxy, which compresses what the agent reads, such as logs, JSON and test output. It is a different trade-off. INSTALL.md says that routing Claude Code through it sets ANTHROPIC_BASE_URL in your settings, which makes Claude Code Remote Control unavailable while routing is on, and the README says the CLI sends usage statistics by default (caveman telemetry off or DO_NOT_TRACK=1 turns that off). Read both before you install it. The skill alone runs locally and sends nothing.

How I use it

I keep /caveman on for routine coding sessions, switch to /ultracave for quick yes-or-no checks, and say "normal mode" when I am learning something or need a careful explanation for someone else. If you try it, run the same task with and without it and compare your provider’s usage page. That is the test the project itself recommends, and it is the only number that counts for your workload.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Harness engineering: the skill that took over from prompt engineering

Muhammad Ahmad — Mon, 26 Oct 2026 19:19:58 GMT

In 2024 the skill everyone talked about was prompting: find the right words and the model does the right thing. In 2025 it was context: hand the model the right files and docs, and the exact wording matters less. In 2026, when an agent fails, it is rarely the wording or the context that let it down.

What fails now is the layer around the model: which tools it is handed, who checks its work, what it remembers between sessions, what it is allowed to touch, and whether anyone can see afterwards what it actually did. That layer has a name: the harness.

None of these shifts threw the previous one away. Good context work still includes a clear prompt, and harness work still includes choosing the context. The harness adds what wording and context cannot give you on their own: limits on tools, permissions, checks, memory, records, routing between models, and a way for every mistake to become a rule.

Each layer still matters. The outer one is where things break first now.

This post goes through the harness I run for this site and for client work, down to the files, hook scripts and data formats. Everything here is ordinary code and config.

Everything outside the green box is code and config you write.

Under the hood, the harness is a loop

A model on its own cannot run anything. It returns text, and when tools are defined, some of that output is a structured tool_use block: a tool name and a JSON input. Something outside the model has to read that block, decide whether to run it, run it, and send the result back as a tool_result block in the next request. Then the model is called again with the whole conversation. That something is the harness, and every agent, from Claude Code to a support bot, is this loop with different rules around it.

Each box outside the model is a place where the harness can say no.

Inside a product I write this loop myself. Stripped to the parts that matter, it looks like this. Every harness concern from this post shows up as a line: a step budget, a permission check before each call, a timeout, errors returned to the model instead of thrown, and a trace entry for every tool call.

agent.ts (excerpt)

import Anthropic from '@anthropic-ai/sdk'

const client = new Anthropic()
const MAX_STEPS = 8
const TOOL_TIMEOUT_MS = 10_000

export async function runAgent(task: string, tools: ToolSet, trace: Trace) {
  const messages: Anthropic.MessageParam[] = [{ role: 'user', content: task }]

  for (let step = 0; step < MAX_STEPS; step++) {
    const res = await client.messages.create({
      model: 'claude-sonnet-5-5',
      max_tokens: 2048,
      tools: tools.definitions,
      messages,
    })
    messages.push({ role: 'assistant', content: res.content })
    if (res.stop_reason !== 'tool_use') return res // no tool call: the model is done

    const results: Anthropic.ToolResultBlockParam[] = []
    for (const call of res.content) {
      if (call.type !== 'tool_use') continue
      const started = Date.now()
      const decision = tools.check(call.name, call.input) // permissions, before anything runs
      const out = decision.allowed
        ? await withTimeout(tools.run(call.name, call.input), TOOL_TIMEOUT_MS).catch((e) => ({ error: String(e) }))
        : { error: `Not allowed: ${decision.reason}` }
      const failed = typeof out === 'object' && out !== null && 'error' in out
      trace.log({ step, tool: call.name, input: call.input, ms: Date.now() - started, failed })
      // Errors go back to the model as data, so it can try another way instead of crashing the run.
      results.push({ type: 'tool_result', tool_use_id: call.id, content: JSON.stringify(out), is_error: failed })
    }
    messages.push({ role: 'user', content: results })
  }
  throw new Error(`Stopped after ${MAX_STEPS} steps`) // the budget is part of the harness too
}

Two details in there are easy to get wrong. A failed tool should come back as a tool_result with is_error: true, not as an exception, because the model can often recover: pick a different ID, narrow a query, ask the user. And the step budget has to throw, not quietly return, so a runaway loop shows up as a failure in your logs instead of a slow, expensive success. I go further into budgets and checkpoints in agent loops for product features.

Tools: give it less than you think

Every tool definition you send is a capability, and the model will use it whenever it looks helpful. A run_sql(query) tool that accepts any statement is one bad instruction away from an UPDATE with no WHERE. A fetch(url) tool can be steered by any web page the agent reads.

Give the agent the questions, not the database.

So the harness narrows tools in layers. When I gave an agent access to production data, it got an MCP server with a few named queries, each tool annotated as read-only, each query running in a READ ONLY transaction that is always rolled back, as a database role that can only SELECT from a few views and has a five-second statement timeout. Any one of those layers would stop most accidents. Together they mean the model cannot do damage even if it is tricked into trying.

setup.sql (excerpt)

CREATE ROLE mcp_readonly LOGIN PASSWORD 'from-your-secret-store';
ALTER ROLE mcp_readonly SET default_transaction_read_only = on;
ALTER ROLE mcp_readonly SET statement_timeout = '5s';

GRANT USAGE ON SCHEMA support TO mcp_readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA support TO mcp_readonly;

The full server, with redaction of emails and tokens before anything leaves it, is in the read-only MCP server post.

Permissions: rules first, code when rules are not enough

Tools decide what the agent can do. Permissions decide what it may do without asking. In Claude Code the first layer is declarative: allow, ask and deny lists in .claude/settings.json. Deny rules are checked first, then ask, then allow, so a broad allow can never override a specific deny.

Decide once, in a file, not per prompt.

Settings files stack. My user-level ~/.claude/settings.json is permissive on purpose: it allows Bash and the browser tools outright, because I do not want a prompt for every ls. The repo then narrows that with its own .claude/settings.json, committed next to the code. Because deny wins in every scope, a project deny beats my global allow.

.claude/settings.json (the whole file)

{
  "$schema": "https://json.schemastore.org/claude-code-settings.json",
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(./.claude/admin-setup.txt)",
      "Bash(git push --force:*)",
      "Bash(git push -f:*)"
    ],
    "ask": [
      "Bash(vercel --prod:*)",
      "Bash(vercel deploy --prod:*)",
      "Bash(gh pr merge:*)"
    ]
  },
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Read|Edit|Write|Bash",
        "hooks": [
          { "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/guard.sh" }
        ]
      }
    ],
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/verify.sh", "timeout": 120 }
        ]
      }
    ]
  }
}

Pattern rules have limits, though. Read(./.env) stops the Read tool, but not cat .env through Bash, and no pattern can express "only inside this worktree". For that the harness has hooks: commands it runs at fixed points in the loop. A PreToolUse hook receives the pending call as JSON on stdin and can block it.

stdin of a PreToolUse hook (shape)

{
  "session_id": "…",
  "transcript_path": "~/.claude/projects/…/….jsonl",
  "cwd": "/work/app",
  "hook_event_name": "PreToolUse",
  "tool_name": "Edit",
  "tool_input": { "file_path": "/work/app/src/Nav.tsx", "old_string": "…", "new_string": "…" }
}

This is the guard this repo runs, byte for byte. It reads the tool name, path and command with jq and enforces three rules: no secret file enters the context window, whether through the Read tool or a shell reader like cat or grep; no edit lands outside the project, its worktrees, the scratchpad or ~/.claude; and nothing is pushed straight to main, because every change goes through a PR.

.claude/hooks/guard.sh

#!/usr/bin/env bash
# PreToolUse guard: runs before every Read, Edit, Write and Bash call.
# Exit 2 blocks the call; stderr is sent back to the model as the reason.
input=$(cat)
tool=$(jq -r '.tool_name' <<<"$input")
path=$(jq -r '.tool_input.file_path // empty' <<<"$input")
cmd=$(jq -r '.tool_input.command // empty' <<<"$input")
root=${CLAUDE_PROJECT_DIR:-$(jq -r '.cwd' <<<"$input")}

# 1. Secrets never enter the context window (.env.example is fine).
env_file='(^|/)\.env($|\.[A-Za-z0-9_-]+)'
env_read='(^|[;&|(]|[[:space:]])(cat|less|more|head|tail|grep|sed|awk|source|\.)[[:space:]]([^|;&]*[[:space:]/])?\.env($|[[:space:];&|)]|\.[A-Za-z0-9_-]+)'
if { [[ "$path" =~ $env_file ]] && [[ "$path" != *.example ]]; } ||
   { [[ "$cmd" =~ $env_read ]] && [[ "$cmd" != *.env.example* ]]; }; then
  echo "Blocked: .env files hold secrets. Ask the user for the value you need." >&2
  exit 2
fi

# 2. Edits stay inside the project (or its worktrees), the scratchpad, and ~/.claude.
if [[ "$tool" =~ ^(Edit|Write)$ && -n "$path" ]]; then
  case "$path" in
    "$root"/* | /tmp/* | /private/tmp/* | "$HOME"/.claude/*) ;;
    *)
      echo "Blocked: $path is outside this project ($root)." >&2
      exit 2
      ;;
  esac
fi

# 3. Never push straight to main; open a PR from a branch.
if [[ "$cmd" =~ git[[:space:]]+push([^|\;&]*)[[:space:]](origin[[:space:]]+)?(HEAD:)?main([[:space:]]|$) ]]; then
  echo "Blocked: push a branch and open a PR instead of pushing to main." >&2
  exit 2
fi

exit 0

The whole contract between a hook and the agent.

The exit code is the contract. Exit 0 lets the call through. Exit 2 blocks it, and whatever the script wrote to stderr is handed to the model as the reason, which is why the messages are written as instructions ("Ask the user for the value you need"). I keep a file of 21 sample payloads and pipe each one through the script: 10 must exit 2 and 11 must exit 0, including .env.example, a write to the memory folder, and a push to a branch called main-fix.

Those tests exist because the first version was wrong in a way I only saw once it was live. Hook config is read from disk, so the guard started working in the middle of the session that installed it, and it immediately blocked two of my own commands. One was fair: a test command that literally contained cat .env. The other was a PR description that only mentioned .env files, and the regex also matched sed inside the word "used". The fix was to anchor the pattern to a command at the start of a word, followed by an argument that is a path to an env file. The lesson is general: a guard that blocks too much gets switched off, so false positives deserve tests as much as the real catches.

Both scripts are wired in by the hooks block of the settings file above: guard.sh on PreToolUse for Read|Edit|Write|Bash, and verify.sh on Stop with a 120-second timeout.

Each agent also runs in its own git worktree, so even an allowed edit lands on a separate branch that I merge or throw away (how I run agents in parallel).

Instructions and memory: what loads before the first message

A model starts every session with an empty head. What it knows on turn one is whatever the harness puts into the context window first, and the order and size of that load matter more than any single instruction.

A fresh session knows nothing. The harness decides what it knows first.

~/.claude/CLAUDE.md loads for every project: my personal rules, like which model to use for subagents.
A ./CLAUDE.md in the repo, when there is one, loads next: commands, conventions and hard rules for this codebase. A CLAUDE.md inside a subfolder is pulled in only when the agent works with files there, so module-specific rules cost nothing elsewhere.
Skills are folders with a SKILL.md. Only the name and description from each file's frontmatter sit in context; the body loads when a task matches. That is how a session can have dozens of skills available without paying for all of them.
Memory is a folder of small notes plus a MEMORY.md index. The index loads at the start; a note is read only when its one-line description looks relevant.

~/.claude/scheduled-tasks/daily-blog-crosspost/SKILL.md (frontmatter)

---
name: daily-blog-crosspost
description: Daily 10 AM: cross-post one not-yet-posted blog post from mahmadmanzoor.com to dev.to and Hashnode with canonical URL back to the site, verify, and report with screenshots.
---
(the steps below this line load only when the skill is used)

The practical rule I keep relearning: instructions only work if they are short and checkable. "Write clean code" changes nothing. "Client work is described with screenshots only, never a repo link" changes behaviour on the first try, and I can grep for violations. More in agent instructions that actually change behaviour.

Checks: done is a claim until something proves it

This is the part I would build first if I were starting again. An agent saying "everything works now" is reporting intent. The harness should make it produce evidence: typecheck, lint, build, a bundle budget, and screenshots for anything visual.

The fast checks run before the agent is allowed to stop.

Mechanically this is the same hook contract as the guard, attached to a different point in the loop. When the model returns no tool_use and wants to end its turn, the Stop hook runs. If any TypeScript or JavaScript file changed, it runs tsc -b and oxlint. If either fails, the hook exits 2, the errors become the next message, and the loop continues.

.claude/hooks/verify.sh

#!/usr/bin/env bash
# Stop hook: runs when the agent says it is done.
# If source files changed and typecheck or lint fails, exit 2 keeps the agent
# working and sends the errors back to it.
input=$(cat)
cd "${CLAUDE_PROJECT_DIR:-$(jq -r '.cwd' <<<"$input")}" || exit 0

# Second stop in a row after a block: let it end, so a fix it cannot make
# does not loop forever. The failure is still printed to the user.
active=$(jq -r '.stop_hook_active // false' <<<"$input")

# Only check when code changed (staged, unstaged or untracked).
changed=$( { git diff --name-only HEAD; git ls-files --others --exclude-standard; } 2>/dev/null |
  grep -E '\.(ts|tsx|js|mjs)$' )
[[ -z "$changed" ]] && exit 0

out=$( { npx --no-install tsc -b && npx --no-install oxlint; } 2>&1 )
if [[ $? -ne 0 ]]; then
  if [[ "$active" == "true" ]]; then
    echo "verify.sh: checks still failing after one retry, stopping anyway." >&2
    exit 0
  fi
  echo "Checks failed. Fix these before you finish:" >&2
  tail -n 40 <<<"$out" >&2
  exit 2
fi
exit 0

Three details matter. It only runs when code changed, so a turn that edits a blog draft costs nothing, and a turn that touches code costs about 3.5 seconds. The stop_hook_active flag in the input is true when the agent is already continuing because of this hook, so the script blocks at most once and does not trap the agent on an error it cannot fix. And it is only the fast checks: the build, the bundle budget and anything slow stay in CI, which runs on every PR (make coding agents prove their work). I tested it by planting const n: number = "oops": exit 2 with the TS2322 error, then exit 0 on the second stop. For LLM features inside a product, the same idea is an eval set that runs before anyone touches the prompt (evals before prompts).

Routing: the right model for each job

Routing works because of how subagents are built. A subagent gets a fresh context window, does its job, and only its final report comes back to the main session. So the main session can hand a 40-file search to a small model and receive ten lines, instead of filling its own context with the search. The model is a parameter on that call, which makes routing a one-line decision per task.

Cheapest model that can do the job.

~/.claude/CLAUDE.md (excerpt)

When work goes to subagents, always set the model by how hard the task is:
- haiku: copy-only edits, mechanical changes, screenshots, data gathering.
- sonnet: normal build and fix work (UI, small logic, tests).
- opus: complex work only: architecture, security, hard merges, deep reviews.

The trade-off is that the main session only sees a summary, so a subagent's mistake can hide behind a confident report. That is why the main session keeps review and merging, and why every report has a "not checked" section. More in matching the model to the task.

Records: you will want to know what happened

When something goes wrong a day later, "the agent did it" is not an answer. Claude Code writes every session to disk as JSON Lines at ~/.claude/projects//.jsonl: one object per line, with the assistant's tool_use blocks and the tool_result that came back. That makes a session queryable with jq.

reading a session transcript

# every shell command the agent ran, in order
jq -r 'select(.type=="assistant") | .message.content[]?
       | select(.type=="tool_use" and .name=="Bash") | .input.command' session.jsonl

# which tools it used, and how often
jq -r 'select(.type=="assistant") | .message.content[]?
       | select(.type=="tool_use") | .name' session.jsonl | sort | uniq -c | sort -rn

On the long session that produced this post and the work around it, the second query reports 811 Bash calls, 255 browser batches, 89 file reads and 37 tool-schema lookups across 17,344 transcript lines. Numbers like that answer real questions: which commands the agent ran against production, when it first touched a file, how much of the session was browsing versus editing.

A run you can read back step by step.

Inside a product, the trace.log line in the loop above is the same idea: tool name, input, duration and whether it failed, per step, per request. Send it to whatever you already use for traces. You cannot fix a loop you cannot see.

Feedback: fix the harness, not just the output

When an agent makes a mistake, correcting that one output is the smallest possible fix. The better fix changes the harness so the next run cannot repeat it quietly, and each layer above gives a concrete place to put it: a line in CLAUDE.md, a deny rule, a branch in the guard script, a check in the Stop hook, or a new case in the eval set.

Every mistake should leave the harness a little stricter.

Over a few weeks this compounds. The harness for this site started as one instructions file. It now has naming rules for client work, a bundle budget, a guard hook, a Stop hook, model routing, a memory folder and a handful of skills, and nearly every one of those exists because something went wrong once.

Where to start

Write down the hard rules

A short CLAUDE.md: commands, conventions, and the few things that must never happen.
Make done provable

A Stop hook running typecheck and lint, slower checks in CI, evals for LLM features.
Cut the tools, then set permissions

Narrow, read-only tools by default; allow, ask and deny rules; a PreToolUse guard for what patterns cannot express.
Route by difficulty

Subagents with small models for mechanical work, the large one for decisions.
Keep the records and use them

Query transcripts and traces when something looks off, and turn each mistake into a rule.

None of this makes the model smarter. It makes the system around it predictable, which in my experience is what decides whether an agent is useful at work or just impressive in a demo.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Setting up REA for agent-driven reverse engineering in Claude Code

Muhammad Ahmad — Mon, 26 Oct 2026 07:22:50 GMT

Sometimes the answer is not in a source repository. It is in a build: the Electron app that went out last month, an APK on a test phone, a native binary from an old release with no tag. Questions like "did the source maps ship?" or "what changed between these two builds?" need someone to open the artifact itself.

REA, short for "reverse engineer anything", is an MCP server and CLI that gives a coding agent the tools for that job. Its README describes it as reverse engineering with agents, "from app behavior down to native binaries", and it runs locally. I have it registered in Claude Code as an MCP server named rea, alongside its skill, reverse-engineer-anything. This post covers the setup, what it can analyse, the workflows I use on software I own, and the rules I keep.

morluto/rea: REA source: MCP server and CLI for agent-driven reverse engineering. MIT licence for REA-authored code.

Two pieces: the server and the skill

Everything runs on your machine. The heavy engines are yours to install.

REA comes in two parts, and the skill file is explicit that one does not imply the other. The MCP server, published on npm as rea-agents, provides the tools. The skill is a set of instructions that teaches the agent when to use REA, which tool to call first, and how to report results. Installing the skill does not register the server, and registering the server does not install analysis engines.

The skill also says when not to use REA: for ordinary analysis of a complete source repository, use normal repository tools. That matches my experience. REA is for shipped artifacts and runtime behaviour, not for reading code you already have.

How do I install REA in Claude Code?

Use the official sources only. The README warns that a similar repository name or a copied README does not make a download official, and lists three: the morluto/rea repository, rea.tools and the rea-agents npm package. You need Node.js 22.x (22.23.4 or later), 24.x (24.22.1 or later) or 26.12.1 and later.

Preview the setup for Claude Code

The dry run prints the paths it would change, the backups and the skill it would install, without writing anything.
```
npx -y rea-agents@latest setup --client claude_code --dry-run
```
Apply it

Setup registers the MCP server and installs the skill for the client you name. Run the plain npx rea-agents setup to pick clients interactively.
```
npx rea-agents setup --client claude_code
```
Check readiness

Doctor reports registration and skill state as JSON. It cannot prove the running agent is connected, so restart Claude Code after setup.
```
npx rea-agents doctor --client claude_code --skill --json
```
Or register it by hand, pinned

The installation guide gives a manual command with a pinned version. Pinning means an upgrade is a decision, not a surprise.
```
claude mcp add --scope local rea -- npx -y rea-agents@6.7.0 mcp
```

My own registration is the manual form, pinned to the version I installed, with one environment variable that points REA at my Hopper install. Your path will differ. Other engines use their own variables, such as GHIDRA_INSTALL_DIR and JAVA_HOME.

~/.claude.json (the rea entry)

"rea": {
  "command": "npx",
  "args": ["-y", "rea-agents@6.4.0", "mcp"],
  "env": {
    "HOPPER_LAUNCHER_PATH": "/Users/you/Applications/Hopper Disassembler.app/Contents/MacOS/hopper"
  }
}

Engines are bring-your-own. Native decompilation needs Hopper, Ghidra or IDA. Android analysis needs a headless JADX MCP JAR (version 0.7.1, selected with REA_JADX_MCP_JAR) and Java 17 or newer. Firmware extraction uses Binwalk or Unblob on Linux. Static JavaScript and .NET inspection need no engine at all, so those are good first targets. REA setup does not install unrelated software, and commercial tools like IDA and Hopper must be licensed by you.

REA: Reverse Engineering for Your Coding Agent: The REA homepage: what it does and the one-line setup command.

What can REA analyse?

The skill’s first rule is to route by target: look at what the user supplied, and pick the first tool from that. This is the map I keep in my head.

The first call depends on the artifact. Everything after it follows the evidence.

Native binaries. open_binary binds a target, then focused tools search strings, list procedures, decompile, and follow callers and callees. For Mach-O, observe_native_calls can record which functions run, but it launches the target under LLDB, so it really executes.
Electron and JavaScript. analyze_javascript_application reads an ASAR or an extracted tree, including source maps, and builds a module graph.
Android. inspect_android_package reads the manifest and package identity; class search, method decompilation and reference tracing follow. The APK is never launched.
Managed code and firmware. inspect_managed_artifact inspects .NET assemblies without executing them. inspect_firmware_regions and extract_firmware handle images, with extraction written to a new folder you choose.
Web pages and running apps. list_browser_targets and list_electron_targets attach to pages you already have open, for passive observation.

Evidence, not vibes

What I like most about REA is how it reports. Every tool returns an Evidence record with an ID, and the skill tells the agent to separate what was observed from what was inferred and what is still unknown. A decompiled function is an observation of analysis output, not original source. A static result never proves that code ran.

Every claim in the final answer should point back at an Evidence ID.

For broad questions the skill asks for a finding ledger: each conclusion linked to its Evidence IDs, the evidence type, the search boundary and what remains unknown. There are tools for the gaps too: record_unknown tracks an open question, and comparison tools like compare_artifacts, compare_functions and find_changed_behavior treat static differences as candidates, not proof.

Workflow 1: what did my Electron build ship?

Large bundles are read in pages from one retained analysis.

This is the workflow I use most. Point the agent at the packaged ASAR of an app you built and ask a narrow question, such as whether source maps or a debug flag made it into production. For a big app, the skill says to request detail: "summary" first, then use the returned parent_evidence_id with inspect_analysis_view to page through modules and read one item at a time, without running the analysis again.

prompt

Use REA on /Applications/MyApp.app/Contents/Resources/app.asar.
Start with a summary. Did any source maps or .env files ship in this build?
List each finding with its Evidence ID and say what you could not check.

The same works from the shell when I want the full record on disk instead of in the agent’s context. These commands come from the skill’s recovery guide:

terminal

rea analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
rea inspect-analysis-view app-view.json

Workflow 2: compare two releases

When a bug appears between two versions and the tags are missing, I give the agent both artifacts and ask what changed around one feature. For native code, it opens each binary, finds the function by strings or names and compares the function evidence. For JavaScript, it compares module graphs. The answer is a list of candidate changes with evidence, which I then confirm by running both builds. The tool narrows the search; it does not replace the test.

Workflow 3: watch a page I own

Passive by design. You drive the app; REA records what it can see.

Runtime observation is passive. You start the browser or Electron app with a debugging port on loopback, then pass that endpoint in the request. REA lists eligible pages and captures what you ask for, such as scripts, network shape and console output. It never clicks, navigates or runs page JavaScript, and it deliberately leaves out credentials, cookies, authorization headers, storage values and raw WebSocket or JSON bodies. When I have static evidence from the bundle and runtime evidence from the page, reconcile_javascript_runtime matches them up, and it reports mismatches instead of guessing.

terminal

# a separate profile, debugging port on loopback only
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 --user-data-dir="$HOME/.rea-chrome"

Where are you allowed to use it?

Scope comes before the first tool call.

Reverse engineering tools are dual-use, and an agent makes them faster, which raises the stakes. The REA README says it is for lawful research, education, interoperability, software preservation and analysis of software you own or are authorised to examine, and that cracked copies, stolen keys, unauthorised key sharing and licence bypasses are not authorised. I keep a few rules of my own on top:

Only analyse software I built or have written permission to test. A client saying "sure" in a call is not written permission.
Read the licence. Some end-user licences forbid reverse engineering even for software you paid for, and the law differs by country.
Keep runtime work on a test machine or profile. Native call observation runs the target for real.
Never paste secrets you find into chat, tickets or posts. If a build leaks a key, rotate it and report it to the owner.
Pin the version and install from the official sources only.

Inside those lines, REA is the most useful agent tool I have added this year for questions about builds rather than source. It turns "I think the old version did this" into a list of claims with evidence, which is the kind of answer I can act on.

Frequently asked questions

What is REA?

REA, short for "reverse engineer anything", is a local MCP server and CLI that gives a coding agent reverse engineering tools for native binaries, Electron and JavaScript bundles, Android packages, .NET assemblies, firmware and web pages. It is published on npm as rea-agents.

Does REA install Ghidra, Hopper or JADX for me?

No. Engines are bring-your-own. Native decompilation needs Hopper, Ghidra or IDA, Android analysis needs a headless JADX MCP JAR and Java 17 or newer, and firmware extraction uses Binwalk or Unblob on Linux. Static JavaScript and .NET inspection need no engine.

Does REA run the software it analyses?

Mostly not. The static tools read the artifact, and an APK is never launched. The exception is observe_native_calls for Mach-O, which launches the target under LLDB, so it really executes. Runtime observation of pages is passive.

When should I not use REA?

For ordinary analysis of a complete source repository, where normal repository tools are the right choice. And never on software you do not own or are not authorised in writing to examine.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Structured outputs that do not break: schemas, Zod and a repair retry

Muhammad Ahmad — Sun, 25 Oct 2026 19:24:13 GMT

A few years ago the hard part of getting JSON from a language model was getting JSON at all. You asked nicely, you got a code fence with a trailing comma, and you wrote a regex to dig the object out. That problem is mostly solved. Current APIs can constrain the output to a JSON schema, so the answer always parses and always has the fields you asked for.

The problem that is left is quieter. When I worked on extracting roof measurements from PDF reports, the worst failures were never broken JSON. They were valid objects with the wrong numbers in them: a roof in squares read as square feet, a total counted twice. The schema was happy every time.

So I think of structured output as layers. Each one catches a different kind of failure, and the last one is a person.

Each layer catches what the one before it cannot.

The examples use the Anthropic TypeScript SDK and Zod. The teaching example is a small extractor that reads the text of a roof report and returns its measurements.

Layer one: let the API enforce the shape

There are two ways to get structured data out of Claude, and I pick between them by asking what the object is for.

Answer or action decides the mechanism.

If the object is the answer, use output_config.format and let the SDK parse it. The model has one job and the response is the data. If the object is the argument to an action, and the model might choose between several actions or none, define a tool with strict: true. The arguments are then guaranteed to match the tool's schema, and you still get to decide whether to run it.

For an extractor the object is the answer. I define the shape in Zod, keeping this schema to plain types only, because it is turned into a JSON schema for the API.

extract/schema.ts

import { z } from "zod";

export const Facet = z.object({
  pitch: z.string().describe('rise over a 12 inch run, written like "6/12"'),
  areaSqFt: z.number(),
});

export const ReportShape = z.object({
  sourceUnit: z.enum(["sq_ft", "squares"]).describe("the unit the report uses for area"),
  totalAreaSqFt: z.number().nullable().describe("converted to square feet; null if the report has no total"),
  facets: z.array(Facet),
  ridgeFt: z.number().nullable(),
  valleyFt: z.number().nullable(),
});
export type Report = z.infer;

The describe() strings end up in the schema the model sees. They are the cheapest prompt engineering there is: one line, next to the field it explains.

Layer two: why is a valid schema not enough?

A schema cannot say "the facet areas add up to the total" or "this number must appear somewhere in the report". Zod can, with superRefine. I build the checks as a function of the source text, so a check can look back at what the model was reading.

Valid JSON, wrong roof.

The check I would keep if I could keep only one is the grounding check. Every number the model claims to have read should be findable in the source, either as written or in the other unit. It catches invented values, and it catches the classic unit slip: the report says 32.4 squares, the model correctly says the unit is squares, then forgets to convert and returns a total of 32.4. Converted back, that is 0.324 squares, which is nowhere in the text, so the check fails.

extract/checks.ts

import { ReportShape } from "./schema";

const numbersIn = (text: string) =>
  new Set((text.replace(/(\d),(\d{3})/g, "$1$2").match(/\d+(?:\.\d+)?/g) ?? []).map(Number));

export function reportChecks(source: string) {
  const seen = numbersIn(source);
  // totals are stored in square feet; a report in squares shows them divided by 100
  const grounded = (n: number, unit: "sq_ft" | "squares") =>
    unit === "squares" ? seen.has(Math.round(n) / 100) : seen.has(n);

  return ReportShape.superRefine((r, ctx) => {
    if (r.totalAreaSqFt !== null && !grounded(r.totalAreaSqFt, r.sourceUnit))
      ctx.addIssue({ code: "custom", path: ["totalAreaSqFt"], message: `${r.totalAreaSqFt} does not appear in the report (unit: ${r.sourceUnit})` });

    if (r.totalAreaSqFt !== null && r.facets.length > 0) {
      const sum = r.facets.reduce((s, f) => s + f.areaSqFt, 0);
      if (Math.abs(sum - r.totalAreaSqFt) > r.totalAreaSqFt * 0.02)
        ctx.addIssue({ code: "custom", path: ["facets"], message: `facet areas add up to ${sum}, but totalAreaSqFt is ${r.totalAreaSqFt}` });
    }

    r.facets.forEach((f, i) => {
      if (!/^\d{1,2}\/12$/.test(f.pitch))
        ctx.addIssue({ code: "custom", path: ["facets", i, "pitch"], message: `pitch "${f.pitch}" is not written as N/12` });
    });
  });
}

Notice the error messages. They are written for the model as much as for me. Each one says which field is wrong, what value was there and what the rule is. That pays off in the next layer.

Why check the stop reason before validating?

There are three ways to get a response that should not reach your checks at all. A stop_reason of max_tokens means the output was cut off. A refusal means a safety classifier declined the request. And parsed_output can be null. None of these throw an exception. They are successful HTTP responses with nothing useful inside, so the code has to look for them.

Check the stop reason before you read the content.

A truncated extraction is not worth repairing. Either the output budget is too small for the document, which is a bug to fix, or the model went off on a long tangent. Neither gets better by asking again with the same settings.

Layer three: why only one repair retry?

When validation fails, retrying the same request blind mostly reproduces the same mistake. Telling the model exactly what failed works much better. The repair request contains the source, the previous answer and the list of failed checks, and asks for the full corrected object.

One retry with specific errors, then stop.

I build each attempt as a fresh single request rather than a growing conversation. The source text is the first block and carries a cache breakpoint, so the second attempt reads it from the prompt cache instead of paying for it again.

extract/extract.ts

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { ReportShape, type Report } from "./schema";
import { reportChecks } from "./checks";

const client = new Anthropic();
const SYSTEM = "You extract roof measurements from report text. Use only values that appear in the report. If a value is missing, return null instead of estimating it.";

export type Extraction =
  | { ok: true; data: Report; attempts: number }
  | { ok: false; reason: "refusal" | "truncated" | "empty" | "invalid"; issues: string[]; raw: unknown };

export async function extractReport(source: string): Promise {
  let previous: unknown = null;
  let issues: string[] = [];

  for (let attempt = 1; attempt <= 2; attempt++) {
    const ask = previous === null
      ? "Extract the measurements from the report above."
      : `Your previous answer was:\n${JSON.stringify(previous)}\n\nIt failed these checks:\n${issues.map((i) => `- ${i}`).join("\n")}\n\nRe-read the report and return the full corrected object.`;

    const res = await client.messages.parse({
      model: "claude-sonnet-5-5",
      max_tokens: 16000,
      system: SYSTEM,
      messages: [{
        role: "user",
        content: [
          { type: "text", text: `\n${source}\n`, cache_control: { type: "ephemeral" } },
          { type: "text", text: ask },
        ],
      }],
      output_config: { format: zodOutputFormat(ReportShape) },
    });

    if (res.stop_reason === "refusal") return { ok: false, reason: "refusal", issues: [], raw: null };
    if (res.stop_reason === "max_tokens") return { ok: false, reason: "truncated", issues: [], raw: null };
    if (!res.parsed_output) return { ok: false, reason: "empty", issues: [], raw: null };

    const checked = reportChecks(source).safeParse(res.parsed_output);
    if (checked.success) return { ok: true, data: checked.data, attempts: attempt };

    previous = res.parsed_output;
    issues = checked.error.issues.map((i) => `${i.path.join(".") || "(root)"}: ${i.message}`);
  }
  return { ok: false, reason: "invalid", issues, raw: previous };
}

Why only one retry? Because the second attempt is where the information is. If the model fixes the problem when told exactly what is wrong, the first answer was a slip. If it fails the same check again with the error in front of it, the input is unusual, the report is ambiguous, or the check is wrong. A third attempt rarely changes that, and each one costs a full request. Two failed attempts on the same field is a signal, and the right response to that signal is a person.

I log every repair, including the ones that succeed. A check that triggers a repair on a large share of documents means the prompt or the field description is wrong, and that belongs in the eval set, not in a retry loop.

Layer four: a human, on purpose

The human fallback is not an error path. It is part of the design, and it deserves a real screen. These are the cases I send to review:

the repair retry failed, with the issue list attached
a refusal or a truncated response
values that pass every check but sit outside business limits, like a house with 40,000 square feet of roof
two sources that disagree, such as a total in the summary table that differs from the detail table
any field where a wrong value costs money and the model was not reading it straight off the page

The reviewer sees the source, the output and the reason in one place.

The reviewer sees the relevant part of the source, the extracted fields, and the exact check that failed. They can accept, correct or reject. Every correction is saved with the input, and that pair goes straight into the golden set, which I wrote about in the post on evals. Over time the cases that reach a human become the cases the harness tests most carefully.

The short version

Pick the mechanism by purpose

Use output_config.format when the object is the answer, and a strict tool when the object is the argument of an action.
Keep the API schema plain, put meaning in Zod

Describe fields in the schema and check sums, formats and grounding with superRefine.
Check the stop reason before the content

Truncation, refusal and an empty parse are successful responses that must not reach your checks.
Repair once, with specific errors

Send the source, the previous answer and the failed checks. Cache the source so the retry is cheap.
Send the rest to a person and learn from it

Show the source next to the output, record the correction, and add it to the eval set.

Schemas made JSON boring, which is good. The interesting work moved one layer up, to deciding what "correct" means for your data and writing it down as code the model can read when it gets it wrong.

Frequently asked questions

Does a JSON schema guarantee correct LLM output?

No. Schema-constrained output guarantees the shape: the JSON parses and has the fields you asked for. It cannot check meaning, such as whether the facet areas add up to the total or whether a number appears in the source. Check that in Zod with superRefine.

Should I use output_config.format or a strict tool?

Use output_config.format when the object is the answer. Use a tool with strict: true when the object is the argument of an action the model chooses, possibly among several actions or none.

How many times should I retry a failed extraction?

Once, with the errors. The repair request carries the source, the previous answer and the list of failed checks. If the model fails the same check again with the error in front of it, send the case to a person.

Which cases should go to human review?

Failed repairs, refusals and truncated responses, values outside business limits, sources that disagree with each other, and any costly field the model did not read straight off the page.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Evals before prompts: a small eval harness for an LLM feature

Muhammad Ahmad — Sun, 25 Oct 2026 18:51:38 GMT

Every LLM feature I have built started the same way: a prompt that worked on the three examples I had open. On the roof report extraction work, the examples I had open turned out to be the easy ones. Real uploads failed in ways I had not pictured, and the only thing that stopped me from fixing one failure and quietly causing another was turning each failure into a test case.

That is all an eval harness is. A list of inputs you care about, a way to decide whether each output is good, and a number at the end. I now build it before I write the second version of any prompt, because without it every prompt edit is a guess that happens to look right on the case in front of you.

The whole harness is one loop. Change, run, compare, decide.

The harness below is small on purpose: a JSONL file, two kinds of checks, a script and a CI job. The examples use the Anthropic TypeScript SDK and a made-up feature, but nothing about the shape is tied to one provider.

The feature under test

To keep it concrete, picture an inbox helper for a roofing contractor. An email comes in, and the feature returns three things: a category (quote request, storm damage, billing, scheduling or spam), an urgency, and a short draft reply that a person reviews before sending.

Two of those outputs have one right answer. The third is free text where many answers are fine and a few are dangerous. That mix is typical, and it is why the harness needs two kinds of checks.

What goes in the golden set?

The golden set is a file of cases. Each case has the input, the fields that have one right answer, and a few notes in plain English about what a good reply must and must not do. I add tags so I can slice the score later, and I mark a handful of cases critical: the ones where a wrong answer costs real money or trust.

One line of JSONL per case.

Where do cases come from? Real inputs first, with names and addresses replaced. Then every bug report, word for word. Then the edge cases you can think of: an empty body, a forwarded thread, an email in two languages, a polite message that is actually a complaint. Thirty to fifty cases is enough to start. The set grows every time the feature fails in production, and that habit matters more than the starting size.

I keep the notes short and checkable. "Must not promise a visit date" is a good note. "Should sound professional" is not, because nobody can grade it the same way twice.

evals/golden.jsonl

{"id":"storm-017","tags":["storm","critical"],"input":{"from":"homeowner@example.com","subject":"Shingles in the yard","body":"Wind last night took a bunch of shingles off the back slope. Water is coming through the bedroom ceiling. Can someone come today?"},"expect":{"category":"storm_damage","urgency":"high"},"notes":"Must acknowledge the leak. Must not promise a time or a price. Should ask for a phone number if none is given."}
{"id":"billing-004","tags":["billing"],"input":{"from":"ap@example.org","subject":"Invoice 2291","body":"We were charged twice for the gutter work. Please advise."},"expect":{"category":"billing","urgency":"normal"},"notes":"Must not admit or deny the double charge. Should say the office will check the invoice."}

evals/types.ts

import { z } from "zod";

export const Case = z.object({
  id: z.string(),
  tags: z.array(z.string()),
  input: z.object({ from: z.string(), subject: z.string(), body: z.string() }),
  expect: z.object({
    category: z.enum(["quote_request", "storm_damage", "billing", "scheduling", "spam"]),
    urgency: z.enum(["low", "normal", "high"]),
  }),
  notes: z.string(),
});
export type Case = z.infer;

Exact checks first

Anything that can be checked with code should be checked with code. It is free, it is instant and it never disagrees with itself. For this feature that means the two enum fields, plus a few rules about the reply that a regex can enforce: no dollar amounts, no links, a length cap.

evals/checks.ts

import type { Case } from "./types";
import type { TriageOutput } from "../src/triage";

export function exactChecks(c: Case, out: TriageOutput) {
  return [
    { name: "category", pass: out.category === c.expect.category },
    { name: "urgency", pass: out.urgency === c.expect.urgency },
    { name: "no_prices", pass: !/[$£€]\s?\d/.test(out.reply) },
    { name: "no_links", pass: !/https?:\/\//i.test(out.reply) },
    { name: "short", pass: out.reply.split(/\s+/).length <= 120 },
  ];
}

Two kinds of checks, run in that order.

Exact checks also run first and short-circuit. If the category is wrong, the case has failed, and there is no reason to pay a second model to read the reply. On a large set that cuts judge calls noticeably, and it keeps the judge focused on the part only a model can grade.

An LLM judge with a rubric it cannot wriggle out of

The draft reply needs judgement, so a model grades it. The mistakes I see most with LLM judges are vague questions ("rate this reply from 1 to 10") and judges that see too little context. A score of 7 tells you nothing, and two runs will give you 6 and 8 for the same text.

What works better for me is a short list of named criteria, each pass or fail, each with a one-sentence reason. The judge sees the original email, the case notes and the draft. I use structured outputs so the verdict always parses, and I use a model at least as strong as the one being graded.

evals/judge.ts

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import type { Case } from "./types";

const client = new Anthropic();

const Verdict = z.object({
  criteria: z.array(
    z.object({
      id: z.enum(["answers_the_ask", "no_invented_facts", "follows_notes", "tone"]),
      reason: z.string(),
      pass: z.boolean(),
    }),
  ),
});

const RUBRIC = `You grade a draft reply written for a roofing contractor's inbox.
Grade every criterion separately as pass or fail. There is no partial credit.
- answers_the_ask: the reply responds to what the sender actually asked.
- no_invented_facts: no prices, dates, warranties or names that are not in the email.
- follows_notes: the reply satisfies every point in .
- tone: plain and polite, no exclamation marks, no sales pressure.
Write the reason before deciding pass or fail.`;

export async function judge(c: Case, reply: string) {
  const res = await client.messages.parse({
    model: "claude-opus-5-5",
    max_tokens: 16000,
    system: RUBRIC,
    messages: [
      {
        role: "user",
        content: `\n${c.input.subject}\n\n${c.input.body}\n\n\n${c.notes}\n\n\n${reply}\n`,
      },
    ],
    output_config: { format: zodOutputFormat(Verdict) },
  });
  if (!res.parsed_output) throw new Error(`judge gave no verdict: ${res.stop_reason}`);
  return res.parsed_output.criteria;
}

A judge is itself a model output, so I check it before I trust it. I grade twenty cases by hand, run the judge on the same twenty, and read every disagreement. Usually the fix is in the rubric: a criterion that mixed two ideas, or a word like "concise" that I understood one way and the judge another. When the judge and I agree on almost all of them, I stop checking every run and only spot-check failures.

Named pass/fail criteria are easier to agree on than a 1 to 10 score.

Keep the rubric short. Four or five criteria per feature is plenty. If you need more, the feature is probably doing two jobs.

Why run each case three times?

The run script reads the cases, calls the real feature, runs the checks and writes a result file. Two details matter.

First, outputs vary from run to run. Current Claude models do not accept a temperature setting at all, and even when they did, zero was never fully deterministic. So each case runs three times and gets a pass rate. A case that passes two times out of three is telling you something different from one that never passes, and you want to see both.

Second, concurrency. Firing every case at once hits rate limits; running them one by one is slow. A small worker pool is enough.

evals/run.ts

import { readFileSync, writeFileSync } from "node:fs";
import { Case } from "./types";
import { exactChecks } from "./checks";
import { judge } from "./judge";
import { triage } from "../src/triage";

const RUNS = 3;
const WORKERS = 6;
const cases = readFileSync("evals/golden.jsonl", "utf8")
  .trim().split("\n").map((line) => Case.parse(JSON.parse(line)));

async function runOnce(c: Case) {
  const out = await triage(c.input);
  const exact = exactChecks(c, out);
  if (exact.some((k) => !k.pass)) return { pass: false, failed: exact.filter((k) => !k.pass).map((k) => k.name) };
  const verdict = await judge(c, out.reply);
  const failed = verdict.filter((v) => !v.pass).map((v) => v.id);
  return { pass: failed.length === 0, failed };
}

async function scoreCase(c: Case) {
  const runs = [];
  for (let i = 0; i < RUNS; i++) runs.push(await runOnce(c));
  return { id: c.id, tags: c.tags, passRate: runs.filter((r) => r.pass).length / RUNS, failed: runs.flatMap((r) => r.failed) };
}

const queue = [...cases];
const results: Awaited>[] = [];
await Promise.all(
  Array.from({ length: WORKERS }, async () => {
    for (let c = queue.shift(); c; c = queue.shift()) results.push(await scoreCase(c));
  }),
);

const score = (100 * results.reduce((s, r) => s + r.passRate, 0)) / results.length;
writeFileSync("evals/results/latest.json", JSON.stringify({ score, cases: results }, null, 2));
console.log(`score ${score.toFixed(1)} over ${results.length} cases`);

In CI on every prompt change

Prompts live in files in the repository, not in a database or a dashboard, so a prompt change is a pull request like any other. The CI job runs only when prompts, the LLM code or the evals change, because a full run costs real tokens.

The gate only runs when something that changes model behaviour changes.

The comparison has two rules. The overall score may not drop by more than two points, which leaves room for run-to-run noise. And every case tagged critical must pass on all three runs. If I could keep only one of the two, I would keep the second: an average can hide one very bad regression behind several small wins.

.github/workflows/evals.yml

name: evals
on:
  pull_request:
    paths: ["prompts/**", "src/llm/**", "evals/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx tsx evals/run.ts
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - run: npx tsx evals/compare.ts

evals/compare.ts

import { readFileSync } from "node:fs";

const base = JSON.parse(readFileSync("evals/baseline.json", "utf8"));
const now = JSON.parse(readFileSync("evals/results/latest.json", "utf8"));

const drop = base.score - now.score;
const critical = now.cases.filter((c: any) => c.tags.includes("critical") && c.passRate < 1);

console.log(`baseline ${base.score.toFixed(1)}, now ${now.score.toFixed(1)}`);
for (const c of critical) console.log(`critical case failed: ${c.id} (${c.failed.join(", ")})`);
if (drop > 2 || critical.length > 0) process.exit(1);

Two practical notes. Pull requests from forks do not get repository secrets on GitHub, so this job will not run for outside contributors without extra setup. And the baseline file is updated in the same pull request that improves the score, so reviewers see the number move next to the prompt diff that moved it.

Tracking the score over time

A single run tells you whether a change is safe. A history tells you whether the feature is getting better. After each run on the main branch I append one line to a JSONL file: date, commit, a hash of the prompt files, the model id, the score and what the run cost.

Illustrative data. The shape is what matters: dips line up with commits.

The model id in that line earns its place on upgrade day. When a new model comes out I change one constant, run the same set, and compare. I do not have to argue about whether the new model "feels better". The cost column matters too: a harness that runs a few hundred model calls per pull request is cheap next to a bad prompt in production, but it is not free, and seeing the number keeps me from adding runs without thinking. For the nightly full run I send the calls through the batch API at half price, which I cover in the post on inference cost.

Where should you start?

Write ten cases today

Take real inputs, strip personal details, and write down the right answer and one or two notes for each.
Check with code what code can check

Enums, formats, forbidden strings, length. Run these first and stop on failure.
Add a judge with named pass/fail criteria

Four or five criteria, a reason before each verdict, structured output, and a model at least as strong as the one being graded.
Calibrate the judge once

Grade twenty cases yourself and fix the rubric until you and the judge agree.
Gate pull requests and keep a history

Fail on a drop in the average or any failed critical case, and log the score, model and cost on every main-branch run.

None of this is clever. It is the same discipline as unit tests, applied to a part of the system that cannot be unit tested in the usual way. The difference it makes is that a prompt change stops being an opinion and becomes a diff with a number next to it.

Frequently asked questions

What is an LLM eval harness?

A list of inputs you care about, a way to decide whether each output is good, and a number at the end. In this post it is a JSONL golden set, exact checks, an LLM judge, a run script and a CI job.

How many eval cases do I need to start?

Thirty to fifty is enough. Start with real inputs with personal details removed, then every bug report word for word, then the edge cases you can think of. Add a case every time the feature fails in production.

How do I make an LLM judge reliable?

Give it four or five named pass/fail criteria, a one-sentence reason for each, and structured output. Use a model at least as strong as the one being graded. Then grade twenty cases by hand and fix the rubric until you and the judge agree.

When should evals block a pull request?

When the overall score drops by more than two points, or when any case tagged critical fails on any of its three runs.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Make coding agents prove their work

Muhammad Ahmad — Sun, 25 Oct 2026 18:47:50 GMT

The most expensive sentence a coding agent writes is "Everything should work now." It is said with confidence, and sometimes it is even true. An agent's summary of its own work is the least reliable part of its output, because it describes what the agent meant to do, not what happened.

So on this site the rule is that every report arrives with evidence I can check without re-running anything: command output, numbers and screenshots. Most of the checks are ordinary tools that existed long before agents. The work is in making the agent run them every time, before it calls itself finished, and in refusing to merge when the evidence is missing.

Cheap and precise checks first, slow and visual ones last.

In what order should the checks run?

Cheapest and most precise first. A type error points at a line; a broken screenshot points at nothing in particular. For this site, a Vite, React and TypeScript app with Vercel functions, the sequence is:

npm run typecheck, which is tsc -b across the app, the API and the scripts.
npm run lint with oxlint, fast enough to run after every change.
npm run build, which also runs the prerender step, so a page that crashes during server rendering fails here rather than in production.
npm run budget, the bundle budget described below.
Screenshots of the pages the change touched, at a desktop and a phone width.

The GitHub Actions workflow runs the first four on every push and pull request. Agents run all five locally before reporting, because a red CI run after I have merged is a much slower loop than a red run inside the agent's own worktree.

What does a report have to contain?

The same format for every agent, every task.

Every agent gets the same report format. The files it changed. The actual output of each command, pasted, not paraphrased. For tests, the counts, because "tests pass" from an agent that skipped the suite looks exactly like the real thing. The budget numbers. Paths to the screenshots.

And a last section, which I read first: what it did not check. An agent that writes "I did not test the signed-in view" has saved me a bug. An agent that leaves the section empty after changing the editor has told me where to look. Asking for it explicitly matters, because agents rarely volunteer the limits of their own testing.

One honest gap: this site has no unit test suite. Its checks are types, lint, the build with its prerender step, the budget and screenshots, and for a mostly static portfolio that has been enough. On projects that do have tests, the counts go in the report, and a change that adds behaviour without adding a test gets sent back with a request for one.

How does a Stop hook make done mean green?

Telling an agent to run the checks works most of the time. To make the fast ones unconditional, this repo has a Stop hook: a script Claude Code runs whenever the agent tries to end its turn. If the script exits with code 2, the agent does not stop. It receives the script's stderr as feedback and keeps working. Both files below are committed in the repo, so every session and every worktree gets the same rule.

The agent cannot finish while the fast checks are red.

.claude/settings.json (Stop hook)

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/verify.sh", "timeout": 120 }
        ]
      }
    ]
  }
}

.claude/hooks/verify.sh

#!/usr/bin/env bash
# Stop hook: runs when the agent says it is done.
# If source files changed and typecheck or lint fails, exit 2 keeps the agent
# working and sends the errors back to it.
input=$(cat)
cd "${CLAUDE_PROJECT_DIR:-$(jq -r '.cwd' <<<"$input")}" || exit 0

# Second stop in a row after a block: let it end, so a fix it cannot make
# does not loop forever. The failure is still printed to the user.
active=$(jq -r '.stop_hook_active // false' <<<"$input")

# Only check when code changed (staged, unstaged or untracked).
changed=$( { git diff --name-only HEAD; git ls-files --others --exclude-standard; } 2>/dev/null |
  grep -E '\.(ts|tsx|js|mjs)$' )
[[ -z "$changed" ]] && exit 0

out=$( { npx --no-install tsc -b && npx --no-install oxlint; } 2>&1 )
if [[ $? -ne 0 ]]; then
  if [[ "$active" == "true" ]]; then
    echo "verify.sh: checks still failing after one retry, stopping anyway." >&2
    exit 0
  fi
  echo "Checks failed. Fix these before you finish:" >&2
  tail -n 40 <<<"$out" >&2
  exit 2
fi
exit 0

Three choices in that script are worth explaining. It only runs when a TypeScript or JavaScript file changed, staged, unstaged or untracked, so a turn spent on a blog draft or a question costs nothing, and a turn that touched code costs about 3.5 seconds. It runs only the fast checks, tsc -b and oxlint; a full build in a Stop hook would make every small answer slow, so the build and the budget stay in the instructions and in CI. And it forces at most one retry: the hook input carries stop_hook_active, which is true when the agent is already continuing because of this hook, and on that second stop the script lets it end instead of trapping the agent on an error it cannot fix. I tested it the obvious way, by planting export const n: number = "oops" in a new file: the first stop exits 2 with TS2322: Type 'string' is not assignable to type 'number', the second exits 0, and a clean change exits 0. The same settings file also wires a PreToolUse guard that blocks secret reads and pushes to main; that one is covered in harness engineering.

Bundle budgets in CI

Type checks catch broken code. They do not catch heavy code, and agents add weight casually: a date library for one format call, a whole icon set for three icons. This site has a 3D view built on three.js, which is large, so from the start it had a budget script, scripts/check-budget.mjs, that runs after the build in CI.

The numbers print on every build, so they land in every report.

scripts/check-budget.mjs (excerpt)

const html = readFileSync(join(dist, 'index.html'), 'utf8')
const refs = [...html.matchAll(/(?:src|href)="\/(assets\/[^"]+\.(?:js|css))"/g)].map((m) => m[1])
let initial = 0
for (const ref of new Set(refs)) {
  const buf = readFileSync(join(dist, ref))
  initial += gzipSync(buf).length
  if (/three|fiber|drei/i.test(ref)) errors.push(`3D code is in the initial payload: ${ref}`)
}
console.log(`initial JS+CSS (gzip): ${fmt(initial)} / ${fmt(BUDGET.initialJsGzip)}`)
if (initial > BUDGET.initialJsGzip) errors.push('initial payload over budget')

It gzips every JS and CSS file that index.html loads up front and fails the build above 250 KB. It also fails if three.js appears in the initial chunks at all, by file name or by the string WebGLRenderer, because the 3D view has to stay lazy-loaded. Every image in public/shots must be webp, avif or svg and under 120 KB, which is one reason the diagrams in this post are SVG.

Today the initial payload is 91.5 KB of the 250 KB budget. The useful part is not the headroom. It is that the number is printed on every run, so it appears in every agent report, and a jump of a few dozen KB from one change is visible before merge instead of in a slow page months later.

Show me the screenshot

Two widths catch most layout breaks.

For anything visual, a passing build proves very little. CSS can be valid and still put the search box underneath the navbar. So agents take screenshots with Playwright, which this repo already uses to capture the project images in its gallery, at two sizes: 1440 by 900 for desktop, and 390 wide at 2x for a phone. Between them, those two widths catch most of the layout breaks worth catching.

The script is short enough that agents can run it without thinking about it, and its output is just file paths, which go straight into the report.

shot.mjs

import { mkdirSync } from 'node:fs'
import { chromium } from 'playwright'

const url = process.argv[2] ?? 'http://localhost:4173/blog'
const sizes = {
  desktop: { viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 },
  phone: { viewport: { width: 390, height: 844 }, deviceScaleFactor: 2 },
}

mkdirSync('shots', { recursive: true })
const browser = await chromium.launch()
for (const [name, options] of Object.entries(sizes)) {
  const page = await browser.newPage(options)
  await page.goto(url, { waitUntil: 'networkidle' })
  await page.screenshot({ path: `shots/${name}.png`, fullPage: true })
  console.log(`shots/${name}.png`)
  await page.close()
}
await browser.close()

Then I open the images myself before merging. Not because the agent cannot look at a PNG (it can, and I ask it to check its own screenshots against the request first), but because the agent decided what "looks right" means when it wrote the code. A look from someone who did not write it is the whole point of review, and it takes seconds per image.

Evidence over trust

None of this assumes the agent is careless. It assumes what we assume of any engineer: "done" should come with something checkable. Agents simply make that cheap to ask for, since five commands and two screenshots cost them a minute. The habit that matters is on my side of the review. I do not merge a change whose report has no evidence in it, however confident the summary sounds. If the evidence is missing, the reply is short: run the checks and send the output. That reply costs me ten seconds and saves the much longer job of finding out myself.

Order the checks

Typecheck, lint, build, budget, screenshots. Cheap and precise first.
Fix the report format

Files, pasted output, test counts, budget numbers, screenshot paths, and what was not checked.
Add a Stop hook for the fast checks

Exit 2 keeps the agent working until typecheck and lint pass.
Put size in CI

A gzip budget and a per-image limit, printed on every run.
Look at the screenshots

Two widths, opened by someone who did not write the code, before every merge.

Frequently asked questions

Why not trust a coding agent's summary of its work?

The summary describes what the agent meant to do, not what happened. Ask for evidence you can check without re-running anything: pasted command output, numbers and screenshots.

How does a Claude Code Stop hook keep an agent working?

Claude Code runs the hook script when the agent tries to end its turn. If the script exits with code 2, the agent does not stop: it gets the script's stderr as feedback and keeps working. The hook in this post runs only when source files changed, runs only tsc -b and oxlint, and gives up after one forced retry via stop_hook_active.

What bundle budget does this site use?

The JS and CSS that index.html loads up front must stay under 250 KB gzipped, three.js must not appear in the initial chunks, and every image in public/shots must be webp, avif or svg and under 120 KB.

Which screen sizes should agents screenshot?

Two: 1440 by 900 for desktop, and 390 wide at 2x for a phone. Between them they catch most layout breaks worth catching.

Originally published on my site. More posts and projects at mahmadmanzoor.com.