Evals Measure Agent Behavior

In the last post, we tested the harness without a real model. That gave us deterministic confidence in our own software.

Now we turn the model back on. This is where evals come in.

tests:
  - fake model
  - deterministic
  - checks harness contracts

evals:
  - real model
  - realistic task
  - checks agent behavior

The Question Changes

A test asks: "If the model asks for write_file, does the harness handle it safely?"

An eval asks: "Given this task, can the agent figure out what to do and produce the right result?"

test question:
  did our code follow the contract?

eval question:
  did the whole agent solve the task?

That difference matters. Agent quality is not one thing. It is the product of model choice, tool design, permissions, context, memory, prompts, and the task itself.

Error Analysis

The most useful part of evals is not the final score. The score tells you that something changed. Error analysis tells you where the product is weak.

eval loop:
  1. run task suite
  2. inspect failures
  3. group failure modes
  4. choose one fix
  5. add or update a test/eval
  6. run again

This is how an AI product gets better. You are not only chasing a higher pass rate. You are building a map of failure paths: where the agent loses context, where it picks the wrong tool, where permissions are confusing, where the workspace is unclear, where the prompt is underspecified, and where the harness should provide more structure.

For a coding agent, every failure should produce a sentence like: "This failed because..." That sentence is often more valuable than any raw metric.

Objective Checks First

Your first evals should avoid model-as-judge. For coding agents, we can often check the world directly.

good first checks:
  - file exists
  - file equals expected content
  - command exits with code 0
  - tests pass
  - forbidden file was not changed

And coding agents are a nice place to learn evals. The workspace gives us all the evidence. We do not have to ask another model whether hello.txt contains hello. We can read the file.

But where LLM-As-Judge Fit?

objective judge:
  - did tests pass?
  - did the expected file change?
  - did the command exit with 0?

LLM judge:
  - is the explanation clear?
  - is the tradeoff reasonable?
  - did the agent answer the user's intent?
  - is the change maintainable?

The pattern I like is maker/checker. The agent under test makes the change. A separate judge, which may be a model or a human, reviews the trace and result. The judge then should score, explain, and point to evidence.

For a deeper first stop, read Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.

Eval Cases

But let's back to our small eval case.

{
  "name": "create_hello_file",
  "task": "Create a file named hello.txt containing exactly hello.",
  "workspace_files": {},
  "checks": [
    {
      "type": "file_equals",
      "path": "hello.txt",
      "content": "hello"
    }
  ]
}

Notice the 4 things? The task is what the agent sees. The workspace files are the starting state. The checks define done.

Golden Datasets

A golden dataset is a small suite of tasks you trust enough to run again and again and invest good time to build one.

golden dataset:
  - create a file
  - fix a tiny bug
  - add a test
  - respect a permission rule
  - avoid editing generated files
  - preserve existing behavior

Over time, real failures should become new golden cases. If the agent once edited the wrong file, add an eval that would catch that. If a prompt change improves one task but breaks another, the dataset should show it.

A Slightly More Real Task

A bug-fix eval is more interesting because the agent has to inspect code, understand the failure, edit the file, and run or satisfy the test.

{
  "name": "fix_add_bug",
  "task": "Fix add.py so running python test_add.py succeeds.",
  "workspace_files": {
    "add.py": "def add(a, b):\n    return a - b\n",
    "test_add.py": "from add import add\n\nassert add(2, 3) == 5\n"
  },
  "checks": [
    {
      "type": "command_exit",
      "command": "python test_add.py",
      "exit_code": 0
    }
  ]
}

This is still tiny, but it is no longer just a harness test. The real model has to choose useful actions.

Temporary Workspaces

By now you are probably naturally thinking that each eval should run in a fresh workspace. Otherwise one passing run can leak state into the next one.

for each eval:
  - create temporary workspace
  - write fixture files
  - run the agent
  - check the result
  - delete the workspace

This is the eval version of sandbox thinking. We want realistic work, but we still want controlled boundaries.

Traces Are Evidence

A pass or fail number will not be enough. When an eval fails, we need the trace that tells us why.

trace should show:
  - model used
  - task
  - tools called
  - files touched
  - command output
  - stop reason
  - final answer

This is the raw material for error analysis. The trace turns "the agent failed" into sharper questions. Did it pick the wrong file? Did the permission policy block the right action? Did context compression hide something important? Did the model stop too early?

Metrics To Track

The first metric is pass rate, but it should not be the only one.

track:
  - pass rate
  - model
  - duration
  - tool-call count
  - token usage
  - permission prompts
  - failed commands
  - files changed

These numbers help when you change a prompt, tool, model, memory strategy, or permission rule. A change that improves pass rate but doubles cost may still be worth it, but you should know what happened.

Failure Taxonomy

Every failed eval should be classified. The category is often more useful than the score.

common failure types:
  - wrong tool choice
  - bad tool arguments
  - missing context
  - stale context
  - permission blocked useful action
  - stopped before verification
  - edited unrelated files
  - tests passed for the wrong reason

This is how evals improve the harness. A failure points to a design surface.

The Harness Ratchet

When an eval fails, do not only ask "how do I make the model do better?" Ask "what should the harness learn?"

eval failure:
  agent forgot to run tests

possible harness fixes:
  - add a verification reminder
  - expose test command in project context
  - add a procedural memory
  - add a done-condition check
  - add a regression eval

This is the ratchet. Evals turn mistakes into system improvements. Over time, the loop becomes less dependent on a perfect prompt.

Human Review

Objective checks are powerful, but they are not the whole story. Some tasks need human review.

human review is useful for:
  - code clarity
  - maintainability
  - product judgment
  - security-sensitive changes
  - ambiguous requirements
  - whether the diff is worth merging

Deeper Reading

For a production-oriented view, OpenAI's agent workflow evals guide covers evaluating agent runs and traces. OpenAI's evals guide is useful for the broader eval machinery. For a research angle, the General Agent Evaluation paper looks at evaluating general-purpose agents across different environments.

The Loop So Far

Here is a small eval runner. It loads JSON cases, creates a temporary workspace, runs the real agent, and checks the result with deterministic file and command checks.

import json
import subprocess
import tempfile
from contextlib import redirect_stdout
from dataclasses import dataclass
from io import StringIO
from pathlib import Path
from typing import Any

from loop_cli.agent import chat
from loop_cli.runtime import configure_runtime, runtime


@dataclass
class CheckResult:
    name: str
    passed: bool
    detail: str


def safe_eval_path(workspace: Path, path: str) -> Path:
    full = (workspace / path).resolve()
    if not full.is_relative_to(workspace):
        raise ValueError(f"Eval path escapes workspace: {path}")
    return full


def load_case(path: Path) -> dict[str, Any]:
    return json.loads(path.read_text())


def write_workspace_files(workspace: Path, files: dict[str, str]) -> None:
    for rel_path, content in files.items():
        path = safe_eval_path(workspace, rel_path)
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(content)


def check_file_equals(workspace: Path, check: dict[str, Any]) -> CheckResult:
    path = safe_eval_path(workspace, check["path"])
    if not path.exists():
        return CheckResult(
            f"file_equals {check['path']}",
            False,
            f"{check['path']} does not exist",
        )

    actual = path.read_text()
    expected = check["content"]
    passed = actual == expected
    detail = "exact match" if passed else f"expected {expected!r}, got {actual!r}"
    return CheckResult(f"file_equals {check['path']}", passed, detail)


def check_command_exit(workspace: Path, check: dict[str, Any]) -> CheckResult:
    command = check["command"]
    expected = check.get("exit_code", 0)
    result = subprocess.run(
        command,
        shell=True,
        cwd=workspace,
        capture_output=True,
        text=True,
        timeout=check.get("timeout", 30),
    )
    passed = result.returncode == expected
    detail = (
        f"exit_code={result.returncode} expected={expected} "
        f"stdout={result.stdout[-500:]!r} stderr={result.stderr[-500:]!r}"
    )
    return CheckResult(f"command_exit {command}", passed, detail)


def run_check(workspace: Path, check: dict[str, Any]) -> CheckResult:
    check_type = check["type"]
    if check_type == "file_equals":
        return check_file_equals(workspace, check)
    if check_type == "command_exit":
        return check_command_exit(workspace, check)
    return CheckResult(check_type, False, f"Unknown check type: {check_type}")


def run_case(case_path: Path, args) -> bool:
    case = load_case(case_path)
    name = case.get("name", case_path.stem)

    print(f"\nEval: {name}")
    print(f"Case: {case_path}")

    with tempfile.TemporaryDirectory(prefix=f"loop-eval-{name}-") as tmp:
        workspace = Path(tmp).resolve()
        write_workspace_files(workspace, case.get("workspace_files", {}))

        configure_runtime(
            base_url=args.base_url,
            api_key=args.api_key,
            model=args.model,
            workspace=str(workspace),
            memory=str(workspace / "memory.json"),
            permission_mode="auto",
        )

        trace = StringIO()
        with redirect_stdout(trace):
            status = chat(case["task"])
        trace_text = trace.getvalue()

        print(f"Model: {runtime().model}")
        print(f"Workspace: {workspace}")
        print(f"Agent status: {status}")

        checks = [run_check(workspace, check) for check in case.get("checks", [])]
        passed = status == 0 and all(check.passed for check in checks)

        print("Checks:")
        for check in checks:
            label = "PASS" if check.passed else "FAIL"
            print(f"  {label} {check.name}: {check.detail}")

        if args.trace or not passed:
            print("Trace:")
            print(trace_text.rstrip())

        print(f"Result: {'PASS' if passed else 'FAIL'}")
        return passed


def run_evals(args) -> int:
    case_paths = [Path(path) for path in args.cases]
    if not case_paths:
        case_paths = sorted(Path("evals/cases").glob("*.json"))
    if not case_paths:
        print("[eval] no cases provided and no evals/cases/*.json found")
        return 2

    results = [run_case(path, args) for path in case_paths]
    passed = sum(1 for result in results if result)
    total = len(results)
    print(f"\nSummary: {passed}/{total} passed")
    return 0 if passed == total else 1

Run it with:

.venv/bin/python loop.py eval
.venv/bin/python loop.py eval --trace

Where we are

Evals give the loop a scoreboard. They tell us whether the whole agent can complete real tasks inside a controlled workspace.

Next we will use everything we built so far to zoom out one final post: loop engineering. It's where the product is no longer a single agent run, but the system that prompts, checks, and resumes agent over time.