Testing The Harness Without A Model

In the last post, we added compression. The harness now owns a lot of behavior: tools, permissions, workspace boundaries, memory, context building, and handoff state.

do not test this with a unit test:
  - will the model choose the perfect tool?

do test this with a unit test:
  - what happens when the model asks for a tool?

Two Layers

A coding agent has two very different layers.

model behavior:
  - probabilistic
  - prompt-sensitive
  - model-version-sensitive
  - evaluated with tasks

harness behavior:
  - deterministic
  - code we wrote
  - should be unit tested
  - should fail loudly when broken

If a test depends on Qwen, Claude, or GPT choosing the same tool every time, the test is really an eval (next on this later)

The Seam

The seam is the model API call. In production, the harness calls an OpenAI-compatible client and receives either text or tool calls. In tests, we replace that client with a fake one.

real run:
  harness -> LM Studio -> model response

test run:
  harness -> FakeClient -> scripted response

This gives us deterministic tests. No network. No local server.

What To Fake

Fake only the response shape the harness actually uses. In Loop, that means:

fake responses:
  - plain assistant text
  - tool call with JSON arguments
  - tool call with invalid JSON
  - final answer after tool results

The fake model lets us ask: if the model returns this exact thing, does the harness do the right next thing?

Tool Round Trip

The most important test is the tool round trip.

1. fake model asks for write_file("hello.txt")
2. harness executes write_file
3. harness appends a tool result
4. fake model receives that result
5. fake model returns final text

This checks the core contract. The model proposes. The harness executes. The result goes back into context.

Bad Tool Arguments

Models sometimes produce malformed arguments. The harness should not crash just because the model produced bad JSON.

fake model:
  read_file("{not-json")

expected harness behavior:
  - do not crash
  - create a structured error
  - send that error back as a tool result
  - let the model recover

This is one of the most important harness tests. The model will make mistakes. The harness decides whether a mistake becomes a crash or useful context.

Permission Tests

Permissions are important harness behavior, so they should be tested without a model too.

permission scenarios:
  - allow once
  - allow for this session
  - deny with a reason
  - deny dangerous paths
  - require approval for sensitive writes

In Loop, the test patches input() so the permission prompt receives scripted answers.

with patch("builtins.input", side_effect=["s"]):
    status, output = run_with_output(messages)

This proves that session approvals are scoped correctly and denial returns a structured tool result instead of silently disappearing.

Workspace Tests

Workspace boundaries should be tested like security boundaries.

workspace tests:
  - writing inside workspace works
  - ../secret.txt is rejected
  - absolute paths outside the root are rejected
  - generated clutter is ignored
  - scratch paths stay under .loop/

These tests do not need a model. They need temporary directories and direct tool calls. The less model is involved, the better.

Context Tests

Context management should also be testable. If a tool result is too large, it should be capped. If the conversation grows beyond budget, old tool output should be compacted or replaced according to harness rules.

context tests:
  - large tool result is capped
  - recent messages are preserved
  - policy text is not compressed away
  - handoff file includes files touched
  - memory search returns relevant memories

Golden Traces

Once the loop grows, single assertions are not always enough. A useful next step is a golden trace.

golden trace:
  - model_call
  - tool_use write_file
  - tool_result ok
  - model_call
  - assistant final_answer

A golden trace records the important shape of a run. The test need to care that the harness took the expected path.

Tests vs Evals

You might be thinkin how this relates to evals.

test:
  if the model asks for write_file, does the harness write safely?

eval:
  given a real task, does the model choose the right steps?

Tests should be deterministic. Evals will involve real models, real tasks, and scoring. We will go there next.

Regression Tests

This connects back to harness engineering. When an agent fails in a repeatable way, the fix often becomes a regression test.

agent failure:
  wrote outside the workspace

harness fix:
  reject paths that escape the root

regression test:
  assert ../secret.txt is denied forever

This is the harness ratchet. Every real failure should leave the system slightly harder to break next time.

The Loop So Far

Here is a compact fake-client test harness. In the real Loop project this lives across loop_cli/fakes.py and tests/test_agent_fake_client.py, but the important idea is the same: script the model, run the harness, assert the result.

import io
import json
import tempfile
import unittest
from contextlib import redirect_stdout
from copy import deepcopy
from dataclasses import dataclass
from pathlib import Path
from unittest.mock import patch

from loop_cli.agent import new_conversation, run_agent
from loop_cli.permissions import reset_session_approvals
from loop_cli.runtime import Runtime, set_runtime


@dataclass
class FakeFunction:
    name: str
    arguments: str


@dataclass
class FakeToolCall:
    id: str
    function: FakeFunction
    type: str = "function"


@dataclass
class FakeMessage:
    content: str | None = None
    tool_calls: list[FakeToolCall] | None = None


@dataclass
class FakeChoice:
    message: FakeMessage


@dataclass
class FakeUsage:
    prompt_tokens: int = 0
    completion_tokens: int = 0


@dataclass
class FakeResponse:
    message: FakeMessage
    usage: FakeUsage | None = None

    @property
    def choices(self) -> list[FakeChoice]:
        return [FakeChoice(self.message)]


class FakeCompletions:
    def __init__(self, responses: list[FakeResponse]):
        self._responses = list(responses)
        self.requests = []

    def create(self, **kwargs):
        self.requests.append(deepcopy(kwargs))
        if not self._responses:
            raise AssertionError("Fake client has no responses left.")
        return self._responses.pop(0)


class FakeClient:
    def __init__(self, responses: list[FakeResponse]):
        self.chat = type("FakeChat", (), {})()
        self.chat.completions = FakeCompletions(responses)
        self.base_url = "fake://local"


def fake_text_response(content: str) -> FakeResponse:
    return FakeResponse(
        FakeMessage(content=content),
        FakeUsage(10, len(content) // 4),
    )


def fake_tool_response(name: str, args: dict) -> FakeResponse:
    return FakeResponse(
        FakeMessage(
            tool_calls=[
                FakeToolCall(
                    id="call_1",
                    function=FakeFunction(name, json.dumps(args)),
                )
            ]
        ),
        FakeUsage(10, 5),
    )


class AgentFakeClientTests(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory()
        self.workspace = Path(self.tmp.name)
        reset_session_approvals()

    def tearDown(self):
        self.tmp.cleanup()

    def configure_fake(self, responses, permission_mode: str = "auto"):
        client = FakeClient(responses)
        set_runtime(
            Runtime(
                client=client,
                model="fake-model",
                workspace=self.workspace,
                memory_file=self.workspace / "memory.json",
                permission_mode=permission_mode,
            )
        )
        return client

    def run_with_output(self, messages):
        output = io.StringIO()
        with redirect_stdout(output):
            status = run_agent(messages)
        return status, output.getvalue()

    def test_tool_call_round_trip(self):
        client = self.configure_fake(
            [
                fake_tool_response(
                    "write_file",
                    {"path": "hello.txt", "content": "hello"},
                ),
                fake_text_response("Wrote the file."),
            ]
        )
        messages = new_conversation()
        messages.append({"role": "user", "content": "Write hello.txt."})

        status, output = self.run_with_output(messages)

        self.assertEqual(status, 0)
        self.assertEqual((self.workspace / "hello.txt").read_text(), "hello")
        self.assertIn("[tool_use] write_file", output)
        self.assertEqual(len(client.chat.completions.requests), 2)

        second_request = client.chat.completions.requests[1]
        self.assertEqual(second_request["messages"][-1]["role"], "tool")
        self.assertIn("Wrote 5 bytes", second_request["messages"][-1]["content"])

    def test_permission_denial_returns_tool_result(self):
        client = self.configure_fake(
            [
                fake_tool_response(
                    "write_file",
                    {"path": "src/app.py", "content": "print('no')"},
                ),
                fake_text_response("I will not write it."),
            ],
            permission_mode="ask",
        )
        messages = new_conversation()
        messages.append({"role": "user", "content": "Write src/app.py."})

        with patch("builtins.input", side_effect=["n", "not now"]):
            status, output = self.run_with_output(messages)

        self.assertEqual(status, 0)
        self.assertFalse((self.workspace / "src" / "app.py").exists())
        self.assertIn('"error_type": "permission"', output)


if __name__ == "__main__":
    unittest.main()

Run it with:

.venv/bin/python -m unittest discover -s tests

Where we are

Testing without a model gives us confidence in the harness itself. The loop can now be checked like normal software: inputs, outputs, boundaries, and contracts.

Next we will turn the real model back on and evaluate behavior. Tests tell us whether the harness works. Evals tell us whether the whole agent succeeds at useful tasks.