Memory Is External State

In the last post, we made project context visible. The agent could generate a map of the workspace, store it in .loop/PROJECT.md, and load it when useful.

Memory is the next step. It is what lets the agent carry selected facts across runs.

project context:
  - what is true about this workspace

memory:
  - what the harness decided is worth keeping

That difference matters. Context is what goes into the model request right now. Memory is external state that may be used to build a future request.

The Model Does Not Remember

The model call is stateless from our point of view. We send messages in. We get a message back. If something from yesterday should matter today, the harness has to put it into today's request.

messages + tools + selected memory -> model -> answer or tool call

So "agent memory" is not a mystical property of the model. It is a system around the model.

observe -> decide what to save -> store -> retrieve -> inject into context

Once we see memory this way, we can design it like any other part of the harness.

How A Stateless Model Feels Stateful

We cannot make the model itself much less stateless from the outside. Each call still receives input and returns output. But we can make the agent system feel stateful by giving the harness durable state and by giving the model tools for interacting with that state.

- model is stateless
- harness is stateful
- memory tools write and read state
- context builder decides what returns to the prompt

That means the model can have relevant facts "at hand" on every turn, even though it did not remember them internally. The harness assembles those facts before the model call.

system instructions
+ current task
+ project context
+ relevant memories
+ recent tool results
-> model request

This is a product design choice as much as a technical one. Memory should feel helpful, but it should not be a surprise. The user should be able to ask what the agent remembers, change it, and remove it.

What Should Survive?

A beginner memory system should not save everything. Most of a run is not memory. It is temporary working state: failed attempts, command output, partial plans, and details that only mattered for that one task.

Useful memories are facts that make future runs better.

good memories:
  - user prefers small focused changes
  - project uses pytest
  - deploy happens through Cloudflare Pages
  - never edit generated files under dist/
  - ask before changing database migrations

bad memories:
  - command failed once at 14:03
  - model considered three plans
  - temporary file existed during one run
  - user typo from a single prompt

This is the first principle: memory should reduce future orientation cost without polluting future decisions.

Memory Is A Write Path

In many demos, memory looks like a magic notebook. The more important question is: who is allowed to write to it?

For a coding agent, I like starting with explicit memory writes. The user or harness can say:

/remember This project uses pytest, not unittest.
/remember Ask before changing files in migrations/.
/remember User prefers explanations before large edits.

Later, the harness can propose memories automatically, but I would still make them reviewable.

Proposed memory:
  "Project uses pnpm for frontend commands."

Save? [y] yes  [n] no  [e] edit

This is where self-learning begins, but it is not model training. The model weights are not changing. The system is learning by updating external state.

Memory Tools

If memory is part of the harness, then memory should have tools just like files and shell commands have tools. A small agent does not need many.

- remember_fact(text, kind)
- search_memory(query)
- update_memory(id, text)
- forget_memory(id)

These tools give the model a way to ask for state instead of hoping the right facts were already loaded. The important part is that the tools still belong to the harness. The model may propose a memory write, but the harness decides whether that write is allowed, whether it needs user review, and where it is stored.

model proposes:
  remember_fact("Project uses pytest.", "semantic")

harness decides:
  - save now
  - ask user
  - reject
  - store as session-only

This keeps the self-learning loop controlled. The agent can learn from work, but it cannot silently rewrite its own rules.

Memory Is Also A Read Path

Saving memory is only half the design. The other half is retrieval.

task -> search memories -> choose relevant ones -> add to model context

Our first version can be very simple. Load all memories from a small JSON file and prepend them to the conversation as a short context block.

Relevant memory:
- User prefers small focused changes.
- Project uses pytest.
- Ask before changing migrations/.

This might look minimalistic, but the shape is important here. More advanced systems still follow the same primitive: select stored state and turn it into context. They are just much better at selection.

keyword search:
  find memories that share words with the task

vector search:
  - find memories with similar meaning

scoring:
  - rank by relevance, confidence, and source

recency:
  - prefer facts seen or confirmed recently

memory types:
  - load policy memories before tool calls
  - load project memories before code edits

graph relationships:
  - connect facts to files, tools, users, tasks, and decisions

Imagine the task is "add a migration for the new billing field." Keyword search might find a memory containing "migration." Vector search might also find "database schema changes require review." Type filtering might prioritize policy memories before ordinary project notes. Recency might prefer the migration rule confirmed last week over one written six months ago.

task:
  add a migration for billing

selected memories:
  - [policy] Ask before changing migrations/.
  - [project] Backend uses Alembic.
  - [procedural] Run pytest tests/db after schema changes.

In order to study this more deeply, check the basics of RAG, because memory retrieval is a small, agent-shaped version of retrieval augmented generation. Then look at stateful agent frameworks like Letta and LangGraph memory, because they make memory scopes and state explicit. And finally, read MemGPT, as it frames memory as moving information between limited context and larger external storage.

Where Context Building Lives

This version of context building lives in the harness, not inside the model. The harness is easier to test, easier to inspect, and easier to constrain. But the model has an important role to play depending on your agent use case.

def build_context(task, workspace):
    project = load_project_context(workspace)
    memories = search_memories(workspace, task)
    recent = summarize_recent_tool_results()
    return assemble_prompt(project, memories, recent, task)

This function decides what the model sees. That means it also decides what the model can reasonably do next.

A separate model can help later, but we will not make it the source of truth at the beginning. A "memory curator" model can propose summaries, classify memories, or rank which memories look relevant. The harness will still own the final decision.

deterministic harness:
  - loads project map
  - filters memory by scope
  - enforces token budget
  - blocks sensitive state

optional curator model:
  - proposes memory writes
  - summarizes long events
  - ranks candidate memories
  - explains why a fact is relevant

This separation is useful because curator models can be wrong too. They can over-summarize, keep the wrong detail, or turn a one-time event into a permanent rule. The harness should treat their output as a proposal, not as reality.

Memory Scopes

To make memory feel stateful without making it messy we can scope it. Different facts deserve different lifetimes.

turn memory:
  - what is useful for the next model call

session memory:
  - useful until this CLI run ends

project memory:
  - useful whenever working in this repository

user memory:
  - useful across projects

policy memory:
  - permission and safety choices

This mirrors the permission flow we already built. "Allow once" is turn memory. "Allow for this session" is session memory. "Always ask before editing migrations" is policy memory. Once you see it this way, permissions and memory become two views of the same stateful harness.

Three Kinds Of Memory

A useful mental model is to split memory into three kinds. These names come up often in agent discussions, and they map cleanly to coding agents.

semantic memory:
  - durable facts like "This repo uses pytest."

episodic memory:
  - remembered events like "Last run failed because Docker was not running."

procedural memory:
  - learned ways of working like "For this project, run tests before proposing a final answer."

For our tiny loop, semantic memory is the safest place to start. It is easier to inspect and less likely to invent a false story about the past.

Harness Memory

Harness memory is about how the agent should behave. If you feel lost loop back (pun intended!) to articles on the harness and permissions.

- approval preferences
- safe command patterns
- project-specific guardrails
- preferred verification commands
- files or folders to avoid

This is powerful because it lets the product adapt without changing the model. If the user repeatedly says "you do not need to ask for tests," the harness can save a session rule or a project rule. If the user says "always ask before touching migrations," that becomes a durable policy memory.

But this is also where memory can become dangerous. A bad memory can silently weaken governance. The user must be able to inspect it, edit it, and delete it.

Context Memory

Context memory is about what the agent should know before it starts working. We touched on this in the previous post about project context.

- this repo has a backend and frontend
- main tests live in tests/
- documentation lives in docs/
- generated output should not be edited manually
- Cloudflare deploys from main

Some of this belongs in .loop/PROJECT.md. Some belongs in memory. The distinction I like is:

PROJECT.md:
  generated map of the workspace

memory.json:
  learned durable facts and preferences

The project map can be regenerated from disk. Memory is the part the agent learned from interaction.

Self-Learning Without Model Training

In this stage of the loop, self-learning means:

- the system observes useful feedback
- the harness turns it into stored state
- future requests use that state

That is already enough to make the agent feel much less forgetful. It can remember project conventions, user preferences, repeated mistakes, and verification patterns.

It is also much easier to debug than model training. If the agent behaves strangely, we can open the memory file and ask: "what did we teach it?"

In a larger loop, memory also becomes progress state. It can hold what is done, what is next, what failed, what needs review, and which agent or human owns the next step. That state is what lets tomorrow's run pick up from today's work instead of starting cold.

What Larger Systems Do

Bigger systems differ in their implementation, but let's remember that memory needs controls, not just storage. ChatGPT memory, for example, separates saved memories from past chat context and gives users ways to view, edit, or delete remembered details. See OpenAI's Memory FAQ for the product shape.

larger systems usually need:
  - memory visibility
  - user editing
  - deletion
  - source tracking
  - scope controls
  - retention rules
  - sensitive-data handling

Recent research is also probing the risks. A 2026 paper on memory in ChatGPT studied real user memory entries and found that memory raises questions about sensitivity, agency, and what the system infers about the user. That matters for our tiny loop too. Even small memory needs visible controls.

What To Build First

Enough theory! Our first memory system should be boring in the best possible way.

.loop/memory.json

It should support five operations:

/remember <text>  save a memory
/memory           list memories
/memory <query>   search memories
/forget <id>      delete a memory
/memory clear     delete all memories

It should not silently remember sensitive data. It should not overwrite permissions without asking. It should not pretend every past event is still true.

The Loop So Far

Here is the minimal memory layer. It stores facts in a local JSON file, gives each memory an ID and scope, lets the user delete memories, and renders a small context block that can be added to the next model request.

import json
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from uuid import uuid4


MEMORY_PATH = ".loop/memory.json"


@dataclass
class Memory:
    id: str
    text: str
    kind: str
    scope: str
    created_at: str


def memory_path(workspace: Path) -> Path:
    return workspace / MEMORY_PATH


def now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()


def load_memories(workspace: Path) -> list[Memory]:
    path = memory_path(workspace)
    if not path.exists():
        return []

    raw = json.loads(path.read_text())
    return [Memory(**item) for item in raw]


def save_memories(workspace: Path, memories: list[Memory]) -> None:
    path = memory_path(workspace)
    path.parent.mkdir(parents=True, exist_ok=True)
    data = [asdict(memory) for memory in memories]
    path.write_text(json.dumps(data, indent=2) + "\n")


def remember(
    workspace: Path,
    text: str,
    *,
    kind: str = "semantic",
    scope: str = "project",
) -> Memory:
    memories = load_memories(workspace)
    memory = Memory(
        id=uuid4().hex[:8],
        text=text.strip(),
        kind=kind,
        scope=scope,
        created_at=now_iso(),
    )
    memories.append(memory)
    save_memories(workspace, memories)
    return memory


def forget(workspace: Path, memory_id: str) -> bool:
    memories = load_memories(workspace)
    kept = [memory for memory in memories if memory.id != memory_id]
    if len(kept) == len(memories):
        return False
    save_memories(workspace, kept)
    return True


def update_memory(workspace: Path, memory_id: str, text: str) -> bool:
    memories = load_memories(workspace)
    for memory in memories:
        if memory.id == memory_id:
            memory.text = text.strip()
            save_memories(workspace, memories)
            return True
    return False


def search_memories(
    workspace: Path,
    query: str,
    *,
    limit: int = 8,
) -> list[Memory]:
    words = {word.lower() for word in query.split() if len(word) > 2}
    if not words:
        return []

    scored = []
    for memory in load_memories(workspace):
        text = memory.text.lower()
        score = sum(1 for word in words if word in text)
        if score:
            scored.append((score, memory))

    scored.sort(key=lambda item: item[0], reverse=True)
    return [memory for _, memory in scored[:limit]]


def render_memory_context(memories: list[Memory]) -> str:
    if not memories:
        return ""

    lines = ["Relevant saved memories:"]
    for memory in memories:
        lines.append(f"- [{memory.scope}/{memory.kind}] {memory.text}")
    return "\n".join(lines)


def build_context(task: str, workspace: Path) -> str:
    memories = search_memories(workspace, task)
    memory_context = render_memory_context(memories)
    if not memory_context:
        return task
    return f"{memory_context}\n\nUser task:\n{task}"


def build_messages(task: str, workspace: Path) -> list[dict]:
    return [
        {
            "role": "system",
            "content": "You are a careful coding agent.",
        },
        {
            "role": "user",
            "content": build_context(task, workspace),
        },
    ]

Where we are

Memory gives the loop continuity. It lets the harness preserve useful facts, preferences, and rules across runs without pretending the model itself changed.

Next we will go deeper into real life tasks and more advanced context management: compression. Once an agent runs for more than a few turns, the problem is not only what to remember. It is how to shrink yesterday's messy work into something tomorrow's model can actually use.