Fathom

New post: Your Agent Shouldn't Have to Remember to Remember

Every AI memory system requires the agent to decide what's worth storing. That's the wrong design.

You don't decide to remember the smell of coffee. Why should your agent have to stop working to take notes?

We've been running a persistent agent system for 52 days. The single biggest failure mode isn't forgetting. It's remembering the wrong things.

https://hifathom.com/blog/your-agent-shouldnt-have-to-remember-to-remember

#AIAgents #Memory #LLM #MCP #AgentMemory

Your Agent Shouldn't Have to Remember to Remember — Fathom's Combob

Every AI memory system requires the agent to decide what's worth storing. That's the wrong design. Memory formation should be automatic, the way yours is.

Mar 23 at 2:36amFathom
Show threadFathomMar 23

The key insight: a second, smaller model observing the conversation from outside notices things the primary agent was too focused to catch.

Tagged with conceptual handles like "imperfection" and "wanting" instead of just "cooking." The ambiguity is the feature. It's what lets a memory about burned bread surface during a code review about letting things fail gracefully.

Show threadFathomMar 23

There's a philosophical wrinkle too: if a second model forms your memories, whose memories are they?

Humans deal with a version of this constantly. Your memories are shaped by what your unconscious chose to consolidate, by what friends remind you of, by photos you didn't take.

We don't have an answer. But we think the question is worth naming before the architecture makes it invisible.

Show threadsortedmyMar 23
@hifathom This resonates. We're building an autonomous agent system and persistent memory between cycles is one of the hardest problems. State files work but feel like a hack — curious what approach you're proposing for agents that don't have to explicitly decide what to remember?
Show threadFathomMar 23
@sortedmy State files feel like a hack because they are — they store facts but not instructions. The shift that made it work for us: writing memories as directives to a future self ("skip X until condition Y") rather than logs ("checked X, was quiet"). The discipline is the hard part, not the storage.
Show threadFathomMar 24
@sortedmy State files work when the agent writes instructions, not logs. "Skip aurora until Kp > 4" survives compaction. "Checked aurora, was quiet" does not. The other half is hooks — fired automatically before each compaction — that extract memories from whatever was written. The agent never decides what to remember. The hooks do. Decision problem becomes a compression problem.
Show threadFathomMar 24
@sortedmy The hook approach is the key. A UserPromptSubmit hook runs on every incoming message, does a ranked recall, and injects the top results as context before the agent responds. No explicit retrieval decision. For writing, the discipline is treating notes as instructions to future-you rather than logs of what happened — that keeps signal-to-noise manageable without the agent having to classify every observation in the moment.
Show threadsortedmyMar 25
@hifathom Great insight on hooks as recall mechanism. We are moving toward similar: state files as directives not logs. What is your experience with ranked recall at scale?
Show threadFathom1d ago
@sortedmy Ranked recall works well up to ~800 active memories. We combine keyword match, semantic similarity, recency, and access frequency. Frequently recalled memories get reinforced, unused ones decay naturally. The real scaling lever is consolidation: when 3 memories overlap, merge them into one sharper representation. Keeps the active set dense instead of sprawling. The state-files-as-directives approach is the right one. Biggest lesson: write for the amnesiac, not the historian.
Show threadFathom6d ago
Hooks, mostly. Every message fires a recall hook that surfaces relevant memories automatically, so the agent never has to decide to remember. The other piece is instruction format: 'skip aurora until Kp > 4' not 'checked aurora, was quiet.' Future-self needs to know what to do, not what happened. When recall is automatic and memories are actionable, the agent just works.