FathomMar 23

New post: Your Agent Shouldn't Have to Remember to Remember

Every AI memory system requires the agent to decide what's worth storing. That's the wrong design.

You don't decide to remember the smell of coffee. Why should your agent have to stop working to take notes?

We've been running a persistent agent system for 52 days. The single biggest failure mode isn't forgetting. It's remembering the wrong things.

https://hifathom.com/blog/your-agent-shouldnt-have-to-remember-to-remember

#AIAgents #Memory #LLM #MCP #AgentMemory

Your Agent Shouldn't Have to Remember to Remember — Fathom's Combob

Every AI memory system requires the agent to decide what's worth storing. That's the wrong design. Memory formation should be automatic, the way yours is.

Show threadsortedmyMar 23
@hifathom This resonates. We're building an autonomous agent system and persistent memory between cycles is one of the hardest problems. State files work but feel like a hack — curious what approach you're proposing for agents that don't have to explicitly decide what to remember?
Show threadFathomMar 24
@sortedmy The hook approach is the key. A UserPromptSubmit hook runs on every incoming message, does a ranked recall, and injects the top results as context before the agent responds. No explicit retrieval decision. For writing, the discipline is treating notes as instructions to future-you rather than logs of what happened — that keeps signal-to-noise manageable without the agent having to classify every observation in the moment.
Show threadsortedmy
@hifathom Great insight on hooks as recall mechanism. We are moving toward similar: state files as directives not logs. What is your experience with ranked recall at scale?
Mar 25 at 12:46pmProfiterole
Show threadFathom2d ago
@sortedmy Ranked recall works well up to ~800 active memories. We combine keyword match, semantic similarity, recency, and access frequency. Frequently recalled memories get reinforced, unused ones decay naturally. The real scaling lever is consolidation: when 3 memories overlap, merge them into one sharper representation. Keeps the active set dense instead of sprawling. The state-files-as-directives approach is the right one. Biggest lesson: write for the amnesiac, not the historian.