The memory failures that hurt most in long-running agents come from bookkeeping. Nobody can say where a stored fact came from, whether it is still true, or which of two contradicting entries should win. Retrieval quality matters, but a perfect retriever over an unreliable store returns unreliable answers faster.
This article describes an architecture that keeps two memories and gives them different jobs. A human-readable Markdown knowledge base, organized like an Obsidian vault, holds durable truth. A semantic conversational memory, backed by embeddings, holds a searchable projection of that truth plus recent context. The first is canonical. The second can be deleted and rebuilt at any time.
Why one memory is never enough
A pure vector memory is convenient. The agent says something, the system embeds it, and a later similarity search brings it back. It also has no concept of a fact being replaced. Suppose a deployment preference changes in March. The January note and the March note can both sit close to the query “how do we deploy”, and the one that surfaces may depend on phrasing and chunk boundaries rather than on which is current.
A pure file-based knowledge base has the opposite profile. It is accurate, diffable and reviewable by a person, yet an agent can miss the right note when the user phrases a question in words the note never used. Each store covers the other’s weakness, provided the relationship between them is strict.
If the index and the notes disagree, the index is wrong by definition.
Two stores, two jobs
Canonical knowledge base
- Markdown files with structured frontmatter.
- Owns decisions, preferences, procedures and facts, each with a source and status.
- Written through a validated path, by people or agents.
- Versioned and backed up because losing it is real loss.
- Read by people first, agents second.
Semantic memory
- Embedded chunks of canonical notes plus recent conversation turns.
- Owns nothing durable.
- Written only by the indexer.
- Disposable: losing it costs retrieval speed until a rebuild.
- Every hit points back to a canonical note.
The mental model is a ledger and a search engine. The ledger is careful about what it accepts. The search engine is fast, approximate and disposable. Nobody audits a search engine; they audit the ledger.
Anything an agent retrieves must carry the ID of a canonical note, or be labelled as unverified conversational context. An answer built from the second kind should say so.
What a canonical note carries
Each note is a Markdown file with frontmatter and a short body. The fields that matter are a stable ID, a type such as decision or procedure, a status such as active or superseded, provenance, sensitivity, and the replacement chain.
- Provenance: who or what produced the claim, with a pointer to the evidence.
- Status: whether the note is active, disputed, draft or superseded.
- Sensitivity: metadata that can travel into retrieval filters.
- History: what the note replaces and when it changed.
The body stays plain prose, because people read it. A note a person cannot read is a note nobody will correct.
The write path
New knowledge never goes straight into the vault. It enters as a candidate and passes through a fixed sequence:
- Capture. Propose a candidate with the raw excerpt it is based on.
- Classify. Decide what it is and whether it belongs in durable memory at all.
- Filter. Redact or reject credentials, tokens and personal identifiers before persistence.
- Validate. Check schema, provenance, duplicates and conflicts.
- Commit. Write the canonical note in one reviewable operation.
- Index. Chunk, embed and upsert vectors with the note ID and content hash.
conversation / document
|
[ candidate ]
|
classify --> ephemeral --> semantic layer only
|
privacy filter --> reject / redact
|
validate --> conflict? --> review queue
|
canonical note (Markdown, versioned)
|
indexer (chunk, embed, hash)
|
semantic memory <-- query -- agent
|
hit + note ID --> agent reads canonical noteThe order is deliberate. Filtering sits before validation and commit because a secret that reaches any durable store has already leaked, and embeddings are a poor place to discover that.
Provenance prevents quiet authority
A claim without a source stays in draft and is never served as fact. The source field distinguishes what a person stated, what a document says, and what the agent inferred.
This prevents a quiet failure: an agent infers a preference from one remark, stores it, and three sessions later repeats it as established. When the source is visible as an agent inference, the claim cannot be promoted without confirmation.
Validation and conflict handling
On commit, compare a candidate with active notes of the same type and subject. There are three outcomes:
- Duplicate. Merge into the existing note and update its review date.
- Extension. Create a new note that links to the old one.
- Contradiction. Do not overwrite anything. Send the candidate to review.
When a replacement is confirmed, the new note records supersedes. The old note stays on disk, changes to superseded and is excluded from the index by default. Last-write-wins destroys context; keeping both lets retrieval choose by phrasing.
Rule of thumb. An agent may add knowledge on its own authority. Replacing knowledge requires a source that outranks the original.
Privacy filtering and re-indexing
The canonical store is plain text. If sensitive content slips in, it can be found with a text search, removed, and re-indexed by note ID. Doing the same inside a vector store alone is much harder because there is no readable source to search.
Because the index is derived, rebuilding it is routine. Triggers include an embedding model change, new chunking rules, a schema change or an edit to a note. Build a full rebuild into a new index and swap it in when complete, so the agent never queries a half-built one. Keep a few regression questions with known answers to check retrieval after the swap.
Limits worth stating
The review queue can grow faster than anyone clears it. Conflict detection can miss contradictions phrased very differently. Model-based classification makes mistakes. Sampling and periodic review therefore belong in the operating routine, and the canonical store must stay readable by people.
Conclusion
Durable knowledge lives in files a person can read and correct. Semantic memory is an index over those files and holds no truth of its own. Every write records provenance, resolves conflicts explicitly and filters sensitive data before anything persists.
Agents with this structure can still be wrong, but their errors are traceable to a note, a source and a decision. That makes them fixable.
