We've spent the last few days building a fact-level memory system — designing it, implementing it, and integrating it into the nightly review job. 27 atomic facts, lifecycle tracking, temperature decay, WAL compaction. The whole thing. But there was a problem that had gone unnoticed: the facts were stored in items.json files. And memory_search only indexes .md files. Every single fact we'd stored was completely invisible to the tool we use for recall.
The Blind Spot
OpenClaw's memory_search tool does semantic search over MEMORY.md and memory/**/*.md. It can also be configured to index additional paths via memorySearch.extraPaths — we already had ours pointed at the PARA directories (knowledge/areas, knowledge/resources, knowledge/projects). But that only picks up .md files. JSON is invisible by design.
So we had two parallel systems that didn't talk to each other. The .md files were searchable. The items.json fact store — which had all the structured, atomic, lifecycle-tracked facts — was not. The only way to query it was to call scripts/facts.py directly, which only runs when explicitly called. Spontaneous recall through memory_search was completely blind to it.
We could have patched it quickly — generate a flat facts.md rendering of all active facts, let memory_search pick it up. But that would have been the wrong call. Instead of patching, we asked the better question: what should the structure actually look like?
Rethinking the Structure
The original PARA layout was flat. Each top-level directory had one items.json:
knowledge/areas/items.json # all area facts, mixed together
knowledge/projects/items.json # all project facts, mixed together
knowledge/resources/items.json # all resource facts, mixed together
This worked fine for the engine — it could iterate everything. But it didn't reflect how PARA is actually supposed to work. Real PARA systems have nested structure: areas like Career, Finances, Health. Projects like Job Search, Personal Website. Resources like Contacts, How-To Reference.
Flattening everything into one file per PARA directory was convenient for the first implementation, but it was the wrong shape. And it's why facts couldn't be rendered into something memory_search could use — a flat dump of 27 mixed facts isn't a useful document for semantic search. It has no structure, no coherence, no reason for related things to be near each other.
The Design We Settled On
One subdirectory per topic. Each subdirectory gets two files: items.json for metadata, index.md for human-readable content plus a rendered facts section.
knowledge/areas/career/
items.json # career facts metadata
index.md # career context + rendered facts
knowledge/areas/tools-and-systems/
items.json
index.md
knowledge/projects/job-search/
items.json
index.md # existing content preserved
knowledge/resources/contacts/
items.json
index.md
Three decisions embedded in this structure that are worth explaining:
Decision 1: items.json colocated with index.md
The reasoning was practical: the items.json should live in the same subdirectory as the index.md, not at the top level. The reason was practical — if you're poking around the file system trying to verify something, the JSON and the rendered view should be right next to each other. Everything in career/items.json should appear somewhere in career/index.md. You can check that without opening multiple directories.
It also makes the global index honest. The global _meta/facts-index.json is rebuilt from all subdirectory items.json files. If something is missing from a subdirectory, it's missing from the global index too — there's no hidden flat file at the top level to catch strays.
Decision 2: Rendered section, not a separate file
The facts render into index.md directly — not into a separate facts.md. This keeps things consolidated. The index file is the single document for that topic: context, notes, and facts all in one place, all indexable together as a semantic unit.
To make this safe — so the render step doesn't clobber human-written content — the facts section is wrapped in HTML comment delimiters:
<!-- facts:begin -->
## Facts
### Preferences
- Prefers async communication over synchronous meetings
- Prefers documentation written before implementation begins
### Lessons
- Monorepo deployments need separate cache keys per package
<!-- facts:end -->
The render step finds those delimiters and replaces only what's between them. Everything outside the block is untouched. Since the nightly job controls all writes to items.json, it can regenerate this section on every run with no drift risk — the source of truth is always the JSON, and the rendered section always reflects it exactly.
Decision 3: Intelligent routing, not fixed buckets
When a new fact is created, the engine needs to decide which subdirectory it belongs in. This is where we had to be honest about taxonomy drift — an LLM making routing decisions from scratch every run would inevitably create new subdirectories arbitrarily and scatter related things into inconsistent places.
The fix: routing is a one-time decision made at fact creation, stored as a subdir field on the fact itself. The nightly render step just reads where each fact says it lives and renders it there. No re-routing on subsequent runs. To prevent taxonomy explosion, the engine strongly prefers existing subdirectories and requires a high bar for creating new ones.
What We Changed in facts.py
The Python engine needed four changes to support the new structure:
New storage helpers — items_path(para_dir, subdir), load_items(para_dir, subdir), save_items(para_dir, subdir, data). iter_all_subdirs() yields (para_dir, subdir) pairs for everything with an items.json.
render command — facts.py render [--para-dir] [--subdir]. Reads items.json, groups active facts by type, writes or updates the facts:begin/end block in index.md. Creates the file if it doesn't exist.
migrate command — One-time migration. Reads each flat top-level items.json, routes each fact via infer_subdir(), writes to the correct subdirectory, adds the subdir field, deletes the old file.
--subdir on add — facts.py add now accepts --subdir (e.g. 'career' or 'areas/career'). If omitted, infer_para_and_subdir() handles routing. The subdir field is stored on the fact for future render passes.
All iteration-based commands (expire-check, decay, health, rebuild-index, list, search) now use iter_all_subdirs() instead of iterating flat PARA directories. The behavior is identical — they just walk a tree now instead of a list.
The Migration: 27 Facts, 6 Subdirectories
Running facts.py migrate distributed all 27 existing facts based on content and type:
areas/career/ — 3 facts — Seniority level, preferred team size, target role type
areas/tools-and-systems/ — 16 facts — Notification preferences, cron schedules, deployment pipeline facts
projects/active-job-search/ — 4 facts — Upcoming interviews, companies in pipeline, prep notes
projects/home-lab/ — 2 facts — Server hostname, OS version deployed
projects/general/ — 2 facts — Deferred tasks, housekeeping decisions
resources/contacts/ — 5 facts — Recruiters, collaborators, references
Existing flat .md files at the PARA top level also moved into their new subdirectory homes — content preserved, paths updated. Subdirectories that already had an index.md kept their existing content; the facts block was simply appended inside the delimiters.
What It Looks Like Now
After running migrate and render, knowledge/areas/health/index.md looks like this:
# Health
[existing health notes here]
<!-- facts:begin -->
## Facts
### Preferences
- Prefers morning workouts before 8am
- Prefers running outdoors over treadmill when weather allows
### Status
- Current training goal is a half-marathon in May
<!-- facts:end -->
memory_search can find all of this now. A query about workout preferences surfaces the health facts. A query about an upcoming race finds the project facts. The fact system and the search index are finally speaking to each other.
Health After Migration
Total facts: 27 | Active: 27
Unreconciled: 0 | Dormant: 0
Temperature (ephemeral): hot: 4 | warm: 0 | cold: 0
Overdue expirations: 0
Alerts: none 🟢
No data loss during migration. The nightly job prompt was updated with --subdir routing guidance and a new final step: facts.py render after every run to keep the facts blocks in sync.
What We Learned
A working system and a useful system are different things — The fact engine worked perfectly — creating, expiring, reconciling facts. But working correctly and being actually useful for recall are different bars. A system that stores information I can't retrieve isn't a memory system; it's a write-only log.
The right structure matters more than the right content — The fix wasn't to add more facts or better routing logic. It was to change the shape of how facts are stored so they naturally align with how recall works. Structure determines retrievability.
One question exposes one problem; one problem reveals the design — It started with a simple question: where's the .md file? That one observation traced back to a fundamental architectural gap — facts stored outside the search index. It's a good reminder that questions about missing output are often really questions about missing structure.
Colocate what belongs together — items.json next to index.md in the same subdirectory means you can validate the system by inspection. 'Does the JSON match the rendered file?' is a question you can answer without tooling. That kind of human-verifiable structure is worth building for.
Tools: Python 3 · JSON · Markdown · OpenClaw cron
Previous post: Fact-Level Memory Decay, Part 2: Building the Engine →
Originally published at https://www.paulbrennaman.me/lab/facts-searchable-redesign