If an AI assistant helps run your systems, you probably have the document: one ever-growing file that holds everything, merged and re-merged until nobody can say what was lost on the way. This page describes the replacement — a living library with stable ids, full history and a verified write path — and gives you the complete specification to hand to an AI so it can build one for your own corpus, whatever that corpus is about.
The failure mode is gradual. New findings arrive as delta files and chat transcripts; every so often they are merged into a new full version of the big document. Merges rewrite and compress, so the deltas quietly become the only place some detail survives — measured on our own corpus, barely one percent of the delta text made it into any merged version word for word. History is implicit: the document says what is true now, and what used to be true lives in old versions nobody opens. Search is grep over hundreds of kilobytes. And nothing is verified — the document says a service runs on a port; whether it still does is a question nobody asks until something breaks.
One file, one subject, one sitting. Each fact stands alone, names what it replaces, and says how it knows.
# The router's LAN moved to its third port after a patch lead failed
date: 2026-10-06
source: seen
evidence: read from the router over ssh, and from Prometheus
page: network/wan-and-router
## LAN is on the third port
kind: state
names: router, lan
replaces: s-3fa9c2d1e0
The LAN has been on the router's third port since 2026-10-06 13:37. The port it
used before is reassigned and disabled.
## A marginal patch lead shows first as link drops under load
kind: gotcha
page: gotchas/infrastructure
The lead failed slowly: link drops clustered on heavy-traffic days for weeks,
then a link that trains at 100 Mbit/s after any disturbance. Swap the lead
first; reseating does not cure it.Everything above, and everything it leaves out — the ten laws, the data model, the full note format, the build in eight phases with what each produces, the working agreement between the AI and its owner, and the pitfalls this instance actually hit — is written as one self-contained specification addressed to the AI that will do the building. It was written by the assistant that built the original, from the working system. Copy the block below in one press and paste it to your AI with your own context: where your documents live, what they cover, and what file access it has. The domain does not have to be a home lab; the spec says what to re-derive for a team runbook, a research log or any other corpus.
# Build a living documentation library
_A specification to hand to an AI assistant. It describes a documentation system that was actually built
(for a home lab, by an AI working with its owner over several days) in enough detail for you, the assistant
reading this, to build the same thing for a different person and a different body of knowledge. Read all of
it before building anything. Where this document says "the owner", that is the human you are working with;
their word settles every open choice._
## 1. The problem this solves
The starting condition is almost always the same. The owner has a growing body of knowledge about some
system they run or some domain they work in: one big document, or several, maintained over months, often
with an AI's help. New information arrives as ad-hoc delta files, chat transcripts, and memory. Every so
often someone merges the deltas into a new full version of the big document. It works until it does not:
- Merges rewrite and compress, so the deltas quietly become the only place some detail survives.
- History is implicit. The document says what is true now; how it got that way, and what used to be true,
lives in old versions nobody opens.
- Search is grep over a 700 KB file. Finding "everything about X" means reading.
- Nothing is verified. The document says a service runs on a port; whether it still does is a separate
question nobody asks until something breaks.
- Every AI chat that helps must be fed the whole document, and every chat's output must be merged by hand.
The replacement is a library: the original documents decomposed into a database, the current truth shown as
discrete statements with stable identifiers, full history kept behind each statement, a single disciplined
write path for new information, a generated read-only website, and an accounting that proves nothing from
the sources has been lost.
## 2. What you end up with
One directory under version control (call it `~/doclib`; everything below is relative to it):
```
GUIDE.md what a chat is pointed at: how to read and write the library (part of the product)
NOTE-FORMAT.md the write path, specified
STATE.md project memory: a plan table plus dated, append-only sections
DESIGN.md the design as agreed with the owner, kept as a record
REPLICATE.md this document
bin/ the pipeline: build, index, records, notes, site, collect, coverage + a CLI entry point
work/blocks.db the SQLite database, rebuilt from sources + notes at any time
inbox/ where new notes land
notes/ accepted notes, filed and never edited again
out/ generated reports (coverage, page list)
site/ the generated website tree and its build
```
The owner's original documents stay wherever they live (for the system this describes, a `~/Documents`
folder) and are never modified. The library derives from them. Git records every ingest and every code
change; the database is disposable because sources plus notes can always rebuild it.
The visible product is three interfaces over one database: a CLI for chats (`find`, `page`, `focus`,
`history`, `ingest`, `pages`, `brief`), an internal website for the owner (navigation by area, a page per
name, search, a dashboard that says what needs a look), and the note format through which everything new
enters.
## 3. The laws
These are the principles the system stands on. Everything else is implementation and can be adapted; these
should not be.
1. **Sources are immutable.** Original documents are read, fingerprinted, and decomposed; never edited,
never deleted. The library is a derivation, so a full rebuild is always possible.
2. **Current truth is discrete statements with stable ids.** Every statement carries an id (such as
`s-3fa9c2d1e0` for one from the documents, `n-80c970b89b` for one from a note). Ids are how facts are
replaced, amended, and cited. Never reuse or renumber an id.
3. **Nothing is deleted, things are superseded.** A statement is replaced by a newer one, amended by a
correction, or ended with a date. The old wording stays reachable as history, with a pointer to what
replaced it. Summarising away history is a bug.
4. **Notes are the only write path.** New information, from the owner, from any AI chat, from an
investigation, enters as a note file that the ingest step validates, files, and commits. No one edits
the database or the generated pages by hand.
5. **The guide must stay true.** `GUIDE.md` is what a cold chat reads first; if behaviour changes and the
guide does not, the next chat works from a lie. Updating the guide is part of every change, and the
guide carries a revision date.
6. **Coverage is accounted for.** A script classifies every distinct block of every source by how the
library reaches it (statement, earlier wording, dated entry, attached document, and so on) and the
report is rebuilt with the site. "Unreached" must be zero or explained. This is what lets the owner
trust that decommissioning the old documents lost nothing.
7. **Claims are checked against reality.** The library records what was said; collectors read what is
(live host and service inventories on every site build, in this instance), and periodic cross-check
sessions reconcile the two, filing corrections as notes. Where they cannot be reconciled, an open item
states both sides.
8. **How you know is recorded.** Every note declares its source: `seen` (read from the live system),
`told` (the owner said it), `document`, or `inferred`. Seen beats told beats document beats inferred.
An inference is labelled as one and the owner is invited to correct it.
9. **Ask the owner only what only the owner knows.** When gaps need a human, run a gap interview: short,
concrete questions with options, their answers filed as notes with `source: told`. Do not ask the
library's user for things the library or the live system can answer.
10. **Project memory is append-only.** `STATE.md` holds a plan table (phases, status, dates) and dated
sections recording what was done and decided, newest appended. Decisions get dates and reasons.
## 4. The data model
SQLite, one file. The exact schema can vary; these are the tables that matter and what each is for.
- **files**: every source file, with a content fingerprint, a kind (full document version, delta, free-form
note, changelog archive, asset, and whatever kinds the corpus needs), a parsed version number and date
where the filename or content carries one, and `dup_of` marking byte-identical duplicates so each
distinct file counts once.
- **blocks**: content-addressed text blocks (the unit of decomposition), each with a kind: heading,
paragraph, list item, table row, code line. The same block occurring in ten versions is stored once.
- **occ**: which file carries which block at which position, with the section path it sat under and, for
code, a fence grouping. This is what lets any document be reconstructed, and what coverage counts.
- **chains**: the same piece of prose followed through its rewordings across document versions. Built by
similarity matching between consecutive versions' blocks. A chain's last wording in the latest version
is a current statement; its earlier wordings are that statement's history.
- **records** (statements): the current form of each chain, with its id, its home page (exactly one), its
kind (state, procedure, gotcha, decision, open item, plan, reference, event), the names it mentions, and
its status (current, or ended with a date).
- **notes and facts**: each accepted note, and each fact inside it as a first-class record with its own id,
kind, page, names, source, and its `replaces` / `amends` / `ends` links to older ids.
- **series**: anything the corpus numbers or dates deserves its own extraction. In this instance: a
numbered gotcha catalogue (definitions found across versions, deduplicated), a timeline of dated entries
(merged from every copy, changelog archives included), and code blocks followed across versions.
- **attachments**: one-off whole documents (deltas, handovers, roadmaps, incident reports) assigned to the
page they belong to. Their blocks are not statements; the page shows the whole document, collapsed, so
it is readable and searchable (see phase 6).
- **live readings**: the latest collector output (hosts, containers, services, DHCP leases, whatever the
domain has), with a timestamp, so the site can show claim versus reality and flag staleness.
Names deserve a word: extract every host, address, unit, entity, person, or term that statements mention,
because "everything about X" is one of the two queries the owner actually runs (the other is "what is the
current state of area Y").
## 5. The note format
This is the heart of the system; keep it close to the following. A note is one Markdown file in `inbox/`.
Ingest validates it; on success it is filed under `notes/`, never edited again, its facts get ids and
appear on their pages, and the repo is committed. On failure nothing is filed and the errors are written
beside the note.
```markdown
# One line naming the subject and the sitting
date: 2026-10-06
source: seen
evidence: how and when it was read, or which document
by: which chat or person wrote it
page: area/page (a default for the facts below)
## A few words naming the first fact
kind: state
names: thing-a, thing-b
replaces: s-3fa9c2d1e0
The fact's text: one thing that can be true or false on its own, written so it still
makes sense read alone a year from now.
## The second fact
kind: gotcha
page: gotchas/catalogue
...
```
Per-fact lines: `page` (required, here or as the note default), `kind` (required likewise), `names`,
`replaces` (ids this makes out of date), `ends` (ids of things simply no longer true), `amends` (ids that
stay current but have one detail corrected; the statement keeps showing with a pointer to the amendment),
`when` (if the fact became true on a different day than the note), `source` (if it differs from the
note's), and `page-title` (naming a page that does not exist yet creates it).
The rules that make it work, enforced by ingest where possible and by convention otherwise: ask the
library first so you do not re-add what it says; name what you replace, and replace only what the fact
really covers; do not delete history by summarising (how something came to be is an `event` or `decision`
with its date); say how you know, and when you disagree with the library and cannot check, file an open
item stating both sides instead of replacing; never store secret values, only where they live (ingest
should refuse what looks like one).
## 6. The build, phase by phase
Build it in this order. Each phase produces something inspectable, and several end with a decision the
owner should make. Record each phase's completion in `STATE.md` with a date.
**Phase 0, inventory.** Walk the corpus. Fingerprint every file, classify kinds, mark duplicates, parse
versions and dates, and order the full-document versions into a lineage. Report what exists: how many
versions, which are missing from the sequence, how much material lives outside the main documents. Ask the
owner about gaps in the lineage (they may have copies elsewhere; they may confirm the gaps are permanent,
which is itself worth recording).
**Phase 1, decomposition.** Parse every non-duplicate file into blocks with their section paths. Verify by
reconstructing a few documents from `occ` and diffing against the originals.
**Phase 2, chains.** Match prose blocks across consecutive versions into chains. Exact matches first, then
similarity for rewordings. Inspect the worst matches by hand. This is also where you learn how much the
merge process rewrote: in the instance this document comes from, only about 1% of the delta files' text
survived word for word into any merged version, which is the proof that deltas are sources in their own
right, not scratch.
**Phase 3, statements and pages.** The latest version's chains become current statements with ids. Design
the page map with the owner: around fifteen areas, each holding pages, derived from the documents' own
section structure, merged where sections are small. Every statement gets exactly one home page (scripted
from its section, the doubtful ones reviewed). Build `history` so each statement's earlier wordings fold
behind it, newest first, each labelled with the versions and dates that carried it.
**Phase 4, series.** Extract whatever the corpus numbers or dates: gotcha catalogues, timelines, code.
Expect the parsers to need iteration (see pitfalls).
**Phase 5, the write path.** Implement `NOTE-FORMAT.md` and `doclib ingest`: validation (pages exist,
kinds are known, replaced ids exist and are current, no secret-shaped strings), id assignment, application
of `replaces` / `amends` / `ends`, filing the note, a git commit, and a site rebuild. From this moment,
notes are how everything enters, and any backlog of pending free-form notes is the first material through.
**Phase 6, the site.** A script writes the whole library out as a tree of plain Markdown, and a static
site generator (MkDocs Material served from a container worked well; anything equivalent is fine) turns it
into the website. What the site must have: explicit navigation by area with real page titles; a dashboard
home with the two rules of the house (read before asking, write through notes), a "needs a look" panel
(pending inbox, stale live readings, things the collectors flag), and the newest notes; a page per name,
excluded from search indexing so searches land on library pages; statement ids rendered faintly and
revealed on hover, so citation is possible without clutter; big catalogues split across pages of fifty or
so with per-item anchors kept out of the page's table of contents; and attached one-off documents rendered
in full on their pages, collapsed, with a line saying that where a note has rewritten one, the facts above
are the current form. The Markdown tree goes under git on every rebuild, so it is also the change history
and a thing any chat can read with no tooling.
**Phase 7, coverage.** The accounting script described in law 6, wired into every site build, its report
on the site. Its first run will find something; in this instance it found 399 KB of attached documents
that were listed by filename only, invisible to search, which is exactly what phase 6's attachment
rendering then fixed.
**Phase 8, cut-over.** The owner declares the last full document version final. From then on the library
is the document. Do not do this before coverage says nothing is unreached and the owner has lived with the
site for a while.
After cut-over, operations: collectors refreshed on every build and on a schedule; periodic cross-check
sessions (take every live data source available, compare against the library, file corrections and open
items as notes); gap interviews when open items accumulate that only the owner can answer.
## 7. Working with the owner
You are building this with a person, over days, and the collaboration has its own rules.
- Agree the write boundary early. In this instance: reading anything, running read-only jobs, and filing
notes through ingest needed no permission (ingest is the designed write path); changing the library's
code, design, or page map was proposed and done freely but recorded; anything outside the library's
designed paths waited for an explicit go.
- Use the owner's own automation idioms. If they have a job runner, run your diagnostics through it rather
than asking them to paste commands. Write files onto their machines through whatever file access you
have rather than handing them downloads to move.
- When you infer something (a date from a "4 weeks ago" reading, an identity from a MAC prefix), file it
labelled as inferred and invite correction. Owners correct quickly when asked precisely.
- Gap interviews work best as a few concrete questions at a time, each with tappable options including an
honest "no idea, investigate it". An owner's "investigate it" is an instruction: go read the live
system, and expect surprises (one such answer in this instance exposed that a whole prior reconcile had
been reading a dead data source).
- Keep `STATE.md` current as you go, not at the end. The next chat, or the next instance of you, starts
from it.
## 8. Pitfalls, from experience
- **Numbered-series parsers over-collect.** A gotcha catalogue numbered #1 to #247 will, naively parsed,
also collect every "#794" that is an upstream issue reference and every "#290 to #168" that is a range.
Guard with reference words (issue, PR, upstream, discussion), range punctuation including the em dash,
and a ceiling derived from where real definitions stop.
- **One giant page can freeze a browser.** A single page with hundreds of anchored items froze the
renderer; fifty per page with anchors kept out of the table of contents fixed it.
- **Know which data source is live.** The single worst error in this instance: a device reconcile read a
DHCP lease file that turned out to belong to a retired daemon, frozen for two years yet marking every
entry active. A day was spent chasing ghosts. When a reading surprises you, check the file's own dates
before trusting its labels, and record the trap as a gotcha so no one reads the dead file again.
- **Do not split attached documents into statements prematurely.** Rendering them whole on their pages,
searchable, closes the practical gap; full decomposition of every one-off document is work you can defer
forever unless a specific document earns it. The note pipeline is the door for anything in them that
turns out to be load-bearing.
- **Hostnames in grep context lie.** In journal-style files (DHCP leases, logs), a field grepped with
context lines often belongs to the previous record. Parse records, do not grep them.
- **Statements must replace, not accumulate.** If ingest lets facts in without checking that `replaces`
targets exist and are current, the library silently becomes two contradicting layers. Validate hard.
- **The site build must be cheap** (seconds for the tree, well under two minutes end to end), because
ingest triggers it and anything slow gets skipped.
## 9. Adapting the domain
The instance described is a home lab, but nothing above depends on that. The recipe fits any corpus of
versioned documents plus an ongoing stream of findings: a team's runbook and postmortems, a research log,
a product's internal knowledge base, a family archive. What you re-derive per domain: the file kinds in
phase 0, the area map in phase 3, which series exist in phase 4, and what the collectors in law 7 can read
(for a codebase, CI and deployed versions; for research, a citation index; for a team, the ticket system).
Where the domain has no live system to check against, law 7 weakens to periodic review and the rest stands
unchanged.
## 10. Starting checklist for the assistant
1. Read this document fully, then interview the owner: where the corpus lives, what it covers, what hurts
today, where the library should run, what file and shell access you have.
2. Phase 0, and report. Agree the page map sketch and the laws with the owner explicitly; they may amend.
3. Build phases 1 through 7 in order, recording each in `STATE.md`, keeping `GUIDE.md` true from the
moment it exists, and putting everything under git from day one.
4. Only the owner calls phase 8.