soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: multi-process safety + read-only inspector mode

Browse source 
Filesystem storage now enforces single-writer, many-reader semantics.
A second writer on the same data directory throws at init time with the
holder's PID, hostname, and heartbeat — replacing the previous silent
stale-reads failure mode.

- New: `Brainy.openReadOnly()` — coexists with a live writer, every
  mutation throws clearly.
- New: writer lock at `<rootDir>/locks/_writer.lock` with 10s heartbeat
  and stale-detection (PID liveness + heartbeat freshness).
- New: cross-process flush-request RPC (filesystem-based, no signals)
  so inspectors can force fresh state on demand.
- New: `brain.stats()`, `brain.explain(findParams)`, `brain.health()`
  for operator-facing introspection.
- New: `brainy inspect` CLI with 13 subcommands (stats, find, get,
  relations, explain, health, sample, fields, dump, watch, backup,
  repair, diff), all read-only by default.
- Same-PID re-opens allowed with a warning (preserves test "simulate
  restart" patterns).
- Storage instances passed directly via `storage: new MemoryStorage()`
  are now honoured instead of silently falling through to the
  filesystem auto-detect path.

Brainy + Cortex compose under this model — the lock covers both because
they share `rootDir`, Cortex segments are immutable mmap files, and
MANIFEST updates use atomic-rename.
This commit is contained in:
David Snelling 2026-05-15 11:25:05 -07:00
parent 1bc6a430c7
commit 4fcdc0fef3
12 changed files with 2342 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

152
docs/concepts/multi-process.md Normal file
View file

@ -0,0 +1,152 @@
---
title: Multi-Process Model
slug: concepts/multi-process
public: true
category: concepts
template: concept
order: 5
description: How Brainy coordinates a single writer with any number of readers on a filesystem data directory — and how to safely inspect a live store.
next:
- guides/inspection
---
# Multi-Process Model
Brainy is a **single-writer, many-reader** database when backed by filesystem
storage. This page explains the model, the guarantees, and the safe ways to
inspect a live store from a second process.
## The rule
For one data directory:
- **One writer** at a time. The writer acquires an exclusive lock on the
directory at `init()` and releases it on `close()`.
- **Any number of readers**, concurrent with each other and with the writer.
Readers open via `Brainy.openReadOnly()` — they never touch the writer
lock.
Any attempt to open a second writer on the same directory throws:
```
BrainyError: Another writer holds this Brainy directory.
PID: 1774431 on host app-host-1
Started: 2026-05-15T14:22:11Z
Heartbeat: 2026-05-15T14:22:34Z
Version: 7.21.0
Directory: /data/brain
```
This is intentional. Two writers sharing a directory would silently corrupt
in-memory indexes and produce wrong query results — the worst possible default
for an operations tool.
## Why a lock?
Brainy keeps its primary indexes (HNSW, metadata, graph adjacency) in memory.
On disk, those indexes are persisted incrementally as writes flush. A second
process opening the same directory:
- Loads the *persisted* state into a fresh in-memory copy.
- Has no awareness of writes the first process buffered but hasn't flushed.
- Will overwrite the persisted state on its own next flush, racing the first
process and corrupting whichever wins.
The fix is the lock: refuse to open a second writer. SQLite has done the same
since the late 1990s (`SQLITE_BUSY`).
## What about Cortex?
Brainy + Cortex compose cleanly under this model:
- Cortex stores its column-index segments inside the same `rootDir` (under
`indexes/_column_index/{field}/`).
- Segments (`*.cidx` files) are **immutable** once written. Cortex mmaps them
read-only.
- The `MANIFEST.json` per field is updated via atomic rename — readers see
either the old or new manifest, never a torn file.
A reader process can safely mmap Cortex segments alongside a live writer
without coordination. The single Brainy writer lock at
`<rootDir>/locks/_writer.lock` covers Cortex too, because Cortex segment
writes happen on the writer's side.
## Stale-lock detection
If a writer crashes or is forcibly killed, its lock file is left behind. To
avoid a permanently-jammed directory, Brainy treats a lock as stale when:
1. The recorded `hostname` equals the current host (cross-host PID checks
are unsafe), AND
2. The recorded `pid` is no longer alive (`process.kill(pid, 0)` returns
`ESRCH`), OR the `lastHeartbeat` field is older than 60 seconds.
A live writer rewrites `lastHeartbeat` every 10 seconds, so a hung writer
that's missed several heartbeats is treated as dead. Stale locks are
overwritten with a warning.
If stale detection cannot prove the existing lock is dead — for example, a
crashed writer on a different host writing to a shared filesystem — pass
`{ force: true }` to override. A warning is logged either way.
## Heartbeat and shutdown
The heartbeat interval rewrites the lock file every 10 seconds. The timer
is unref'd, so it does not keep the event loop alive on its own.
On normal shutdown the writer releases the lock in `close()`. The shutdown
hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also
release the lock so a container restart doesn't strand the directory.
## How to inspect a live writer
Use `Brainy.openReadOnly()`. It does not acquire the writer lock, so it
coexists with whatever the writer is doing:
```typescript
const reader = await Brainy.openReadOnly({
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
})
const stats = await reader.stats()
const bookings = await reader.find({ where: { entityType: 'booking' } })
await reader.close()
```
What the reader sees reflects the writer's most recent **flush** to disk. If
you need fresher state, ask the writer to flush before opening:
```typescript
const reader = await Brainy.openReadOnly({
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
})
const acked = await reader.requestFlush({ timeoutMs: 5000 })
if (!acked) {
console.warn('Writer did not respond; results reflect last natural flush.')
}
const fresh = await reader.find({ where: { entityType: 'booking' } })
```
The CLI `brainy inspect` subcommands all do this for you by default
(`--no-fresh` to opt out).
## What's not enforced (yet)
- **Cloud storage backends** (S3, GCS, R2, Azure) do not currently enforce
multi-process locking. Two processes pointing at the same bucket can both
succeed at `init()` in writer mode and clobber each other's writes. A
best-effort warning is logged in writer mode against a non-filesystem
backend. Lock semantics for cloud backends will land in a future release.
- **Long-running readers** do not automatically pick up new Cortex segments
the writer publishes. One-shot inspector calls re-open the store and see
fresh segments; a reader that stays open for hours sees its column store
as-of the time it opened.
## Reading material
- `Brainy.openReadOnly()` — [API reference](../api/brainy.md)
- `brainy inspect` — [inspection guide](../guides/inspection.md)
- Cortex columnar storage — see `node_modules/@soulcraft/cortex/README.md`

187
docs/guides/inspection.md Normal file
View file

@ -0,0 +1,187 @@
---
title: Inspecting a Live Brainy
slug: guides/inspection
public: true
category: guides
template: guide
order: 30
description: Operator recipes for diagnosing a running Brainy data directory — counts, queries, query plans, health checks, and snapshots — without stopping the live writer.
next:
- concepts/multi-process
---
# Inspecting a Live Brainy
When something is wrong in production, you need to see what's actually in the
store. This guide covers the safe ways to query a running Brainy directory.
## The cardinal rule
**Never open a second writer on the same directory.** It will throw on
filesystem storage; on cloud storage it'll silently overwrite the live
writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI
instead.
## The CLI is the fastest path
```bash
# What's in this brain?
brainy inspect stats /data/brain
# Find specific entities
brainy inspect find /data/brain --type Event --where '{"status":"paid"}' --limit 20
# Single entity by ID
brainy inspect get /data/brain 0b7a9...
# Why is this query returning empty?
brainy inspect explain /data/brain --where '{"entityType":"booking"}'
# Quick invariants
brainy inspect health /data/brain
# Random sample (no query needed)
brainy inspect sample /data/brain --type Event --n 20
# Tail new writes as they happen
brainy inspect watch /data/brain --type Event
# Save a snapshot
brainy inspect backup /data/brain /backups/brain-$(date +%Y%m%d).tar
```
Every subcommand internally:
1. Asks the live writer to flush via the cross-process RPC (skip with `--no-fresh`).
2. Opens the data directory via `Brainy.openReadOnly()`.
3. Runs the query.
4. Closes cleanly.
Results are JSON by default. Add `--pretty` for indented output.
## When a query returns surprising results
If `find()` returns `0` for a query you expect to match: run `inspect
explain` first. It shows which index path will serve each `where` clause:
```bash
$ brainy inspect explain /data/brain --where '{"entityType":"booking","status":"paid"}'
{
"query": { "where": { "entityType": "booking", "status": "paid" } },
"fieldPlan": [
{ "field": "entityType", "path": "none", "notes": "No index entries for field..." },
{ "field": "status", "path": "column-store", "notes": "O(log n) binary search..." }
],
"warnings": [
"Field \"entityType\" has no index entries. find() will return [] silently."
]
}
```
The `"path": "none"` is the smoking gun. It means the field has no column
store manifest and no sparse chunked index — so `find()` will return `[]`
regardless of what's actually on disk. Likely causes:
- The writer registered the field in memory but hasn't flushed. Run
`brain.requestFlush()` from the writer side, or use `brainy inspect
--fresh` (default).
- The field name has a typo or wrong casing.
- The field is genuinely absent from every entity.
## Health checks
`inspect health` runs a fixed battery of cheap invariant checks:
```bash
$ brainy inspect health /data/brain
{
"overall": "warn",
"checks": [
{ "name": "index-parity", "status": "pass", "message": "HNSW (1851) and metadata (1851) agree." },
{ "name": "field-registry", "status": "pass", "message": "23 fields registered for 1851 entities." },
{ "name": "seeded-records", "status": "warn", "message": "15 entities tagged _seeded:true." },
{ "name": "writer-heartbeat", "status": "pass", "message": "Writer healthy (PID 1774431...)." }
]
}
```
Each check returns `pass`, `warn`, or `fail`. The exit code is `2` when any
check fails — useful for piping into monitoring or CI.
## Programmatic inspection
```typescript
import { Brainy } from '@soulcraft/brainy'
const reader = await Brainy.openReadOnly({
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
})
// Force the writer to flush before reading
await reader.requestFlush({ timeoutMs: 5000 })
// What's in there?
const stats = await reader.stats()
console.log(`${stats.entityCount} entities`, stats.entitiesByType)
// Why is this query empty?
const plan = await reader.explain({ where: { entityType: 'booking' } })
for (const f of plan.fieldPlan) {
console.log(`${f.field} -> ${f.path}`)
}
// Run invariants
const health = await reader.health()
console.log(health.overall)
await reader.close()
```
Every mutation method (`add`, `update`, `delete`, `relate`, `commit`,
`fork`, `branch`, ...) throws on a read-only instance with a clear message.
## Backups
`brainy inspect backup` asks the writer to flush first, then tars the
directory. The snapshot reflects the writer's state at the moment of the
flush:
```bash
brainy inspect backup /data/brain /backups/brain-2026-05-15.tar
```
For periodic backups (hourly, daily), schedule this via cron or your
container scheduler. For point-in-time recovery, use Brainy's COW
`commit()` API — the snapshots there are content-addressed and never
overwritten.
## Comparing two stores
`brainy inspect diff` returns a JSON summary of counts and a sample of
entity IDs present in one but not the other. Useful when debugging
replication or migrations:
```bash
brainy inspect diff /data/brain-prod /data/brain-staging
```
Sample-based — for a full diff, dump both with `inspect dump` and compare
the JSONL.
## Repairing a corrupted store
If invariants fail and you suspect index corruption, `inspect repair`
opens the store in writer mode and rebuilds all indexes from raw storage.
**Stop the live writer first** — `repair` will throw if another writer
holds the lock. Add `--force` only if you have personally verified the
existing lock is stale.
```bash
brainy inspect repair /data/brain
```
## Multi-process safety summary
See [concepts/multi-process](../concepts/multi-process.md) for the lock
semantics, heartbeat behavior, and what's not yet enforced on cloud
backends.