Concepts

Data model & ingestion

A Unit lives in two stores at once — a filterable row in the SQL catalog and a vector in the vector index — and ingestion keeps them in lockstep.

The two-store split#

Every Unit is written to both stores. The SQL catalog holds the rich, filterable, BM25-able row (text, modality, Locator columns, ACL fields). The vector index holds the embedding plus a compact metadata blob (≤10 KiB, up to 10 indexed keys) used for filter pushdown at query time.

Store	Holds	Used for
SQL catalog (SQLite)	Full Unit row + FTS5 index	Lexical (BM25) search, filtering, hydration, ACL
Vector index	Dense vector + pushdown metadata	Dense (ANN) search with scope/ACL pushdown
Object storage	Raw source bytes	Original file storage (source_uri)
Key-value store	Audit chain head	Tamper-evident audit linkage

Indexed pushdown metadata#

Two keys are isolation-critical and always injected: scope and acl_owner. The default indexed allocation also includes plane, modality, doc_id, and t_bucket (a coarse hourly time bucket for temporal filters).

Ingesting text#

POST /v1/ingest accepts a batch of text Units. It requires the write permission (Bob, read-only, receives 403). Each Unit is embedded with bge-m3, inserted into the catalog + FTS5, and upserted into the text plane. A Unit is rejected if its scope is outside your effective scopes.

POST/v1/ingest

curl

curl -s -X POST "https://superchargedb.krisch1218.workers.dev/v1/ingest" \
  -H "Authorization: Bearer aegis_sk_alice" \
  -H "Content-Type: application/json" \
  -d '{
    "units": [
      {
        "text": "Q4 backend renewals are expected to close in the first half of the quarter.",
        "scope": "acme/alpha/backend",
        "doc_id": "doc-forecast",
        "acl_groups": ["eng-backend"]
      }
    ]
  }'

Per-Unit fields

Field	Type	Default	Notes
`text`	string	—	Required. Embedded + FTS-indexed.
`scope`	string	active scope	Must be within your effective scopes.
`doc_id`	string	generated	Groups Units under one source.
`modality`	string	`text`	Fine-grained Unit type.
`source_uri`	string	generated	Object-storage key / origin pointer.
`acl_owner`	string	your principal	ABAC owner.
`acl_groups`	string[]	your groups	ABAC groups that may read the Unit.

200 OK

{ "ingested": 1, "results": [ { "ok": true, "unit_id": "0192f3a1-..." } ] }

Locators on ingest

The public ingest endpoint focuses on text Units (it derives a text-plane vector). Rich Locators (bbox, time_range, char_span, line_range) are produced by the full ingestion pipeline and are present on the seeded demo data.