Multimodal hybrid search

The pipeline#

Each search flows through a fixed sequence of stages, all reported in the response trace:

• embed — the query is embedded with bge-m3 (1024-d).
• vectorize — dense ANN query per plane, scope-pushed-down against the vector index.
• d1_fts — a BM25 lexical query against the catalog FTS5 index, scope-clamped in SQL.
• rrf — the ranked lists are fused with Reciprocal Rank Fusion.
• rerank — a cross-encoder (bge-reranker-base) reorders the ACL-visible top candidates.

Multimodal planes#

Text is the universal join key: every encoder aligns to text, so one query reaches images, audio, video, and scanned documents without you writing a query per modality. Choose the planes to search with the planes field; results from every plane are fused into a single ranked list.

Every hit carries a locator — the discriminated union described in Core concepts — so a video match resolves to a timestamp window, an image match to a bounding box, a page match to a page bbox, and a text match to a character span. That is the differentiator: the exact moment, region, passage, or line, not just a document id.

Filtering by modality

Within (or across) planes you can restrict retrieval to specific modalities with the modalities field, or with a structured filter on the modality key. For example, limit an audio-plane search to only spoken utterances:

{
  "q": "what did the CFO say about the failover",
  "planes": ["audio", "visual", "doc_visual"],
  "modalities": ["asr", "audio_utterance", "video_shot"],
  "top_k": 10,
  "rerank": true
}

Reciprocal Rank Fusion#

RRF combines ranked lists without needing comparable raw scores. For an id at rank i (0-based) in a list, it contributes 1 / (rrf_k + i + 1); an id’s fused score is the sum of its contributions across all legs. Agreement across legs is what lifts a result to the top.

dense leg:   [A, B, C]      A -> 1/61, B -> 1/62, C -> 1/63
lexical leg: [B, A, D]      B -> 1/61, A -> 1/62, D -> 1/63

fused: A = 1/61 + 1/62,  B = 1/62 + 1/61,  C = 1/63,  D = 1/63
=> A and B (top in both legs) outrank C and D (listed once).

A larger rrf_k flattens the curve (less emphasis on exact rank); a smaller value sharpens it.

Score breakdown#

Each hit carries a scores object so you can see why it ranked where it did:

The top-level score is the rerank score when reranking ran, otherwise the fused score. Assert on stable invariants (ordering, membership, ACL counts) rather than exact float scores — models evolve.

The trace#

Cost is a first-class citizen. The trace reports per-stage latency and compute cost, a total, an estimated USD cost (at $0.011 / 1,000 inference units), and whether the result was a cache hit.

{
  "stages": [
    { "stage": "embed", "ms": 42, "neurons": 120, "detail": "bge-m3 1024d" },
    { "stage": "vectorize", "ms": 61, "detail": "text dense; 30 candidates" },
    { "stage": "d1_fts", "ms": 9, "detail": "18 lexical candidates" },
    { "stage": "rrf", "ms": 1, "detail": "k=60; 34 fused" },
    { "stage": "rerank", "ms": 180, "neurons": 900, "detail": "12 reranked" }
  ],
  "total_ms": 300,
  "total_neurons": 1020,
  "est_cost_usd": 0.01122,
  "cache": "miss"
}

Cited answers

Search returns raw hits; when you want a synthesized, grounded answer, call POST /v1/answer (or set answer: true). It extends the pipeline with three more trace stages — localize, verify, and synth — and populates the answer field with inline citations, each resolving to a sub-unit Locator, plus an nli_faithfulness score. On the plain POST /v1/search route with no answer requested, answer is null and you build responses from hits and their Locators.