Search Architecture

This describes how SearchService (backend/src/v1/search/search.service.ts) actually resolves a query, and - more importantly - the constraints and past experiments behind why it works this way. Some of this exists only in past conversation/decision history, not in code comments or commit messages, so it's written down here for both human developers and AI agents working on search: getting these decisions wrong silently regresses ranking quality rather than throwing an error.

The three sources a query can hit

1. Postgres full-text search (SearchEngine.POSTGRES) - ts_rank over a to_tsvector('english', ...) index on searchableText. The original, still-default engine; a search with no inEngines/ninEngines given resolves to Postgres only, preserving pre-multi-engine behavior exactly.
2. OpenSearch full-text search (SearchEngine.OPENSEARCH) - plain multi_match across content.title^5, content.summary^2, searchableText, content.body, boosted by field specificity. The title^5 boost specifically exists because crawled pages share a large nav/sidebar boilerplate block in their body text, which otherwise makes a generic query match on some other page's nav mention rather than the actual target page (found in the search-tuning loop's first iteration, lab/alan/search-tuning/eval-history.jsonl).
3. Vector search - one call per requested embeddingConfigId, each against its own embedding_chunks_{id} OpenSearch index. One index per config, not one shared index, because dense_vector fields have a fixed dimension count per field and different embedding configs can use different models/dimensions. This is deliberately not a SearchEngine enum value - embedding_config rows are dynamic and user-created with no fixed enumeration, unlike Postgres/OpenSearch - so it's an orthogonal embeddingConfigIds array on the request instead, combinable with any engine selection.

A query can hit any combination of these simultaneously. The fast path (exactly one text engine, no vector configs) skips merging entirely and returns that engine's results directly, unchanged from pre-hybrid-search behavior.

Merging: normalized-score-sum fusion, not raw scores or rank position

Once more than one ranked list is in play, results are merged in mergeRankedLists() by normalized-score-sum fusion (a CombMNZ-style technique from classical IR), not by comparing raw scores directly. This is necessary, not a stylistic choice: ts_rank, OpenSearch's BM25 _score, and cosine similarity from a vector search are not on a comparable numeric scale, so sorting by "highest raw score wins" across sources would be meaningless.

This replaced an earlier reciprocal rank fusion (RRF) approach, which scored purely by position within each list (weight / (RRF_K + rank)). RRF was dropped because it discards how confident a match actually was: a handful of near-tied, low-confidence vector hits (common for a broad query against a batch of similar templated/newsletter-style content) could occupy top rank positions and outvote one high-confidence text-engine match, even though the text match's own score said it was far more relevant. Score magnitude is what should decide that, and RRF never looks at it.

The current approach instead min-max normalizes each list's own scores to [0, 1] within that list before weighting and summing - this works regardless of each engine's native scale, and needs nothing from the content itself (no date, no metadata), since every engine always produces a query-time relevance score for whatever it returns.

TEXT_ENGINE_WEIGHT = 1 and VECTOR_WEIGHT = 3 are the current per-engine weights (search.service.ts). These are carried over from the old RRF regime's tuning, not re-validated under this fusion - the code's own comment is explicit about this: the eval numbers below measured RRF's weight/(RRF_K+rank) formula, not this fusion's weight*normalizedScore, so they're an untested starting point here, not an assumed-still-optimal value. Re-run the eval loop (lab/alan/search-tuning/) against the current fusion before trusting them:

Vector weight (under the old RRF regime)	hit@1	hit@3
1 (unweighted hybrid)	64.7%	88.2%
2	76.5%	94.1%
3 (carried over as current)	82.4%	94.1%
4	76.5%	94.1% (regressed - an exact-term FAQ query the text engine used to win outright dropped to hit@2)

Under RRF, weight 3 was a genuine local maximum: the point where vector similarity won ties on semantic-mismatch queries (e.g. "backend coding standards" vs. a page titled "Backend Getting Started") while still leaving room for the text engine's own exact-term wins. Whether 3 is still the right weight under normalized-score-sum fusion is an open question - the mechanism is different enough that this needs its own eval run, not an assumption that the old answer carries over.

Rejected approach: a vector similarity floor

A hard cutoff (VECTOR_SIMILARITY_FLOOR = 0.8, filtering out low-confidence vector hits before merging) was tried and reverted (289d2543 → 9c3b5932, recorded in 24cd8da5), back when fusion was still RRF. It did exactly what it was calibrated to do - but was a net regression (hit@1 77.4%/hit@3 86.8%, down from 79.2%/92.5% baseline). Filtering vector's contribution on weak-confidence queries didn't just remove false vector confidence, it removed vector's otherwise-adequate covering signal - which exposed that OpenSearch's own multi_match ranking has real problems on generic or paraphrased queries that the vector weight had been masking, not just compensating for. One example: "can I use this commercially without paying" dropped from hit@1 to rank 5, with an unrelated Wikipedia article outranking the correct FAQ page on pure text relevance.

Lesson, not just history: a hard confidence cutoff on one signal can silently regress overall ranking by removing that signal's coverage value, even when the cutoff is doing precisely what it was tuned to do in isolation. Letting a low-confidence vector hit still contribute something, proportional to its weight, has held up better than a binary include/exclude cutoff - true under RRF, and the underlying reasoning (coverage matters, not just confidence) carries over to the current normalized-score-sum fusion too, even though this specific experiment predates it. The underlying text-engine ranking weakness this experiment exposed has not been separately fixed - it's currently compensated for by the vector weight, not resolved.

Constraint: content.metadata is never a ranking signal

Do not filter, score, or boost search results based on any content.metadata field (e.g. the category field the experiment-wikipedia-ingestion integration writes), in SearchService or any future ranking work.

Why: metadata is a free-form, per-integration bag with no shared schema across integrations - the crawler integration doesn't emit category at all, and nothing enforces that any future integration would. Building ranking logic on a field only some integrations populate would silently do nothing for every other source, so it cannot be a general-purpose scoring signal. metadata exists for other retrieval/display purposes (showing context to the user in the UI), not as ranking input.

Ranking/scoring changes should only use signals present on every result regardless of source: text match strength (ts_rank / OpenSearch _score), vector similarity, and the per-engine fusion weight. A similarity floor or threshold is the kind of change that fits this constraint (see above for why that specific one was still rejected on the evidence); anything keyed off metadata.* does not fit it at all, independent of whether it would otherwise help.

Debug mode

SearchReadManyInput.debug: true adds a debug field to the response containing the exact, already-built query sent to each engine/vector index (not a reconstruction) and a summary of what ran and how it was merged. Off by default - a vector search's raw debug request includes the full query embedding array, which isn't free to build or return, and there's no reason to pay that cost on the normal search path. There is no plan to expose debug in the frontend; it's an API-level tool, used by calling the API directly.

Search summaries

GET /v1/search/summary takes every GET /v1/search parameter plus a required ragConfigId, runs the same SearchService.search(), and has that RAG config's model summarize the results (backend/src/v1/search/search-summary.service.ts). It returns the standard one-result response (operation: 'search.summary') with:

• result.summary - the summary text, or "".
• result.status - a readable outcome: processed from N results, failed from N results or skipped from 0 results.
• result.ragConfig - the RAG config that was used, credentials removed, so the frontend can show which model wrote the summary.

It deliberately does not return the search results. The frontend calls it alongside GET /v1/search (or after it, when the user asks for a summary) and shows those results instead. The two calls run the search separately, so in principle they can see different results if content changes between them; keeping them in sync is left for later. It also doesn't record the query in search analytics - GET /v1/search already does.

A skipped or failed summary is still a successful request (HTTP 200); status is how the caller tells them apart:

• 0 results - the model isn't called at all.
• Provider failure (wrong or revoked key, outage, unknown model, the 120 s timeout) - logged as an error with Failed to summarize search results with RAG config '<id>': <reason>, and returned with an empty summary.
• RAG config not found, or another user's - 404, before any search runs.

The prompt lives in search-summary.prompt.ts: the RAG config's prompt (its personality) followed by fixed summarizing instructions, then the query and the results numbered in rank order, each with its source, date and searchableText (or the content JSON when there is none). Each result is capped at 6,000 characters and the whole set at 60,000 (roughly 15k tokens), dropping the lowest-ranked results first. The summary itself is capped at 1,024 tokens.

Where to re-tune this

lab/alan/search-tuning/ holds the eval harness (run-eval.js, with a --config flag for comparing embedding models/weights) and the full historical record of every tuning iteration (eval-history.jsonl). Any change to TEXT_ENGINE_WEIGHT, VECTOR_WEIGHT, or the embedding model default should go through that loop and get recorded there - and given the fusion mechanism itself changed from RRF to normalized-score-sum fusion, the current weight values specifically need a fresh run before they can be called justified by data rather than carried over from the old regime.