v0.8.44: more search improvements - #7961
Merged
Merged
Conversation
waleedlatif1
commented
Sep 18, 2026
waleedlatif1
commented
Collaborator
- chore(db): drop unused ANN indexes on embedding (chore(db): drop unused ANN indexes on embedding #7955)
- fix(file-search): seek the backfill cursor instead of rescanning each page (fix(file-search): seek the backfill cursor instead of rescanning each page #7956)
- fix(search): preserve connector failure diagnostics (fix(search): preserve connector failure diagnostics #7958)
- improvement(knowledge): instrument the search source overview and bound its probes (improvement(knowledge): instrument the search source overview and bound its probes #7957)
- fix(search): restore submission layout and delay filters (fix(search): restore submission layout and delay filters #7959)
- fix(knowledge): keep vector candidate selection bounded (fix(knowledge): keep vector candidate selection bounded #7960)
* chore(db): drop unused ANN indexes on embedding Approximate retrieval is served by the compact embedding_search projection. The only vector ordering left on embedding is an exact rerank wrapped as (distance) + 0, which the planner cannot match to an index expression, and the binary_quantize expression indexes were never referenced by any query. The last app version that ordered by a bare distance has drained, so these ten indexes were maintained on every chunk write while never being scanned. * fix(knowledge): let the scale benchmark own its ANN index The bulk-seed path required embedding_vector_hnsw_idx to exist, dropped it before seeding and rebuilt it after. With no ANN index left in the schema it threw before measuring anything. The benchmark seeds embedding directly and never populates embedding_search, so it now creates its own HNSW index after the load rather than borrowing a schema one, for both seed paths.
… page (#7956) * fix(file-search): seek the backfill cursor instead of rescanning each page The hourly backfill walks every live workspace file by `(workspace_id, id)`, but no index supplied that order under its predicate, so each page sorted the whole remaining set and the dispatcher's 10s statement timeout aborted the transaction before any page committed. Adds the matching partial index and compares the cursor row-wise. The previous `workspace_id > :ws OR (workspace_id = :ws AND id > :id)` spelling is only ever an index filter, never an index condition, so even with the index each page restarted at the low end and rescanned every page before it. On a prod-shaped fixture (5.5M files, 94k live) a full 95-page walk goes from 786ms to 42ms; the index alone accounts for 786ms -> 267ms and the row-wise cursor for the rest. * refactor(file-search): sharpen the backfill walk test and its docs Bound the walk loop by the expected page count so a cursor regression fails on the first extra page instead of looping; keep the fixture size off a page multiple explicitly; drop a distinct-count assertion the primary key already guarantees. Separate the two coupled causes in the TSDoc so neither reads as a consequence of the other. * fix(file-search): assert the backfill walk still plans as an index seek Addresses both cubic findings. Mark the index `.concurrently()` so a schema reconciliation outside the hand-written migration also builds it without blocking writes, matching how workflow_execution_logs_workspace_activity_idx is declared. Drizzle now emits CREATE INDEX CONCURRENTLY itself; the migration keeps its hand-written wrapper for lock_timeout and replay recovery. The walk test only proved logical pagination, which the OR spelling also satisfies. Record the statements the dispatcher issues and EXPLAIN the exact backfill SELECT, requiring the cursor to appear as a row-wise index condition. Reverting to the OR spelling now fails on the plan itself, not just on the SQL text. That assertion also exposed a third divergence in the fixture: workspace_id was declared NOT NULL where production has it nullable, which let PostgreSQL drop the walk's IS NOT NULL clause and then refuse to match the partial index at all.
…nd its probes (#7957) * improvement(knowledge): instrument the search source overview and bound its probes The source overview backs the Sim Search indexing banner and source filter, but it emitted no telemetry, so a slow request could only be observed as a duration on the route. It now shares the search diagnostics trace, timing availability resolution, the provider read, and each per-batch indexing and searchable probe, and records how many access batches and live source proofs a request consumed. The access batch generator is instrumented in the same trace, separating the connector discovery query from the live source proof that reaches the provider over the network, so a slow batch says which half was slow for every caller. The searchable probe also stops repeating: one searchable document is the whole answer, yet the bounded existence query ran again for every remaining batch. * fix(knowledge): report the free first access predicate separately The batch counter started at one to include the ordinary predicate, which is yielded without a connector query or a live source proof, while the field documented every batch as costing both. The count now says what it measures, and it is recorded before the first yield so a caller that stops consuming there still reports the predicate it used.
* fix(knowledge): keep vector candidate selection bounded An underfilled ANN traversal rescored the whole compact projection and joined the visible document set to it. That fallback is O(corpus): on a 132k-chunk index it took 1.9s, and on a corpus an order of magnitude larger it exceeds the retrieval budget, so the vector leg returned nothing at all rather than fewer rows. Widening the search has no affordable form here. Measured at 10% visibility on the same index, scanning 6.5k tuples instead of 1.5k took 5.1s and 9.7s on consecutive identical runs, and joining visibility before scoring took 8.4s because it turns a sequential scan into a random lookup per document. The bounded traversal itself costs ~115ms whether or not it fills. The traversal is now the whole candidate set. An underfilled one yields fewer candidates and is reported as such, which strictly beats a leg that times out. * fix(knowledge): match the latency plan assertions to the bounded traversal The plan assertion still required the removed CTEs, and the predicates selecting which captured query to assert against matched the old CTE name with a lowercase fallback the rendered statement never produces. The candidate assertions would have stopped running rather than failing, so they now key on the visibility lateral through one shared predicate. The plan check drops the fallback-specific expectations and gains the one that guards this change: the traversal must never reach the projection by document lookup or sequential scan, at any candidate count.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.