Repository navigation
Add Scout semantic and hybrid search and the Turbopuffer engine - #50
Conversation
Ports laravel/scout #1007, #1008, #1009 and #1012 together, because they share the Builder API, configuration, EngineManager, fixtures and documentation, and #1012 reworks #1008's Meilisearch embedding code. The Builder gains semantic() and hybrid(). Engines that implement SupportsSemanticSearch run them; others reject semantic searches and run hybrid searches as normal text searches. The capability check runs after the Builder preparation callback, so a callback that enables semantic search is checked too. Meilisearch and Typesense support native embedders and precomputed document embeddings. Their query vectors come from the native embedder or the vector option. Typesense sends semantic and hybrid searches through its multi-search endpoint, because a serialized embedding can exceed the query string length. Native hybrid searches also disable prefix search on the embedding field when it is already in query_by; upstream left the default prefix enabled there, which Typesense rejects for remote embedders. The new Turbopuffer engine keeps Hypervel's document and index-settings preparation callbacks and operation reporting. Filtered deletion repeats the request while Turbopuffer reports remaining matches, under one reported operation. Two upstream defects are fixed: - Turbopuffer returns 404 for a namespace that was never written or has been flushed. Searches and pagination now return no results, and deletes and flushes do nothing, instead of failing scout:import --fresh and searches after a flush. - Paginated full-text totals counted every filtered document. They now count only documents containing a query token in a weighted searchable attribute, matching what BM25 ranking returns. Turbopuffer requests are tagged for Telescope, and Scout now requires hypervel/http. Scout cannot generate embeddings yet, so the default embedding driver (hypervel-ai) reports that generation is unavailable, and the database engine does not support semantic search. The Scout documentation, README and porting guide describe this. Upstream reference: laravel/scout 11.x at ce2542f5a7; laravel/docs 13.x at 2bb1a3edca. Validation: the changed test files, the Scout test suite, Meilisearch and Typesense integration tests, formatting and PHPStan pass. Turbopuffer is tested with HTTP fakes.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PR Summary by QodoAdd Scout semantic and hybrid search with a Turbopuffer engine
AI Description
Diagram
High-Level Assessment
Files changed (26)
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if ($builder->semanticSearch | ||
| || $builder->hybridSearch !== null | ||
| || isset($builder->options['rank_by']) | ||
| || $builder->query === '' | ||
| || $builder->query === '*') { | ||
| return $filters; |
There was a problem hiding this comment.
If an index contains documents without a vector in the embedding attribute, this filter-only count includes them even though the semantic ranking query cannot return them. paginate() then reports more results than can be retrieved, leaving users with empty pages.
There was a problem hiding this comment.
Declining. Turbopuffer requires every document in a namespace with vector indexes to include all vector attributes, and precomputed writes add a vector to every record. So the index can't hold documents without a vector for the count to include.
There was a problem hiding this comment.
That’s a valid clarification. Given Turbopuffer’s invariant that every document in a namespace with a vector index must provide the configured vector attribute—and this engine adds precomputed vectors to every record—the filter-only count cannot include documents that semantic search would omit. I’m withdrawing this finding.
Code Review by Qodo
1. Turbopuffer ignores the minimum similarity
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
Turbopuffer pagination rejected any page whose window ended past 10,000 results. The paginator links the last page of a full result set, so with 15 results per page, page 667 holds the last 10 of 10,000 results but ends at 10,005, and the engine threw. Laravel Scout has the same check. Pages are now only rejected when they start past the 10,000th result, so the last page returns its remaining rows. Deeper pages still throw. Filtered deletion requires a filter, but an empty `filters` option was passed through as the filter and skipped that check. Empty native filters now count as no filter, so the deletion throws before any request. Turbopuffer has no similarity threshold, so, as in Laravel Scout, it ignores the `minSimilarity` argument. The Scout documentation now says that Meilisearch and Typesense apply it and Turbopuffer doesn't. Validated with the Turbopuffer engine tests, the Scout test suite, composer lint:fix and composer analyse.
1 participant
This adds Scout's semantic and hybrid search and its Turbopuffer engine from laravel/scout
11.xatce2542f5a7: laravel/scout pull requests 1007, 1008, 1009 and 1012. They're ported together because they share the builder API, configuration, engine manager, test fixtures and documentation, and pull request 1012 reworks 1008's Meilisearch embedding code. The documentation follows laravel/docs13.xat2bb1a3edca.Scout can't generate embeddings yet, so semantic and hybrid search work with each engine's native embeddings or with precomputed vectors. Details are below.
What it adds
semantic()andhybrid()on the search builder.semantic()ranks results by vector similarity, with an optional minimum similarity.hybrid()combines full-text and semantic ranking using the given weights. Engines that support them implementSupportsSemanticSearch. On other engines, a semantic search throwsNotSupportedExceptionand a hybrid search runs as a normal full-text search.hybridsearch parameter with the configured embedder, and the minimum similarity becomesrankingScoreThreshold. With themeilisearchembedding driver, Meilisearch embeds documents and queries itself; Scout adds no document vectors and leaves any_vectorsthe model supplies alone. Otherwise,toSearchableEmbedding()returns a precomputed embedding, which Scout adds to_vectorsunder the embedder's name, and searches pass their query embedding through thevectoroption.typesensedriver. Semantic searches then query only the embedding field, and hybrid searches add it toquery_bywhen it isn't already listed. Otherwise, precomputed embeddings are stored in the configured attribute and searches pass their query embedding through thevectoroption. Any search that sends avector_querygoes through Typesense's multi-search endpoint, because a serialized query embedding can exceed the query string length limit. That covers searches with a query embedding, hybrid weighting or a minimum similarity; a native semantic search without a minimum similarity sends novector_queryand uses the normal search endpoint. Multi-search errors become the same Typesense exceptions as normal search errors, so a missing collection is still created and the search retried.embedsetting, or precomputed. Scout sends the configured schema and distance metric with each write. Turbopuffer has no similarity threshold, so, as upstream, it ignoresminSimilarity; the documentation now says which engines apply it.Hypervel adaptations and fixes
DeletesByFilter. Turbopuffer deletes a limited number of matches per request, so Scout repeats the request while Turbopuffer reports remaining matches, and reports it as one operation. Deletion requires a filter, and an emptyfiltersoption doesn't count as one.hypervel/http.scout:import --freshfailed on a new index and searches failed after a flush. Searches and pagination now return no results, and deletes and flushes do nothing.query_by. Upstream only did this when Scout added the field itself, and Typesense rejects prefix searches on fields with a remote embedder.Not available yet
The default
hypervel-aiembedding driver can't generate embeddings, because Hypervel doesn't have the Laravel AI SDK. It throws when a model'stoSearchableEmbedding()returns text, or when a search needs a query embedding and none is given. The database engine's semantic and hybrid search always generate the query embedding, so the database engine doesn't support them yet: semantic searches throw and hybrid searches fall back to full-text search. The Scout documentation, README and Laravel porting guide describe this.The changed tests, the Scout test suite, the Meilisearch and Typesense integration tests, formatting and static analysis pass locally. Turbopuffer is tested with faked HTTP responses, since there's no Turbopuffer service to test against.
Note
Add Scout semantic/hybrid search and the Turbopuffer engine
Builder::semantic()andBuilder::hybrid()query modes with validated weights and minimum similarity; engines implement the newSupportsSemanticSearchmarker contract. Semantic search on unsupported engines throwsNotSupportedException, while hybrid search falls back to normal full-text search.TurbopufferEnginewith aTurbopufferClient/TurbopufferNamespaceservice layer, driver registration inEngineManager, aturbopufferconfig section, and aTelescopeTag::Turbopuffertag.MeilisearchEngineandTypesenseEnginewith per-model embedding settings, embedding enrichment during indexing, and native or precomputed vector support for semantic and hybrid queries. Typesense vector queries route through the multi-search endpoint with mapped error exceptions.generateEmbeddingsthrowsScoutExceptionand models must supply precomputed vectors or use engine-native embeddings.SupportsSemanticSearchduring builder preparation (seeBuilder::preparedEnginein Builder.php), hybrid weight/similarity validation, and Turbopuffer HTTP 202 responses raisingScoutException.Macroscope summarized 6e77511.