Repository navigation
feat(spanner): event queue optimization, property graph AST, hybrid RRF search, and ADK artifact store - #834
Merged
Merged
Conversation
…and expand queue tests
… search, and graph DDL
…, and ADK enhancements
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #834 +/- ##
==========================================
+ Coverage 81.72% 81.76% +0.03%
==========================================
Files 521 522 +1
Lines 79815 80462 +647
Branches 11534 11628 +94
==========================================
+ Hits 65228 65786 +558
- Misses 10862 10919 +57
- Partials 3725 3757 +32
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimizes Cloud Spanner durable event queues (
sqlspec.extensions.eventsandsqlspec.adapters.spanner.events), extends thespannerandspangresSQLGlot dialects with Spanner Property Graph (CREATE/DROP PROPERTY GRAPH,GRAPH_TABLE) and Vector/Search Index AST support, and upgrades the Spanner ADK stores with batchedUNNESTdeduplication, 4-mode Hybrid Reciprocal Rank Fusion (RRF) + Vector + Full-Text Search, Property Graph DDL overlays, andSpannerSyncADKArtifactStore/SpannerAsyncADKArtifactStore.Key Changes
1. Event Queue Runtime Hints & Transaction Lifecycle (
sqlspec/extensions/events/,sqlspec/config.py)EventRuntimeHints: Addedcleanup_on_ack: bool = Trueanduse_run_in_transaction: bool = False._queue.py): Added_provide_session(read_only: bool = False)onSyncTableEventQueueandAsyncTableEventQueueso unlocked candidate reads (_fetch_candidatewhenselect_for_update=False) requestprovide_session(transaction=False)when supported by the config. On Spanner, this uses a snapshot read instead of checking out a read-write transaction and issuing an emptyCommitRPC on every poll._queue.py): Routed write operations (_execute,publish_many) throughdriver.run_in_transaction(...)whenuse_run_in_transaction=Trueso Spanner read-write transactions automatically retry on transientAbortedconflicts.DELETEonack(): Guarded the post-ackDELETE FROM {table} WHERE status = 'acked' AND acknowledged_at <= :cutoffquery behindif self._hints.cleanup_on_ack:so backends with native row TTL (ROW DELETION POLICYon Spanner) skip unindexed full-table scan deletes on every acknowledgement.EventsConfig,BaseEventQueueStore): Addedshard_count: int = 1toEventsConfigand_shard_countvalidation toBaseEventQueueStore.2. Spanner Event Queue Store (
sqlspec/adapters/spanner/events/store.py,sqlspec/adapters/spanner/config.py)_DEFAULT_SESSION_TRANSACTION: ClassVar[bool] = TrueonSpannerSyncConfigandSpannerAsyncConfigsoprovide_session(transaction=False)cleanly selects snapshot reads while preserving default read-write transactions whentransaction=None.SpannerSyncConfig._get_event_runtime_hintsandSpannerAsyncConfig._get_event_runtime_hintswithcleanup_on_ack=Falseanduse_run_in_transaction=True.shard_idhash-shard prefixing (MOD(FARM_FINGERPRINT(event_id), {shard_count})in GoogleSQL;MOD(ABS(HASHTEXT(event_id)), {shard_count})in PostgreSQL-dialectspangres) whenshard_count > 1, and addedSTORING (payload_json, metadata_json, attempts, leased_until, created_at)(INCLUDE (...)inspangres) to the queue index to avoid base-table back-joins during dequeue polling.spangres) DDL Support: Added fullspangresDDL generation (VARCHAR,JSONB,BIGINT,TIMESTAMPTZ,TTL INTERVAL 'N days' ON acknowledged_at, andINCLUDEcovering indexes) alongside GoogleSQL DDL (STRING,JSON,INT64,TIMESTAMP,ROW DELETION POLICY,STORING).create_table()anddrop_table()onSpannerSyncEventQueueStoreandSpannerAsyncEventQueueStoreto route DDL throughdatabase.update_ddl(...)instead of DMLexecute_script().3. Spanner & Spangres SQLGlot Dialects (
sqlspec/dialects/spanner/)_expressions.py,_parsers.py,_generators.py): AddedSpannerPropertyGraphandSpannerGraphTableAST nodes,graph_table()builder helper, and parser/generator support forCREATE [OR REPLACE] PROPERTY GRAPH [IF NOT EXISTS] ... NODE TABLES (...) EDGE TABLES (...),DROP PROPERTY GRAPH [IF EXISTS], andGRAPH_TABLE (graph_name MATCH ... COLUMNS (...)).IF NOT EXISTSandSTORING (...)support onCREATE VECTOR INDEX, addedDROP VECTOR INDEX [IF EXISTS]andDROP SEARCH INDEX [IF EXISTS]viaSpannerDropIndex, and addedHIDDENcolumn constraint parsing/generation forTOKENLIST AS (TOKENIZE_FULLTEXT(...)) HIDDENgenerated columns (including compatibility with compiledsqlglot[c]).approx_cosine_distance(column, vector, options=...)to emitoptions => JSON '...'viaexp.Kwarg, and addedsnippet()andscore_ngrams()expression builders.SPANNER_TRANSFORMS.update(...)andSPANGRES_TRANSFORMS.update(...)in_generators.pywith loop-based registration so module re-imports never overwrite custom Spanner expression transforms with base SQLGlot transforms.4. Spanner ADK Memory & Artifact Stores (
sqlspec/adapters/spanner/adk/,docs/)UNNESTDeduplication (store.py): Replaced the N+1 per-entry_event_existssnapshot read loop inSpannerSyncADKMemoryStore._insert_memory_entrieswith in-memory batch deduplication plus a singleSELECT event_id FROM {table} WHERE event_id IN UNNEST(@event_ids)query andtransaction.batch_update(...)inside onerun_in_transactioncall, backed byCREATE INDEX idx_{table}_event_id ON {table}(event_id).store.py): Added optionalembedding ARRAY<FLOAT32>(vector_length=>N)column andCREATE VECTOR INDEXDDL (vector_index_enabled,vector_dimensions,vector_distance_type,vector_tree_depth,vector_num_leaves), upgraded FTS mode to rank bySCORE(content_tokens, @query) DESC, and implemented 4-modesearch_entries:enable_hybrid_search=True+use_fts=True+embeddingprovided): CombinesAPPROX_COSINE_DISTANCEvector similarity andSEARCH/SCORElexical search via Reciprocal Rank Fusion (1.0 / (@rrf_k + rank)).embeddingprovided): Orders byCOSINE_DISTANCE/APPROX_COSINE_DISTANCE.use_fts=True): Filters withSEARCH(content_tokens, @query)and orders bySCORE(content_tokens, @query) DESC.LOWER(content_text) LIKEfallback.store.py): Added optionalCREATE PROPERTY GRAPH/DROP PROPERTY GRAPHDDL generation (enable_memory_graph,memory_graph_name,memory_nodes_table,memory_edges_table) and ordered_spanner_drop_statement_tableso dependent property graphs and indexes are dropped before base tables.artifact_store.py): AddedSpannerSyncADKArtifactStoreandSpannerAsyncADKArtifactStoreimplementingBaseSyncADKArtifactStoreandBaseAsyncADKArtifactStorewith composite primary key(app_name, user_id, session_id, filename, version),session_id = ""normalization for user-scoped artifacts, and optionalROW DELETION POLICYTTL cleanup.docs/usage/adapters/spanner.rstanddocs/usage/driver_Patterns/events.rstwith Spanner event queue sharding, Property Graph AST, Hybrid RRF search, and ADK Artifact Store usage.Verification
891 passedacrosstests/unit/extensions/test_events/,tests/unit/adapters/test_spanner/,tests/unit/dialects/test_spanner*.py, andtests/unit/dialects/test_spangres*.py.ruff check,mypy,pyright, andslotscheckpass with0 errors.sphinx-build -M html docs docs/_build/ -W --keep-goingpasses with0 warnings.