fix(indexer): close PageIndex's local SQLite connection and forward LLM credentials - #250
Open
sebastianbraun25 wants to merge 2 commits into
Conversation
added 2 commits
September 8, 2026 09:05
…LM credentials index_long_document() now closes PageIndex's local SQLite connection(s) in a finally block covering both the success and the failure path (via a new best-effort _close_pageindex_client() helper, reaching into the pinned pageindex version's private LocalBackend/SQLiteStorage since it exposes no public close()/context-manager API). Without this, a subsequent mutation rollback could not unlink/rename pageindex.db on Windows while this process still held it open (WinError 32), leaving a dirty mutation journal and aborting the rest of a multi-file 'openkb add' batch (VectifyAI#249). Also resolves the KB's LlmCredentialBundle (the same LLM_API_KEY/base_url compiler.py's own _llm_call/_llm_call_async use) and forwards it into PageIndex's own internal LLM calls via IndexConfig(llm_params=...) -- PageIndex's pinned version scopes llm_params per-call (pageindex.config.llm_params_scope, context-isolated, safe under concurrent multi-KB use) but had no wiring from OpenKB's side, so a custom LLM_API_KEY/gateway base_url never reached PageIndex's TOC/tree/summary generation calls, which instead fell back to LiteLLM's default provider-key/env-var lookup (VectifyAI#219). Guarded by the same IndexConfig.model_fields check already used for max_concurrency, so it degrades gracefully against an older pinned pageindex.
Header-only gateway auth (litellm.extra_headers, e.g. a proxy Bearer token with no LLM_API_KEY) never reached PageIndex's own LLM calls, only api_key/base_url did (see VectifyAI#219) - PageIndex's internal indexing calls had no credentials at all in that setup and failed with AuthenticationError.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.
Problem
index_long_document()inopenkb/indexer.py(local PageIndex indexing forlong PDFs) has two related robustness problems observed during a real batch
openkb addrun:.openkb/pageindex.db) isnever closed. If a later step in the same
addfails, the mutationrollback in
openkb/mutation.pytries tounlink()/renamepageindex.db/-wal/-shm— on Windows this hitsWinError 32becausethe connection opened earlier in the same process is still holding the
file open, leaving a dirty mutation journal and aborting the rest of the
batch (fix(indexer): close PageIndex's local SQLite connection so rollback doesn't hit WinError 32 #249).
receive the KB's resolved model credentials.
index_long_document()onlyreads
PAGEINDEX_API_KEY(PageIndex Cloud auth) — a customLLM_API_KEY/gatewaybase_url(the same credentialscompiler.py's own_llm_call/_llm_call_asyncuse) never reaches PageIndex, which fallsback to LiteLLM's default provider-key/env-var lookup instead ([Bug] PageIndex long document indexing fails with custom OPENAI_API_BASE / 401 Unauthorized #219).
Root Cause
col = client.collection()opens a WAL-mode SQLite connection viaPageIndex's local backend, and nothing in
index_long_document()evercloses it — the connection lives for the rest of the process.
index_long_document()never callsopenkb.config .resolve_credential_bundle(), so it has noapi_key/base_urlto givePageIndex in the first place. Separately, the pinned
pageindexversiononly accepts LLM credentials for its own calls via a dedicated
IndexConfig(llm_params={...})field (scoped per-call throughpageindex.config.llm_params_scope, context-isolated so it's safe underconcurrent multi-KB use) — not via
PageIndexClient(api_key=...), which isPageIndex Cloud's own auth and unrelated to the underlying model's LLM
credentials in local mode.
Solution / Changes
openkb/indexer.py:_close_pageindex_client()helper: best-effort closesclient._backend._storage(the only place the pinnedpageindexexposesa
close()— there's no public API onPageIndexClient/Collectionitself). A no-op in cloud mode (no local backend/storage). Never raises.
index_long_document()'s whole body now runs inside atry/finallythat calls the helper above — closed on both the success and the failure
path.
index_long_document()now resolvesresolve_credential_bundle(kb_dir)and forwards it into
_build_index_config()._build_index_config()takes an optionalbundleand forwards itsnon-empty
api_key/base_urlasIndexConfig(llm_params={...}), guardedby the same
IndexConfig.model_fieldscheck already used formax_concurrency— degrades gracefully (with a warning) against an olderpinned PageIndex that predates the
llm_paramsfield.tests/test_indexer.py:TestBuildIndexConfigLlmParams:bundle→llm_paramsforwarding(full/partial/empty/
Nonebundle, and the unsupported-version fallback).TestClosePageindexClient: closes the local backend's storage, is ano-op for a client without one, and swallows a
close()exception.TestIndexLongDocument: closes the client on both success and failure,and the resolved credential bundle reaches the real
IndexConfigpassed to
PageIndexClient(not just the isolated_build_index_configunit tests).
Backward compatible: no config/CLI surface changes. A KB with no custom
LLM_API_KEY/base_urlsees nollm_paramsat all (unchanged behavior).Issues
Resolves #249.
Resolves #219.