Skip to content

Preserve omitted fields in mixed-field bulk upserts - #874

Open
skulitom wants to merge 1 commit into
simonw:mainfrom
skulitom:codex/fix-upsert-omitted-fields
Open

skulitom wants to merge 1 commit into
simonw:mainfrom
skulitom:codex/fix-upsert-omitted-fields

Conversation

@skulitom

@skulitom skulitom commented Sep 22, 2026 •

Copy link
Copy Markdown

Fixes #873.

upsert_all() can erase existing values when records supply different fields: updating one row's age and another row's name writes NULL into the omitted fields. This also happens across batches because the column list accumulates. Group consecutive records by their supplied fields so each upsert updates only those fields, preserving record order and explicit None values.

Keep the original batch atomic and infer new column types from the whole batch when alter=True. Ordinary inserts, replacement inserts, and positional input keep their existing behavior. Adjacent records with matching fields still share a bulk statement.

Added regression coverage for both upsert implementations, multiple batch sizes, iterator input, trigger-visible update order, explicit nulls and defaults, rollback on invalid keys, hash IDs, compound keys, conversions, and mixed-type schema inference.

Validation on Windows, Python 3.13.2 / SQLite 3.47.1:

  • Full suite: 1,514 passed, 17 skipped.
  • Focused upsert, positional-input, transaction, extraction, and conversion suites with --sqlite-autocommit: 86 passed.
  • Black, Flake8, mypy, Pyright, ty, Cog, and git diff --check passed. The reported failure was reproduced on unpatched main before implementing the fix.

Implemented and tested with OpenAI Codex.


📚 Documentation preview 📚: https://sqlite-utils--874.org.readthedocs.build/en/874/

Group consecutive records by supplied columns while preserving batch atomicity and schema inference. Fixes simonw#873.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

upsert_all() writes NULL to omitted columns in mixed-field batches

1 participant