You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
merge_parquet dropped a property that each part kept as a constant in its collection when the parts disagreed on it: it was neither a column nor in the merged collection. Now it becomes a column again, with each part's value, as vec merge does by reading with hydrate=True. Found in fiboa/cli#338, where the Spanish province code disappeared from the merged file.
Only what the merged collection does not carry is hydrated; a constant all parts share stays in the collection. Collection-only properties (title, license, ...) are never hydrated.
A part that lacks the property gets NULL; a part that already has it as a column keeps its column.
With something to hydrate, the source is one SELECT per part joined with UNION ALL BY NAME; otherwise the query is unchanged.
* Fix merge gaps, merge with DuckDB in vec merge
* Merge: hydrate only what the parts disagree on (review #1)
* Merge: keep rows as they are with DuckDB, like in memory (review #2)
* Report constants that don't fit their schema type (review #3)
* Merge: reject two versions of one schema in a collection (review #4)
* DuckDB: key ids by collection only with several collections (review #5)
* Merge: apply excludes to the loaded data in memory (review #6)
* DuckDB: quote required property names with _sql_name (review #7)
* Share the CRS comparison of merge and DuckDB (review #8)
* Merge: warn when includes drop a required collection-only property (review #9)
* Merge: accept a collection that can't be determined (review #10)
* Add strict mode to vec merge, minor improvements and bug fixes (#60)
Co-authored-by: Matthias Mohr <m.mohr@moregeo.it>
Co-authored-by: Ivor <ivorbosloper@gmail.com>
m-mohr
changed the title
merge_parquet: hydrate the constants the parts disagree on
Reimplement vec mergeSep 28, 2026
Use atomic replacement for DuckDB output to existing targets
vecorel_cli/merge.py:133
The new automatic DuckDB path regresses merges to an existing target: DuckDB COPY ... TO refuses an existing file by default, while the previous GeoParquet.write path overwrote it. This also breaks using one input as the output path. Write to a temporary sibling and atomically replace the target after success rather than unlinking first, since the target may still be a source.
Validate required collection metadata before skipping properties
vecorel_cli/validation/geoparquet.py:167
Required collection-only properties are skipped without checking the collection metadata. For example, a non-strict merge that removes a differing required producer leaves its requirement in schemas:custom, but vec validate reports no error here. Check that the metadata contains a non-null value before continuing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge_parquetdropped a property that each part kept as a constant in its collection when the parts disagreed on it: it was neither a column nor in the merged collection. Now it becomes a column again, with each part's value, asvec mergedoes by reading withhydrate=True. Found in fiboa/cli#338, where the Spanish province code disappeared from the merged file.NULL; a part that already has it as a column keeps its column.SELECTper part joined withUNION ALL BY NAME; otherwise the query is unchanged.🤖 Generated with Claude Code