DuckDBBaseConverter (0.2.18) is the answer for a source too large to hold in memory, but only if that source is already Parquet: read_parquet is the only reader in vecorel_cli/conversion/duckdb.py. Sources that are large and delivered as GDAL formats still need a different converter, which is why fiboa-cli keeps a PerFileBaseConverter of its own (fiboa/cli#281) for Spain's FEGA release: 52 provincial GeoPackages, 17.9M fields, converted file by file and merged.
DuckDB can already read them. Against one of those GeoPackages:
INSTALL spatial; LOAD spatial;
SELECT count(*) FROM st_read('1501_ALAVA_cd_2025.gpkg', layer = 'cultivo_declarado');
-- the declared columns come back typed, and the geometry with its CRS:
-- dn_oid BIGINT, provincia INTEGER, ..., dn_geom GEOMETRY('EPSG:4258')
SELECT st_astext(st_centroid(dn_geom)) FROM st_read(...) LIMIT 1;
-- POINT (-3.247698939136412 42.88831906019736)st_read_meta() lists the layers of a file, so layer_filter has an equivalent too.
What would have to exist
- A non-Parquet source path:
read_parquet(...) or st_read(..., layer = ?) chosen by the source, with the same union_by_name behaviour across files. The rest of the converter — the column selection, the KV metadata, the GeoParquet post-processing, the Hilbert sort — does not care where the rows came from.
- A SQL form for the mapping tables. This is the harder half. fiboa's converters get their harmonised crop columns from a mixin that joins a CSV code list onto the frame in
post_migrate, and the DuckDB path never builds a frame. read_csv(...) plus a LEFT JOIN in the generated statement would do it, but it needs a declarative shape a converter can express — something like a joins = {...} alongside columns.
With both, es moves to SQL and the per-file converter in fiboa-cli goes away. With only the first, a source whose mapping is pure column arithmetic already benefits.
Happy to do the work if the direction is right — tell me whether you would rather have (1) alone first.
DuckDBBaseConverter(0.2.18) is the answer for a source too large to hold in memory, but only if that source is already Parquet:read_parquetis the only reader invecorel_cli/conversion/duckdb.py. Sources that are large and delivered as GDAL formats still need a different converter, which is why fiboa-cli keeps aPerFileBaseConverterof its own (fiboa/cli#281) for Spain's FEGA release: 52 provincial GeoPackages, 17.9M fields, converted file by file and merged.DuckDB can already read them. Against one of those GeoPackages:
st_read_meta()lists the layers of a file, solayer_filterhas an equivalent too.What would have to exist
read_parquet(...)orst_read(..., layer = ?)chosen by the source, with the sameunion_by_namebehaviour across files. The rest of the converter — the column selection, the KV metadata, the GeoParquet post-processing, the Hilbert sort — does not care where the rows came from.post_migrate, and the DuckDB path never builds a frame.read_csv(...)plus aLEFT JOINin the generated statement would do it, but it needs a declarative shape a converter can express — something like ajoins = {...}alongsidecolumns.With both,
esmoves to SQL and the per-file converter in fiboa-cli goes away. With only the first, a source whose mapping is pure column arithmetic already benefits.Happy to do the work if the direction is right — tell me whether you would rather have (1) alone first.