Repository navigation
DuckDBPyRelation::{order, limit, pl(lazy=True)} can cause panics and silently drop rows with polars #460
Description
Activity
- changed the title
[-]`DuckDBPyRelation::{order, limit, pl(lazy=True)}` can cause panics and silently drop rows[/-][+]`DuckDBPyRelation::{order, limit, pl(lazy=True)}` can cause panics and silently drop rows with polars[/+]on May 15, 2026 - added a commit that references this issue
on May 19, 2026 Thanks for the report, @OutSquareCapital! Fixed in #465
Reacted by Thibaud Wilson StettlerUnfortunately after updating duckdb to 1.5.3 the script above still fail on the last case, with a crash similar.
The
From DataFrame sorted + limitnow correctly return 1 row however!PS C:\Users\tibo\python_codes\pql> uv run f.py ---- Original data ---- From DataFrame: ┌───────┐ │ x │ │ int64 │ ├───────┤ │ 3 │ │ 1 │ │ 2 │ └───────┘ From Python: ┌───────┐ │ x │ │ int32 │ ├───────┤ │ 3 │ │ 1 │ │ 2 │ └───────┘ ---- After transformation ---- From DataFrame sorted + limit -> 0 rows ! ┌───────────────────────────┐ │ TOP_N │ │ ──────────────────── │ │ Top: 1 │ │ │ │ Order By: │ │ df_registered.x ASC │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ ARROW_SCAN │ │ ──────────────────── │ │ Function: ARROW_SCAN │ │ Projections: x │ │ │ │ Filters: │ │ optional: x IS NULL OR │ │ Dynamic Filter (x) │ │ │ │ ~1 row │ └───────────────────────────┘ ┌───────┐ │ x │ │ int64 │ ├───────┤ │ 1 │ └───────┘ -------------------------------------------------- From Python sorted + limit -> OK ! ┌───────────────────────────┐ │ TOP_N │ │ ──────────────────── │ │ Top: 1 │ │ │ │ Order By: │ │unnamed_relation_89e69c4116│ │ 30a91b.x ASC │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ x │ │ │ │ ~1 row │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ UNNEST │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ DUMMY_SCAN │ └───────────────────────────┘ ┌───────┐ │ x │ │ int32 │ ├───────┤ │ 1 │ └───────┘ -------------------------------------------------- Conversion to DataFrame and execution there -> OK ! 0 1 2 ┌──────────────────────────────────────────────────────────────────────────── │ │ ╭─────────╮ 0 │ │ SORT BY │ │ ╰────┬┬───╯ │ ││ │ │╰───────────────────────╮ │ │ │ │ ╭──────┴──────╮ │ │ │ expression: │ ╭───┴────╮ 1 │ │ col("x") │ │ FILTER │ │ ╰─────────────╯ ╰───┬┬───╯ │ ││ │ │╰────────────────────────────╮ │ │ │ │ ╭──────────────┴───────────────╮ ╭──────────┴──────────╮ │ │ predicate: │ │ FROM: │ 2 │ │ col("x").dynamic_predicate() │ │ DF ["x"] │ │ ╰──────────────────────────────╯ │ PROJECT */1 COLUMNS │ │ ╰─────────────────────╯ shape: (1, 1) ┌─────┐ │ x │ │ --- │ │ i64 │ ╞═════╡ │ 1 │ └─────┘ -------------------------------------------------- Conversion to LazyFrame and execution there -> will crash on show ! 0 1 ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────── │ │ ╭─────────╮ 0 │ │ SORT BY │ │ ╰────┬┬───╯ │ ││ │ │╰────────────────────────────────────────────────────╮ │ │ │ │ │ ╭───────────────────────────────────────────┴────────────────────────────────────────────╮ │ ╭──────┴──────╮ │ SORT BY [slice: (0, 1, dynamic_pred: d014c470-2d53-4ec7-94d9-8fb3e42572a4)] [col("x")] │ │ │ expression: │ │ PYTHON SCAN [] │ 1 │ │ col("x") │ │ PROJECT */1 COLUMNS │ │ ╰─────────────╯ │ SELECTION: col("x").dynamic_predicate() │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯ thread 'tokio-runtime-worker' (18212) panicked at crates\polars-plan\src\plans\conversion\dsl_to_ir\expr_to_ir.rs:639:13: internal error: entered unreachable code note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace --- PyO3 is resuming a panic after fetching a PanicException from Python. --- Python stack trace below: Traceback (most recent call last): File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\duckdb\polars_io.py", line 307, in source_generator yield pl.from_arrow(record_batch).filter(predicate) # type: ignore[arg-type,misc,unused-ignore] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\dataframe\frame.py", line 5509, in filter .collect(optimizations=QueryOptFlags._eager()) ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\_utils\deprecation.py", line 97, in wrapper return function(*args, **kwargs) File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\lazyframe\opt_flags.py", line 343, in wrapper return function(*args, **kwargs) File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\lazyframe\frame.py", line 2510, in collect return wrap_df(ldf.collect(engine, callback)) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ pyo3_runtime.PanicException: internal error: entered unreachable code thread 'async-executor-7' (7972) panicked at crates\polars-stream\src\nodes\io_sources\batch.rs:107:18: called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(9), "internal error: entered unreachable code", ...) Traceback (most recent call last): main() ~~~~^^ File "C:\Users\tibo\python_codes\pql\f.py", line 20, in main _test_pl(from_df, lazy=True) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tibo\python_codes\pql\f.py", line 33, in _test_pl lf.show() ~~~~~~~^^ File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\lazyframe\frame.py", line 9680, in show self.head(limit).collect(engine="streaming").show( ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\_utils\deprecation.py", line 97, in wrapper return function(*args, **kwargs) File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\lazyframe\opt_flags.py", line 343, in wrapper return function(*args, **kwargs) File "C:\Users\tibo\python_codes\pql\.venv\Lib\site-packages\polars\lazyframe\frame.py", line 2510, in collect return wrap_df(ldf.collect(engine, callback)) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(9), "internal error: entered unreachable code", ...)
Reacted by Denis LussonAh, that's interesting, this only crashes with Polars >= 1.39. Before that, Polars' planner doesn't push a dynamic predicate.
They translate like this:
{ "Display": { "inputs": [{"Column": "x"}], "fmt_str": "dynamic_pred: df47c229-809d-4474-9909-f6b560e3beff" } }We should just skip them - there's not an actual expression in there we can apply, afaict.
Would it be possible to reopen the issue, since this is still causing panics with
Polars >= 1.39.If you'd prefer, I'm also happy to open a new issue instead specifically for
Polars >= 1.39.- added a commit that references this issue
on Jun 9, 2026 This should be fixed by #482.
Reacted by Denis Lusson
What happens?
The interaction between polars and duckdb can lead to unexpected results and panics.
See the example below. Cause the same issues on
1.5.2and1.5.3.dev24.my current project aims to make polars <-> duckdb interops a first class citizen.
Thus, I would be very keen to help with the polars_io module, if some documentation can be found somewhere.
To Reproduce
Minimal repro:
Output:
OS:
Windows
DuckDB Package Version:
v1.5.3.dev24
Python Version:
3.13.7
Full Name:
Stettler Thibaud
Affiliation:
None
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a nightly build
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
Did you include all relevant configuration to reproduce the issue?