Conversation
DataFusion 54.1.0 added a `schema` field to `RecursiveQuery`, breaking the struct literal in `PyRecursiveQuery::new()`. Build the node through the new fallible `RecursiveQuery::try_new()` constructor, which reconciles the schema of the static and recursive terms. Pin the workspace requirements to `54.1` so the build cannot resolve back to 54.0.0, which no longer satisfies this code.
|
@timsaucer for context, Ballista Rust crates upgraded to 54.1.0 to bring in a needed correctness fix, but the Ballista Python module could not upgrade due to the breaking API change in DF 54.1.0 |
|
Do you want a 54.1.0 release for |
That's a bit concerning |
It would be nice to have, but I don't think it is a hard requirement for us to release Ballista 54.1.0 since we can leave the Python bindings locked to 54.0.0. So, I wouldn't worry about it unless you get more requests. |
|
Sorry I forgot to merge this and 55.1.0 is now in |
|
Hi @timsaucer I just bumped into this same issue and due to one of our dependencies we need to use datafusion 54.1. Is it possible to backport this to datafusion-python 54.1? Thank you! |
|
@ryansun96 Starting the backport #1772 |
3 participants
Which issue does this PR close?
Closes #1644.
Rationale for this change
The workspace dependencies were declared as
version = "54", so Cargo was freeto resolve them to DataFusion 54.1.0 — but the crate no longer compiled when it
did. DataFusion 54.1.0 added a
schema: DFSchemaReffield todatafusion_expr::logical_plan::RecursiveQuery, andPyRecursiveQuery::new()built that struct with a literal initializer:
What changes are included in this PR?
"54"to"54.1"and updateCargo.lockto 54.1.0. Pinning the minor version keeps the build from silentlyresolving back to 54.0.0, which no longer satisfies the code below.
RecursiveQuerythrough the newRecursiveQuery::try_new()constructorinstead of a struct literal.
try_new()computes the output schema byreconciling the static and recursive terms, so it is fallible;
PyRecursiveQuery::new()now returnsPyDataFusionResult<Self>.test_recursive_queryto round-trip the node back through the Pythonconstructor, covering the new schema-reconciliation path.
Are there any user-facing changes?
datafusion.expr.RecursiveQuery(...)can now raise when the static andrecursive terms do not have the same number of columns. Previously such
arguments produced a
RecursiveQuerynode that was invalid downstream, so thisturns a late failure into an immediate one. No other public API changes.