Repository navigation
Pushdown Filter Type BLOOM_FILTER is not currently supported in PyArrow Scans #463
Description
What happens?
Hi,
I am running into this error message and am unsure what to do.
_duckdb.NotImplementedException: Not implemented Error: Pushdown Filter Type BLOOM_FILTER is not currently supported in PyArrow Scans
It seems I can either disable pushdown filtering through joins, or materialize the query before doing my join. I would rather not do either, since it is a lot of data that I am processing.
Reading the source files with a non-pyarrow reader is unfortunately not an option at this time.
To Reproduce
Unfortunately, I don't really have a reproducer, but I think the maintainers will know what the issue is anyways from the error message. It seems to me that Pyarrow do support bloom filter, but the pushdown is just not implemented?
OS:
Ubuntu 22
DuckDB Package Version:
1.5.2
Python Version:
3.11
Full Name:
Jakob Stigenberg
Affiliation:
Qubos Systematic
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
No - I cannot share the data sets because they are confidential
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
Activity
evertlammerts commentedon May 19, 2026
Hey @jakkes, thanks for the report!
There does seem to be a bug in pyarrow_filter_pushdown.cpp, we don't support bloom filter pushdown in pyarrow but instead of just refusing to push down, we throw an exception.
But I can't seem to trigger this path. Bloom filter table filters are only generated when CanUseBloomFilter is true, which depends on cardinality stats being available. But we d not expose those for arrow right now. Me and Claude tried hard to hit the path but for now, no dice.
Please share some more details on what you're doing:
jakkes commentedon May 20, 2026
Hi, thanks for looking into it. I'll do my best to give your more context. I should note (forgot in the description), that I also am having a really hard time reproducing this locally. It triggers consistently when run in our kubernetes environment, but I haven't been able to trigger it locally. So unfortunately, I have a bit of a hard time getting more info about the plan, other than the error message above.
I'll try to extract more info from there though, and come back to you.
jakkes commentedon May 20, 2026
Can I share the query plan with you privately, somehow?