Repository navigation
fix: [branch-1.1] fall back from native V2 scans when the plan reads input_file_name (#6754) - #7018
Merged
Conversation
…me (apache#6754) * fix: fall back from native V2 scans when the plan reads input_file_name The native Iceberg and CSV V2 scans do not set InputFileBlockHolder, so input_file_name, input_file_block_start and input_file_block_length returned empty values. Fall back to Spark's reader as the V1 path does. Closes apache#6707 * refactor: reuse readsInputFileBlock in the V2 scan check * test: cover the native CSV V2 scan's input_file_name fallback Document the fallback in the CSV section of datasources.md. * test: move the Iceberg input_file_name test to a SQL fixture Match the full fallback reason there and in the CSV test. Adapted for branch-1.1: CometCsvNativeReadSuite on this branch lacks the timestamp and TIME tests that apache#6754 sits beside, so only the input_file_name test and the CometCsvNativeScanExec import are added. The production change is identical to upstream. (cherry picked from commit 79dd56f)
github-actions
Bot
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6707 on
branch-1.1. #6754 already closed it onmain.Rationale for this change
This backports #6754 to
branch-1.1.The native Iceberg scan is on by default (
spark.comet.scan.icebergNative.enabled=true). It does not setInputFileBlockHolder, which Iceberg's Spark reader sets as it opens each data file. Soinput_file_name(),input_file_block_start()andinput_file_block_length()over an Iceberg table return"",-1and-1for every row, with no error. The native CSV V2 scan has the same problem, but it is off by default.transformV1Scanalready falls back from the native Parquet scan when the plan uses these expressions (#3312), andtransformV2Scanhad no such check.It is a silent wrong-result bug (
priority:critical,correctness) that default settings reach. It is not a 1.1.0 regression, because the native Iceberg scan has had the same gap since it became the default. I am proposing it because of the silent wrong results, in the same way as #6270 and #6700.What changes are included in this PR?
One cherry-pick with
-x. See #6754 for the details.transformV2Scannow takes the whole plan and declines the scan, with a fallback reason, when the plan readsinput_file_name,input_file_block_startorinput_file_block_length. The Iceberg and CSV V2 scans then stay in Spark's reader. TheCometScanRule,iceberg.mdanddatasources.mdchanges are identical tomain.One adaptation, recorded in the commit message:
CometCsvNativeReadSuiteon this branch lacks the timestamp and TIME tests that #6754 adds its CSV test next to, so the cherry-pick conflicted there. The resolution keeps only the newinput_file_nametest and adds theCometCsvNativeScanExecimport it needs.How are these changes tested?
Run locally on this branch with JDK 17, on the default Spark 4.1 profile:
CometCsvNativeReadSuite(4 tests, including the new one) and theiceberg/input_file_name.sqlfixture inCometSqlFileTestSuitepass. Scalastyle and spotless ran in the same build.CometScanRulechange taken out, both new tests fail. The other three pass.By CI's path routing, which I checked with
dev/ci/compute-changes.py, this PR runs the all-profiles Linux build, macOS, the Spark SQL jobs for 3.5, 4.0 and 4.1, and the four Iceberg versions, so norun-*label is needed.