Native CSV scan reads every partition through the first partition's object store #6747

Describe the bug

The native CSV scan builds one object store for the whole scan, from the first file of the first partition (planner.rs#L2003-L2026). It then reads the current partition's files through that store, keeping only each file's path. A CSV scan over two buckets therefore reads every partition from the first bucket. Unlike the Parquet scan, this also happens when no partition mixes buckets.

Steps to reproduce

MinIO through CometS3TestBase, native CSV enabled (spark.comet.scan.csv.v2.enabled=true and spark.sql.sources.useV1SourceList=""). Two buckets each hold one small CSV file with columns id and bucket ('A' or 'B'). Set spark.sql.files.openCostInBytes=128MB so that each file gets its own partition, then run:

spark.read.schema("id INT, bucket STRING").csv("s3a://repro-a/csv1-same", "s3a://repro-b/csv1-same").collect()

The plan is CometProject <- CometCsvNativeScan. Rows below are sorted after collecting.

• Same key in both buckets (csv1-same/f.csv): partition 0 is s3a://repro-a/csv1-same/f.csv and partition 1 is s3a://repro-b/csv1-same/f.csv. Spark returns [1,A] [1,B] [2,A] [2,B] [3,A] [3,B]. Comet returns [1,A] [1,A] [2,A] [2,A] [3,A] [3,A], with no error.
• Different keys: repro-a/csv1-a/a.csv and repro-b/csv1-b/b.csv fail with [FAILED_READ_FILE.FILE_NOT_EXIST] ... reading file csv1-b/b.csv. Listing the bucket-b path first flips it, and the scan fails on csv1-a/a.csv. Every partition uses the store of whichever file comes first.
• Both buckets in one partition: this fails the same way as the Parquet scan.

Expected behavior

The same rows as Spark, or a fallback to Spark.

Additional context

Reproduced on main at e038cf0 with the default Spark 4.1 profile. Building the store from the current partition's first file, as the Parquet scan does, fixes the separate-partition case. A partition that mixes buckets then has the Parquet scan's problem (#6746). #6059 declines CSV scans whose ABFS files span more than one container or account.