feat: add native Delta Lake scan contrib module (page/row-group pruning)

.github/workflows/README.md

@@ -24,9 +24,9 @@ ruleset in `.asf.yaml`. That splits CI into three tiers:
   and one Iceberg version, both the default profile's.
 - **Nightly tier** (`nightly`): the regression sweep of everything else, once
   a day against `main` as it stands. The Comet test suites against the other
-  four Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, and Iceberg
-  1.8/1.9/1.10. See [Nightly tier](#nightly-tier) below for how a failure
-  surfaces.
+  four Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg
+  1.8/1.9/1.10, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See
+  [Nightly tier](#nightly-tier) below for how a failure surfaces.
 
 Every queue-only and nightly job has a `run-*` label that opts a pull request
 into it early, listed in the diagram below. The Lint Java matrix compiles
@@ -129,6 +129,9 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
                                       iceberg_1_8         run-iceberg-tests
                                       iceberg_1_9         run-iceberg-tests
                                       iceberg_1_10        run-iceberg-tests
+                                      delta_3_5           run-delta-tests
+                                      delta_4_0           run-delta-tests
+                                      delta_4_1           run-delta-tests
 
         |                                   |                                   |
         +-----------------------------------+-----------------------------------+
@@ -165,6 +168,9 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
 | `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check`                                                                                                                                                                                                      | benchmark sources only              |
 | `delta_build_gate`   | merge group, **or** PR with `run-delta-build-gate`                                                                                                                                                                                                     | main sources, poms, `contrib/delta` |
 | `pyarrow_udf_test`   | merge group, **or** PR with `run-pyarrow-udf-tests`                                                                                                                                                                                                    | map-in-batch and Python runner code |
+| `delta_3_5`          | nightly, **or** PR with `run-delta-tests`                                                                                                                                                                                                              | Delta contrib and native sources    |
+| `delta_4_0`          | nightly, **or** PR with `run-delta-tests`                                                                                                                                                                                                              | Delta contrib and native sources    |
+| `delta_4_1`          | nightly, **or** PR with `run-delta-tests`                                                                                                                                                                                                              | Delta contrib and native sources    |
 | `docs`               | push to main, paths matched, **or** dispatch on `main`                                                                                                                                                                                                 | `.asf.yaml`, `docs/**`, `docs.yaml` |
 | `spark_3_5`          | nightly, **or** PR with `run-spark-3.5-tests`                                                                                                                                                                                                          | Spark 3.5 sources                   |
 | `spark_4_1`          | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests`                                                                                                                                         | Spark 4.1 sources                   |
@@ -303,6 +309,7 @@ umbrella doesn't watch, or operate independently of the rest of CI:
 | `pr_build_macos.yml`              | `pr_build_macos`                                                |
 | `pr_benchmark_check.yml`          | `pr_benchmark_check`                                            |
 | `delta_build_gate.yml`            | `delta_build_gate`                                              |
+| `delta_contrib_test.yml`          | `delta_3_5`, `delta_4_0`, `delta_4_1`                           |
 | `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                              |
 | `docs.yaml`                       | `docs`                                                          |
 | `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` |

.github/workflows/ci.yml

@@ -263,6 +263,9 @@ jobs:
       iceberg_1_9:  ${{ steps.compute.outputs.iceberg_1_9 }}
       iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
       iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }}
+      delta_3_5:    ${{ steps.compute.outputs.delta_3_5 }}
+      delta_4_0:    ${{ steps.compute.outputs.delta_4_0 }}
+      delta_4_1:    ${{ steps.compute.outputs.delta_4_1 }}
     steps:
       - uses: actions/checkout@v7
         with:
@@ -469,6 +472,43 @@ jobs:
       spark-full: '3.5.9'
       java: 17
 
+  delta_3_5:
+    name: Delta Contrib Tests (Spark 3.5)
+    needs: changes
+    permissions:
+      contents: read
+    # Nightly by default; PRs need the `run-delta-tests` label.
+    if: needs.changes.outputs.delta_3_5 == 'true'
+    uses: ./.github/workflows/delta_contrib_test.yml
+    with:
+      spark: '3.5'
+      java: '17'
+
+  delta_4_0:
+    name: Delta Contrib Tests (Spark 4.0)
+    needs: changes
+    permissions:
+      contents: read
+    # Nightly by default; PRs need the `run-delta-tests` label.
+    if: needs.changes.outputs.delta_4_0 == 'true'
+    uses: ./.github/workflows/delta_contrib_test.yml
+    with:
+      spark: '4.0'
+      java: '17'
+
+  delta_4_1:
+    name: Delta Contrib Tests (Spark 4.1)
+    needs: changes
+    permissions:
+      contents: read
+    # Nightly by default; PRs need the `run-delta-tests` label. No Spark 4.2
+    # caller: the contrib profile is dormant until a Delta release supports it.
+    if: needs.changes.outputs.delta_4_1 == 'true'
+    uses: ./.github/workflows/delta_contrib_test.yml
+    with:
+      spark: '4.1'
+      java: '17'
+
   spark_4_0:
     name: Spark SQL Tests (Spark 4.0)
     needs: changes
@@ -616,6 +656,9 @@ jobs:
       - pyarrow_udf_test
       - spark_3_4
       - spark_3_5
+      - delta_3_5
+      - delta_4_0
+      - delta_4_1
       - spark_4_0
       - spark_4_1
       - spark_4_2
@@ -684,7 +727,7 @@ jobs:
               failed.length ? 'Jobs that did not succeed:' : 'No individual job reported a failure; see the run for details.',
               ...failed,
               '',
-              'The nightly tier runs the Spark and Iceberg versions the merge queue does not',
+              'The nightly tier runs the Spark, Iceberg and Delta versions the merge queue does not',
               '(see `POLICY` in `dev/ci/compute-changes.py`). Close this issue once the failure',
               'is understood; the next red nightly opens a new one.',
             ].join('\n');

.github/workflows/delta_build_gate.yml

@@ -15,12 +15,15 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# Build-gate verification for the optional contrib/delta integration.
+# Build-gate verification for the Delta split of the native library and the JVM build.
 #
 # A single cheap job that runs dev/verify-contrib-delta-gate.sh, which proves:
-#   - the DEFAULT cargo / mvn / dylib build carries ZERO Delta surface
-#     (no comet-contrib-delta / delta_kernel in the cargo tree, no io.delta in
-#     the effective pom, no Delta symbols in libcomet); and
+#   - the kernel-backed `comet-contrib-delta` crate stays out of every shipped
+#     build (no comet-contrib-delta / delta_kernel in the default or
+#     --no-default-features cargo tree, no io.delta in the effective pom, no
+#     contrib symbols in libcomet), while the small default-on `delta` feature
+#     (deletion-vector decoding) is deliberately in and its symbol footprint is
+#     pinned; and
 #   - the gated build (`--features contrib-delta` / `-Pcontrib-delta`) compiles
 #     and pulls the right delta-spark per Spark profile.
 #