feat: make native Iceberg writes opt-in for 1.2.0

[andygrove]

Which issue does this PR close?

Part of #5644.

Rationale for this change

#6664 turned spark.comet.write.iceberg.enabled on by default. This PR ships 1.2.0 with the native
Iceberg writer as a supported, opt-in feature instead, and leaves turning the default on until after
the 1.2.0 branch is cut.

• The native writer merged on 2026-09-02 and shipped in 1.1.0 only under a testing-category setting,
  so 1.2.0 is its first release as a user-facing feature. Turning it on by default there skips the
  opt-in stage. The native Iceberg scan was opt-in from 0.12.0 to 0.15.0 and became the default in
  0.16.0.
• The write path has had a steady stream of bug reports since it merged, two of them critical
  (Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138, Native Iceberg write silently drops S3 settings it cannot honour (credentials provider, SSE, ACL, tags, remote signing) instead of falling back #6139). Recent ones (Native Iceberg writer counts NaNs under NULL structs and NULL list or map entries #6562, Native fanout Iceberg writes fail when their open partitions outgrow the memory pool #6771) are the kind that the still-open "manifest metrics
  parity per data type" item in Enable the split-operator plan and native Iceberg writes by default #5644 exists to catch. A write bug leaves bad files or metrics in
  tables that every other engine reads, so the bar for a default is higher than for a scan.
• Nothing measures native writes on object storage yet. The benchmark numbers in Enable the split-operator plan and native Iceberg writes by default #5644 are from a
  local filesystem, no test asserts a native write against MinIO, and the native path reads each
  written file's footer back with two sequential requests (Hand natively written Iceberg files' Parquet footers to the JVM instead of reading them back #6772).
• With the default on, an unpartitioned or clustered write whose open file outgrows the off-heap
  pool fails its task, where Comet 1.1.0 wrote it with iceberg-java on the JVM heap.
• Four of the five "Before 1.2.0 ships" items in Enable the split-operator plan and native Iceberg writes by default #5644 are open, and its count of ten nightly runs
  starts with the 2026-10-14 nightly. The 1.2.0 milestone is due on 2026-10-23.

Test coverage does not change. Every Iceberg Spark test diff under dev/diffs/iceberg/ sets
spark.comet.write.iceberg.enabled=true itself, and both Iceberg write benchmarks set it for each
arm, so the Iceberg CI jobs, and the nightly evidence #5644 asks for, run the native writer as
before.

What changes are included in this PR?

• spark.comet.write.iceberg.enabled defaults to false. Everything else from feat: enable native Iceberg writes by default #6664 stays: the
  setting is in the query execution category, it alone plans the split operator and the native
  writer, merge-on-read writes keep Spark's operator, and spark.comet.exec.enabled=false keeps
  Spark's write operator.
• CometIcebergWriteActionSuite: the test of what an application that sets neither write flag gets
  now expects Spark's own write operator, and a new test checks that the write flag alone writes an
  eligible write natively under the split plan. The merge-on-read and transition-reversion tests
  that relied on the default now set the write flag themselves.
• Docs: the user guide's Iceberg writes, Iceberg, data sources and operators pages, the FAQ, the
  Gluten comparison, the contributor guide, the roadmap and the Iceberg write review skill describe
  the native writer as opt-in. The 1.2.0 upgrade-guide entry now covers the move from 1.1.0's
  testing settings to the one supported setting, and points to the user guide for what turning it
  on costs.

Once the 1.2.0 branch is cut, the default can go back to true on main, so that it runs as the
default for a full release cycle before 1.3.0.

How are these changes tested?

The updated and new tests in CometIcebergWriteActionSuite cover the default and the single
switch. Locally, on the default Spark 4.1 profile, CometConfSuite, CometIcebergWriteActionSuite,
CometIcebergWriteDetectionSuite, CometIcebergSystemFunctionSuite,
CometIcebergRewriteActionSuite and CometIcebergNativeSuite pass: 358 passed, 0 failed, and 1
canceled, the SPARK-55626 test that Spark 4.1+ skips.

The run-iceberg-tests and run-all-spark-profiles labels run the Iceberg Spark tests, which set
the write setting themselves, and the Comet suites on the other Spark profiles, where the
merge-on-read test needs Spark 3.5+.

[sunchao]

Reviewed all 13 changed files in the full diff from 55ba50d45137781f50ec1b7323da7109e66f36f9 to 25b5dba6dd9302c6a489ff5b574eb3b7ff6ea8cf. The PR is not a draft. The snapshot and live discussion contain no existing reviews, conversation comments, or inline comments.

Routed skills: review-comet-pr and review-comet-iceberg-write-pr, following AGENTS.md.

Summary

• Prior state and problem: Main enabled native Iceberg writes by default after #6664. This PR restores an opt-in rollout for 1.2.0 while retaining the supported write setting.
• Design approach: Change spark.comet.write.iceberg.enabled to default to false, keeping the existing planning and conversion gates.
• Correctness: With both write flags unset, IcebergWriteStrategy returns Nil, leaving Spark responsible for writing, committing, and aborting. I checked Spark’s DataSourceV2Strategy, V2TableWriteExec, and memoized V2CommandExec execution. The changed tests verify the default plan, explicit native engagement, one committed snapshot, resulting rows, and transition reversion with AQE both enabled and disabled. No introduced P1/P2 correctness issue was identified.
• Compatibility analysis: Compared the relevant Spark sources at 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0. Compared with the latest release branch, branch-1.1, the default again selects Spark’s writer. The migration guide accurately distinguishes that unchanged default from the previously introduced configuration rename and single-switch opt-in.
• Key design decisions: Explicit opt-in still enables both the split plan and eligible native writes. The testing split flag retains its independent effect. Merge-on-read stays on Spark’s operator unless the testing split flag is enabled, and the global Comet execution guards remain intact.
• Implementation sketch: The production change is .createWithDefault(false) and its description. Tests now unset both flags for the default case, explicitly enable native writing for opt-in cases, and preserve the suite’s configuration afterward. Documentation consistently describes the opt-in behavior.
• Performance: The default deliberately forgoes native data-file writing relative to main. Explicitly enabled workloads retain the same implementation. No new execution overhead was introduced. Iceberg test diffs and both write benchmarks explicitly configure the flag, preserving their intended coverage. No throughput improvement is claimed.
• Design: Reusing the existing switch is a simple, coherent way to stage the rollout. The eligibility rules and write lifecycle need no additional branching.
• Abstraction & complexity: No new abstraction is introduced. The tests reuse the existing configuration and plan-capture helpers, and distinguish the default from explicit opt-in directly.
• Behavioral changes worth calling out: Applications setting neither flag return to Spark’s combined write operator. Applications retaining the testing split flag still get the split plan. The documentation preserves the native writer’s footer-read and memory-cost explanations for users enabling it.
• Suggested improvements: No additional change meeting the P1/P2 reporting bar was identified for this PR.

Exact-head CI: 85 checks succeeded and 77 were skipped, with no failed or pending check runs. The default build, additional Spark profiles, and Iceberg matrix succeeded. I inspected test logs and confirmed the default, opt-in, and AQE reversion cases passed on Spark 3.4, 3.5, 4.0, and 4.1. The merge-on-read case passed on 3.5–4.1 and was intentionally canceled on 3.4.

Validation limits: Spark 4.2’s Iceberg cases were canceled because Iceberg was unavailable on its classpath. Stock Spark SQL suites and macOS CI were skipped. I did not run a local JVM/native build, runtime reproduction, or object-storage benchmark. Runtime evidence comes from the exact-head CI logs.

No introduced P1/P2 issues found within this review.