You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Enhancement] Enable mixed partial/final execution for approx_count_distinct (HyperLogLogPlusPlus) #4820
The initial approx_count_distinct support (PR #4819) stores its HyperLogLog++ registers in Spark's exact packed-Long buffer layout (numWordsLong columns, 10 six-bit registers per word), so Comet's partial-aggregation state is byte-identical to Spark's HyperLogLogPlusPlus.aggBufferSchema.
CometApproxCountDistinct currently leaves supportsMixedPartialFinal at the default false, which means that whenever a plan has an approx_count_distinct at a Comet/Spark boundary, allAggsSupportMixedExecution forces both the partial and final aggregate onto the same engine.
Proposal
Because the intermediate buffer format now matches Spark exactly, supportsMixedPartialFinal should be safe to set to true (as CometMin, CometMax, and the bitwise aggregates already do). This would let Comet accelerate the partial aggregate even when the final falls back to Spark (and vice versa), broadening native coverage.
Work required
Set supportsMixedPartialFinal = true in CometApproxCountDistinct.
Add a partial-merge test (see spark/src/test/resources/sql-tests/expressions/aggregate/partial_merge.sql) that exercises Comet-partial + Spark-final and Spark-partial + Comet-final, confirming the result stays bit-identical to Spark.
This was deferred from PR #4819 because it needs interop verification rather than being a trivial one-line change.
It is intentionally draft while #4819 is still open because the branch is stacked on that work. Once #4819 lands, I will rebase/update the PR so reviewers only need to look at the mixed partial/final change for this issue.
Mixed partial/final for approx_count_distinct shipped with #4819 and still holds after #5421: both directions are enabled in CometApproxCountDistinct (serde/aggregates.scala), covered by the two mixed-execution tests in CometExecRuleSuite. I think this can be closed, and #4823 looks superseded.
Activity
michaelmitchell-bit commentedon Jul 4, 2026
I opened draft PR #4823 for this: #4823
It is intentionally draft while #4819 is still open because the branch is stacked on that work. Once #4819 lands, I will rebase/update the PR so reviewers only need to look at the mixed partial/final change for this issue.
dwsmith1983 commentedon Oct 4, 2026
Mixed partial/final for approx_count_distinct shipped with #4819 and still holds after #5421: both directions are enabled in
CometApproxCountDistinct(serde/aggregates.scala), covered by the two mixed-execution tests inCometExecRuleSuite. I think this can be closed, and #4823 looks superseded.