Skip to content

[Enhancement] Enable mixed partial/final execution for approx_count_distinct (HyperLogLogPlusPlus) #4820

Open

Description

@andygrove

Background

The initial approx_count_distinct support (PR #4819) stores its HyperLogLog++ registers in Spark's exact packed-Long buffer layout (numWords Long columns, 10 six-bit registers per word), so Comet's partial-aggregation state is byte-identical to Spark's HyperLogLogPlusPlus.aggBufferSchema.

CometApproxCountDistinct currently leaves supportsMixedPartialFinal at the default false, which means that whenever a plan has an approx_count_distinct at a Comet/Spark boundary, allAggsSupportMixedExecution forces both the partial and final aggregate onto the same engine.

Proposal

Because the intermediate buffer format now matches Spark exactly, supportsMixedPartialFinal should be safe to set to true (as CometMin, CometMax, and the bitwise aggregates already do). This would let Comet accelerate the partial aggregate even when the final falls back to Spark (and vice versa), broadening native coverage.

Work required

  • Set supportsMixedPartialFinal = true in CometApproxCountDistinct.
  • Add a partial-merge test (see spark/src/test/resources/sql-tests/expressions/aggregate/partial_merge.sql) that exercises Comet-partial + Spark-final and Spark-partial + Comet-final, confirming the result stays bit-identical to Spark.

This was deferred from PR #4819 because it needs interop verification rather than being a trivial one-line change.

Activity

  1. michaelmitchell-bit commented on Jul 4, 2026

    @michaelmitchell-bit
    Contributor

    I opened draft PR #4823 for this: #4823

    It is intentionally draft while #4819 is still open because the branch is stacked on that work. Once #4819 lands, I will rebase/update the PR so reviewers only need to look at the mixed partial/final change for this issue.

  2. dwsmith1983 commented on Oct 4, 2026

    @dwsmith1983
    Contributor

    Mixed partial/final for approx_count_distinct shipped with #4819 and still holds after #5421: both directions are enabled in CometApproxCountDistinct (serde/aggregates.scala), covered by the two mixed-execution tests in CometExecRuleSuite. I think this can be closed, and #4823 looks superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:aggregationHash aggregates, aggregate expressionsenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions