GH-17211: [C++] Add `hash32` and `hash64` scalar compute functions by kszucs · Pull Request #45001 · apache/arrow

kszucs · 2024-12-11T12:29:11Z

Rationale for this change

Support for calculating elementwise hashes.

The PR adds two scalar functions hash32() and hash64() using the existing internal hashing machinery.

What changes are included in this PR?

Continuation of #39836 with the following changes:

Use column oriented hash-combine rather than flattening nested elements
Support arbitrary nesting levels with an optimization that only hash child arrays if they are also nested
Carry nullness in the output validity bitmap rather than reserving a hash value as a null sentinel: a null input row produces a null output row. A null struct field at any depth makes the whole row null; a null list or map element does not, since only the row's own validity matters there.
Hash dictionaries by their decoded values, so arrays encoding the same logical values via different dictionaries agree, and a valid index pointing at a null dictionary entry produces null.

Are these changes tested?

Yes. scalar_hash_test.cc covers the supported types, slicing of nested and independently-offset children, null propagation through nesting, and the unsupported-type errors. test_compute.py adds hypothesis tests asserting the null contract and that hashing a slice equals slicing the hash. Also verified under ASAN.

Are there any user-facing changes?

There are two new compute kernels, hash32 and hash64, available, documented in compute.rst. Null input rows produce null output rows.

GitHub Issue: [C++][Compute] Add scalar_hash function #17211

kszucs · 2024-12-11T13:16:39Z

Seems like we generate the same hash for both NULL and 0 which is not ideal.

In [1]: import pyarrow as pa

In [2]: import pyarrow.compute as pc

In [3]: pc.hash_64([None])
Out[3]:
<pyarrow.lib.UInt64Array object at 0x124247be0>
[
  0
]

In [4]: pc.hash_64([0])
Out[4]:
<pyarrow.lib.UInt64Array object at 0x1033027a0>
[
  0
]

zanmato1984

Some first glance comments. I'll look into more details later.

…tirely HashArray already zeroes out[i] for every genuinely-null row of `sliced` (via ZeroNulls or the valid-0 remap, both of which correctly use sliced's own offset), so null-ness is already fully encoded in the hash values themselves. A validity buffer is therefore unnecessary -- and reusing the child's raw one would need rebasing anyway, since it's unshifted while the returned ArrayData has offset 0. Simpler and cheaper than repacking a copy.

The rest of compute.rst uses a single blank line between sections; the Hash Functions insertion had picked up an extra one on each side.

…os in hot benchmark loops StructArray::Slice() doesn't reslice child_data, so a struct's nested (list/struct) field was being hashed in full (child.length rows) even when only a small slice of the struct was requested -- the same class of bug fixed for list/map child data earlier, just for struct fields. Hash only the referenced range instead (~580x faster for a heavily sliced struct with a nested list field, per the new Hash64StructWithNestedListHeavilySliced benchmark). Also switch scalar_hash_benchmark.cc's hot loops from ASSERT_OK_AND_ASSIGN to CallFunction(...).ValueOrDie(), since the gtest assertion machinery isn't meant for and adds needless overhead inside a benchmarked loop.

{input_keycol} constructed a fresh std::vector on every iteration, adding allocation overhead that distorted the measurement, especially for small inputs.

They claimed the result is always an Array and referenced a "NestedArray" type that doesn't exist in Arrow. Clarify that the result matches the input's shape (Array/ChunkedArray), that nested types (struct, list, map, etc.) combine child values per row recursively, and mention the null sentinel behavior and lack of cross-version hash stability.

For LIST/LARGE_LIST/FIXED_SIZE_LIST/MAP, rel_start was computed as offsets[0] - values.offset and then HashChild was called with values.offset + rel_start, which algebraically cancels to just offsets[0] -- so values.offset was never actually applied. This produced incorrect hashes whenever the values/items child itself carried a pre-existing nonzero offset independent of the parent array (e.g. a list built via FromArrays with an already-sliced values array). Fix: define rel_start/rel_end as pure logical indices into `values` (relative to values.offset, matching how offsets buffers and GetValues<T> already work), and correspondingly adjust CombineOffsetRows's bias and the FIXED_SIZE_LIST per-row start formula so they no longer assume the old (buggy) rel_start definition.

…tinel Copilot review flagged that HashStructArray (and, by the same pattern, HashListArray) fed field/element hashes into HashMultiColumn/CombineRange without remapping a 0 result the way the leaf path already does -- so a struct whose fields are all valid could still legitimately combine to the same 0 used for a null struct row. Confirmed with a repro: struct{f0: 0} (int64) hashes to exactly 0 for both hash32/hash64, indistinguishable from a null struct. Fix reuses the leaf path's remap, but struct fields need an extra exclusion: a field independently null within an otherwise-valid struct row is documented to hash to 0 too (apacheGH-17211), including transitively through nested structs, so the remap must skip any row where a direct or nested child is null -- otherwise it would incorrectly overwrite that legitimate 0 with a nonzero sentinel. Also strengthens the hash32/hash64 hypothesis tests, which previously only checked determinism, to assert the null-sentinel and no-collision invariants against arbitrarily-shaped generated arrays.

…G variance MinGW CI failed TestScalarHash.RandomPrimitive: hash_set.size() was 48 vs a required 48.02 (tolerance 0.98). This isn't a hashing bug -- the test generates its arrays via RandomArrayGenerator, which uses std::uniform_int_distribution directly; that distribution's algorithm is implementation-defined, not just seed-defined, so the same seed can legitimately produce a different sequence (and occasionally a duplicate value, hence a correctly-duplicate hash) on a different platform/standard library. Loosen the tolerance to 0.9, enough to absorb an incidental duplicate or two without masking a real hash-quality regression. HashQuality already covers hash quality rigorously with inputs that are unique by construction, unaffected by this.

HashStructArray tracked any_child_null correctly but only skipped the null-sentinel remap for those rows, relying on HashMultiColumn to have already produced a literal 0. That only holds for column 0's null rows; a null in any later column instead combines with the running hash of earlier columns (the behavior HashMultiColumn's other caller, hashing independent group-by/join key columns, needs). Force the struct-level invariant explicitly instead. Extends the existing regression test with a multi-field case, since the single-field case couldn't catch this.

The three functions were identical except for the string-length range passed to MakeStructArray. Collapse into one Hash64StructWithStrings, parameterized via benchmark::State::range() and registered with ->Args() per size bucket.

The kernels declared OUTPUT_NOT_NULL and encoded a null row as the hash value 0, which forced remapping any valid row that legitimately hashed to 0 and made every nested combine step preserve that reserved value. Nullness now lives in a real output validity bitmap: HashArray and friends thread an out_validity parameter, so a valid row may hash to anything, and ZeroNulls and RemapValidZeroHashes are gone. The rules themselves are unchanged: a null row is null, a struct row with an independently-null field at any depth is null, and a list/map row's own validity is all that matters for it. Also: decode dictionaries to their logical values rather than hashing raw indices, so different dictionaries encoding the same values agree and a valid index into a null dictionary entry is null; canonicalize a null child's hash value before a parent folds it in, or list<struct<f0:int32>> rows [{f0: 7}] and [null] (whose f0 slot also holds 7) collide; reject unsupported dictionary value types at dispatch instead of deep inside Cast; and speed up validity handling via CopyBitmap/BitmapAnd and by not deep-copying ArraySpan per field (hash64 over int64 2.2x, over list<int64> 1.5x). Docs and the Python tests asserted the old contract and are updated.

fixed_size_binary(0) carries no data, so every value is the same empty string, yet rows hashed differently and an array disagreed with its own slice. ToColumnArray can only describe the type as a fixed-width column of length 0, exactly how a bit-packed boolean is encoded too, so HashMultiColumn called HashBit and took each row's hash from a bit that doesn't exist -- uninitialized memory, varying with the row's bit offset. Give every row one fixed hash in HashArray instead. Broken for the plain type all along, and reachable as dictionary(_, fixed_size_binary(0)) once dictionaries began being decoded; found by the pyarrow hypothesis tests. No behavior change otherwise: zero a null element's hash only in HashListArray, whose CombineRange folds values without consulting validity, and inline that helper into its one caller -- struct fields need none of it, since HashMultiColumn receives their validity and already fixes each null row's contribution. Drop single-use CombineOffsetRows so both row-folding branches read alike, and tighten scoping and comments.

A NullType field has no validity bitmap at all, so HashStructArray's per-field BitmapAnd silently skipped it and left the row valid even though every NullType row is null.

Prevents the compiler from eliding HashMultiColumn calls whose output is otherwise never read back within the benchmark loop.

Replace the bit-by-bit GenerateBitsUnrolled pass over the output validity bitmap with a CopyBitmap/CountSetBits pair, matching how validity is already copied elsewhere in this file. Also lowercase mid-sentence "hash functions" and hyphenate "run-end encoded"/"view-encoded" in the compute docs.

HashableMatcher only inspected the top-level type id (after unwrapping extension/dictionary), so an unsupported type nested inside a supported one -- list<binary_view>, struct<..., binary_view>, map<.., REE> -- passed dispatch and then failed deep inside ToColumnArray with a raw TypeError instead of a clean NotImplemented. Matches() now recurses into child fields, the same fix already applied for an extension's storage type.

A struct's non-nested children went straight to ToColumnArray, bypassing HashArray's dedicated zero-width branch, so struct<fixed_size_binary(0)> reintroduced the nonexistent-bit read already fixed for the plain type: rows holding the same empty value hashed differently. NeedsRecursiveHash now takes the DataType rather than just its id, so it can claim zero-width fixed_size_binary for the recursive path.

initialize.cc calls RegisterScalarHash unconditionally, but scalar_hash.cc was only listed in CMake, so Meson builds would compile the caller without the definition and fail to link. Wires up all four new sources to match CMake: scalar_hash.cc into the compute lib, scalar_hash_test.cc into arrow-compute-scalar-utility-test, and the scalar_hash and key_hash benchmarks.

A null list/map element had its hash canonicalized to 0 before the fold, dropping its validity. A valid integer 0 also hashes to 0, as does HashMultiColumn's substitution for a null slot, so [null] and [0] hashed alike -- and so did a null struct field, where the element itself is present: map<utf8, int32> entries {"a": null} and {"a": 0}. Any other constant would only narrow the collision, so fold nulls into a second accumulator instead: a valid element folds its hash, a null one folds its position, and the two mix at the end. Nothing there can be mistaken for a value hash, positions keep [null, x] and [x, null] apart, and a row without nulls folds nothing extra, so only real nulls cost anything -- Hash64ListInt64 goes from 89.9us to 103.2us. The validity is the one HashChild propagates, so a struct row with a null field counts as a null element, per the documented semantics.

A map is stored as list<struct<key, item>>, and the struct rule that a null field nullifies the row marked an entry with a null item absent, so its key was never folded: [["a", null]] and [["b", null]] hashed alike, and every map with null items collapsed whatever its keys. Arrow requires non-null keys and allows null items (MapArray::ValidateChildData), so that rule must not apply to a map's entries. Fold the keys and the items as two list folds over the map's own offsets instead, which keeps every key contributing and encodes a null item just as a null list element is. It recurses like any other nested type, so a map's key or item may itself be a map, to any depth. Hashing a map costs about 28% more as a result -- two passes over the entries rather than one fused pass over both columns -- while lists and primitives are unchanged.

- BinaryLike: replace an exact-duplicate CheckBinary call with a case covering a repeated value across rows - ZeroValueIsValid: add float16, the only fixed-width HashIntImp type the test's own header comment claimed to cover but omitted - CheckHashQuality: hoist the null-collapse explanation above both hash32/hash64 branches and fix it mispointing readers to a hash64 branch 'below' when it is actually above - UnsupportedNestedChildType: add a struct nesting an unsupported-value- type dictionary, since UnsupportedDictionaryValueType only exercises that case at the top level - RandomPrimitive: add decimal32/decimal64, which RandomArrayGenerator already supports alongside decimal128/decimal256

scalar_hash.cc uses std::vector/std::string/std::shared_ptr, and the two hash benchmarks use std::shared_ptr/std::unique_ptr, without including their headers directly and relying on transitive includes.

The executor promotes an all-scalar span to length-1 arrays before Exec, so the kernel only sees arrays; pin that down with a test that a scalar hashes as its array row does.

The docs described only array input, though a scalar argument returns a scalar and a chunked one returns chunked. Cover the chunked shape with a test too, which nothing exercised before.

These strings surface through the bindings' function help, so they should carry the same contract the C++ API docs do rather than a subset.

Copilot

🟡 Changes recommended

A critical issue remains where oversized execution spans can wrap to uint32_t and leave hashes unwritten.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 20/20 changed files
Comments generated: 1
Review effort level: Lite

It narrows the count to a uint32, and nothing upstream caps what reaches the kernel: the executor does not split spans by default, and a list's values child can be longer than its parent. Past UINT32_MAX the count wrapped and the tail of the output was left unwritten.

Copilot

🟡 Changes recommended

Fix the critical null dereference in variable-length hashing and add regressions for empty/all-null inputs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

cpp/src/arrow/compute/kernels/scalar_hash.cc:381

The kernel accepts fixed-size binary and documents that null inputs produce null outputs, but scalar promotion calls ArraySpan::FillFromScalar for an invalid FixedSizeBinaryScalar, whose value is null; that code unconditionally dereferences scalar.value before this kernel runs. Thus hash32/hash64 on MakeNullScalar(fixed_size_binary(...)) can crash instead of returning a null scalar. Please handle invalid fixed-size-binary scalars in promotion or add an equivalent pre-kernel short-circuit, with a regression test.

    } else if (!NeedsRecursiveHash(*array.type)) {
      ARROW_ASSIGN_OR_RAISE(auto column, ToColumnArray(array));
      std::vector<KeyColumnArray> columns{column.Slice(array.offset, array.length)};
      HashMultiColumnChunked(columns, hash_ctx, out);
      // A plain column's own validity is the whole story, and HashMultiColumn has
      // already folded it into the hash values via ToColumnArray's buffer.
      WriteOwnValidity(array, out_validity);

cpp/src/arrow/compute/kernels/scalar_hash.cc:291

A null FixedSizeListScalar reaches this path with a zero-length child span (ArraySpan::FillFromScalar), but rel_end - rel_start is still list_size. HashChild therefore widens that span and the primitive hash reads past its zero-length buffers (for example, hash32(MakeNullScalar(fixed_size_list(int32(), 8)))), causing undefined behavior/ASAN failures; the same applies when such a null fixed-size-list is nested in another value. Avoid reading a missing child range for invalid rows, or materialize a list_size-sized child during scalar promotion.

    ARROW_ASSIGN_OR_RAISE(auto value_hashes,
                          HashChild(values, values.offset + rel_start,
                                    rel_end - rel_start, hash_ctx, exec_ctx));

Files reviewed: 20/20 changed files
Comments generated: 1
Review effort level: Lite

+    if (array.GetBuffer(2) != nullptr) {
+      var_length_buffer = array.GetBuffer(2)->data();
+    }
+  } else if (is_large_binary_like(type_id)) {
+    metadata = KeyColumnMetadata(false, sizeof(uint64_t));
+    if (array.GetBuffer(2) != nullptr) {
+      var_length_buffer = array.GetBuffer(2)->data();


github-actions Bot added Component: C++ Component: Documentation awaiting committer review Awaiting committer review labels Dec 11, 2024