Skip to content

fix(array): scale decimal-to-float scalar casts across the whole scale domain - #9883

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/decimal-scalar-scale-factor
Open

jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/decimal-scalar-scale-factor

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

DecimalScalar::cast computed the scale factor for the float target as
10_i128.pow(self.decimal_type.scale() as u32)
(vortex-array/src/scalar/typed_view/decimal/scalar.rs:98). DecimalDType::try_new bounds scale
above at MAX_SCALE (76) and not at all below, so that expression is wrong at both ends. Measured
in release:

input produced correct
decimal(76,39) storing 1 → f64 -4.796830079047167e-38 1e-39
decimal(5,-5) storing 1 → f64 inf 100000.0

Above scale 38 the factor overflows i128 and wraps. A negative scale reaches pow as a wrapped
exponent, since -5i8 as u32 is 4294967291 — and semantically a negative scale should scale the
value up, not divide by anything.

Three places in the same crate already get this right: DecimalValue::rescale_i256
(dvalue.rs:127) widens the scale to i16 before branching on its sign and takes the factor
through the fallible decimal_scale_factor, and the array kernel
arrays::decimal::compute::cast::cast_to_f64 (cast.rs:196) takes it as 10f64.powi(-scale).
Only the scalar float branch did not.

The extreme scales are already pinned for the integer target, on both sides:
cast_decimal_to_integer_policy (cast.rs:508) has cases at scale -2, 76 and -128 and
asserts the array and scalar casts agree, and cast_decimal_to_integer_wide_storage (:691) does
the same for decimal(76,40). That target goes through DecimalToIntegerCast, which is fine. The
float target had no equivalent.

Fix

Take the factor in f64 — the widest the -128..=76 domain needs is 10^128, well inside range —
and branch on the sign.

Division is kept for non-negative scale rather than multiplying by the inverse, so every scale up
to 22, where 10^scale is exactly representable, returns the same bits as before. Multiplying by
the inverse instead would shift 12345 / 100 by an ulp.

The i128 funnel above (to_i128, which errors for an i256 value wider than i128 where the
array kernel yields a float) is a third scalar/array divergence, left alone here: turning that
Err into an Ok is a semantics change, not a bug fix. The neighbouring TODO(connor) still
applies to the remaining as casts.

Tests

cargo test --release -p vortex-array --no-fail-fast: 3535 + 73 + 1 passed, 3526 + 73 + 1 before.
Two arrays::listview - should panic tests fail identically before and after — they rely on
debug_assert, which is off in release.

Restoring the old expression fails all five scale-domain cases and the f32 case while leaving the
three exactness cases green, which is what shows the common path is untouched.

Scale 23..=38 can shift by up to ~2 ulp, since powi accumulates rounding where the old exact
i128 factor did not. No DecimalDType in the tree uses a scale in that band: the literal scales
across the workspace are {-128, -67, -19, -11, -10, -8, -5, -4, -2, 0..6, 10, 20, 40}.

AI assistance

Written with agentic AI assistance; I reproduced both wrong values before changing anything, and
the first version of the fix (multiply by the inverse, matching the array kernel exactly) was
rejected by its own exactness test.

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Sep 30, 2026
…e domain

`DecimalScalar::cast` computed the scale factor as
`10_i128.pow(self.decimal_type.scale() as u32)`. `DecimalDType` admits any
`i8` scale up to `MAX_SCALE` (76), so that expression is wrong at both ends:

- a scale above 38 overflows `i128`, which in release wraps to a garbage
  factor — `decimal(76,39)` storing 1 cast to `-4.8e-38` instead of `1e-39`;
- a negative scale reaches `pow` as a wrapped `u32` exponent, because `-5i8
  as u32` is 4294967291 — `decimal(5,-5)` storing 1 cast to `inf` instead of
  `100000.0`. A negative scale should scale the value up, not divide it.

Take the factor in `f64` and branch on the sign, the way the array kernel
`arrays::decimal::compute::cast::cast_to_f64` already does. The widest
factor the domain needs is `10^128`, well inside `f64` range.

Division is kept for non-negative scale rather than multiplying by the
inverse, so every scale up to 22 — where `10^scale` is exactly
representable — returns the same bits as before.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@jackylee-ch
jackylee-ch force-pushed the fix/decimal-scalar-scale-factor branch from b2fc5e5 to 355ebbc Compare September 30, 2026 11:39
@codspeed

codspeed Bot commented Sep 30, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 12.37%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 2069 untouched benchmarks
⏩ 503 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ WallTime dict_canonicalize_gt_u8_neon[1000000] 548.2 µs 487.8 µs +12.37%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:fix/decimal-scalar-scale-factor (355ebbc) with develop (91285b8)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions github-actions Bot removed the stale This PR is stale and will be auto-closed soon label Oct 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant