Skip to content

zlib: add crc32c - #66828

Open
pipobscure wants to merge 1 commit into
nodejs:mainfrom
pipobscure:zlib-crc32c
Open

pipobscure wants to merge 1 commit into
nodejs:mainfrom
pipobscure:zlib-crc32c

Conversation

@pipobscure

Copy link
Copy Markdown
Contributor

Add zlib.crc32c(data[, value]), which computes the CRC-32C checksum (Castagnoli polynomial, RFC 3720 section 12.1). CRC-32C has better error detection properties than the CRC-32 of zlib.crc32() and is the checksum required by many current protocols, formats and services, among them:

  • iSCSI header and data digests (RFC 7143)
  • SCTP packet checksums (RFC 9260)
  • iWARP MPA framing (RFC 5044)
  • NVMe/TCP header and data digests
  • ext4, Btrfs and XFS metadata checksums
  • the Snappy framing format
  • Apache Kafka record batches
  • LevelDB and RocksDB block checksums
  • TensorFlow TFRecord files
  • Google Cloud Storage and Amazon S3 object checksums

Implementing any of these in JavaScript currently needs a userland CRC-32C, which is far slower than native code.

The function mirrors zlib.crc32(): it accepts a string (encoded as UTF-8) or an ArrayBufferView, takes an optional 32-bit unsigned starting value so a checksum can be continued across chunks, and has a V8 fast API path.

The checksum is computed by absl::ExtendCrc32c() from the bundled Abseil, which is already built and linked into node, so this adds no new dependency and no new checksum implementation. absl_crc32c is added to the libraries linked with --shared-abseil.

Abseil selects its hardware-accelerated CRC-32C path (SSE4.2 and PCLMULQDQ on x64, the CRC32 and PMULL extensions on arm64) at compile time, so with node's default compiler flags the portable table-driven implementation is used. It runs at about the same speed as zlib.crc32() (about 6.5 GB/s on a recent x64 machine), while the hardware path reaches about 38 GB/s. Builds whose target CPU baseline already includes those instructions get the hardware path without changes. Enabling it for generic builds is left out on purpose: on x64, Abseil uses the accelerated code for unrecognized CPUs without a runtime feature check, so compiling it in unconditionally would fault on CPUs without SSE4.2.

I had agentic help here, but I did review and check every single line.

Add zlib.crc32c(data[, value]), which computes the CRC-32C checksum
(Castagnoli polynomial, RFC 3720 section 12.1). CRC-32C has better
error detection properties than the CRC-32 of zlib.crc32() and is the
checksum required by many current protocols, formats and services,
among them:

- iSCSI header and data digests (RFC 7143)
- SCTP packet checksums (RFC 9260)
- iWARP MPA framing (RFC 5044)
- NVMe/TCP header and data digests
- ext4, Btrfs and XFS metadata checksums
- the Snappy framing format
- Apache Kafka record batches
- LevelDB and RocksDB block checksums
- TensorFlow TFRecord files
- Google Cloud Storage and Amazon S3 object checksums

Implementing any of these in JavaScript currently needs a userland
CRC-32C, which is far slower than native code.

The function mirrors zlib.crc32(): it accepts a string (encoded as
UTF-8) or an ArrayBufferView, takes an optional 32-bit unsigned
starting value so a checksum can be continued across chunks, and has a
V8 fast API path.

The checksum is computed by absl::ExtendCrc32c() from the bundled
Abseil, which is already built and linked into node, so this adds no
new dependency and no new checksum implementation. absl_crc32c is
added to the libraries linked with --shared-abseil.

Abseil selects its hardware-accelerated CRC-32C path (SSE4.2 and
PCLMULQDQ on x64, the CRC32 and PMULL extensions on arm64) at compile
time, so with node's default compiler flags the portable table-driven
implementation is used. It runs at about the same speed as zlib.crc32()
(about 6.5 GB/s on a recent x64 machine), while the hardware path
reaches about 38 GB/s. Builds whose target CPU baseline already
includes those instructions get the hardware path without changes.
Enabling it for generic builds is left out on purpose: on x64, Abseil
uses the accelerated code for unrecognized CPUs without a runtime
feature check, so compiling it in unconditionally would fault on CPUs
without SSE4.2.

Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run. labels Oct 18, 2026
@codecov

codecov Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.62500% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.49%. Comparing base (f513ec7) to head (fbeb1b3).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
src/node_zlib.cc 85.00% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #66828      +/-   ##
==========================================
- Coverage   90.49%   90.49%   -0.01%     
==========================================
  Files         797      797              
  Lines      277686   277824     +138     
  Branches    53362    53376      +14     
==========================================
+ Hits       251292   251413     +121     
+ Misses      16796    16795       -1     
- Partials     9598     9616      +18     
Files with missing lines Coverage Δ
lib/zlib.js 98.14% <100.00%> (+0.01%) ⬆️
src/node_zlib.cc 80.28% <85.00%> (+0.07%) ⬆️

... and 38 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants