Initial support for lhc4codec - #23391
Open
apeters1971 wants to merge 2 commits into
Open
apeters1971 wants to merge 2 commits into
apeters1971 wants to merge 2 commits into
Conversation
Integrate optional lhc4codec into ROOT compression and RNTuple writing, including Auto and the Beam/Mosaic/Oracle/Crystal backends, builtin fetch with bundled bzip3, and tree2ntuple/rntuple2rntuple utilities to compare native and LHC4 codecs with verify, no-out, and compression diagnostics. Builtin pin is 922d272 (GitLab master). Auto races the ROOT default set (zstd|beam|crystal) via R__SetLHC4AutoCodecs / --auto-codecs; use all or a comma list to change it.
---
Build with a system package (searches /usr and Homebrew /opt/homebrew, /usr/local):
cmake -S /path/to/root -B /path/to/root-build -Dlhc4codec=ON
cmake --build /path/to/root-build -j
# AlmaLinux 9 / 10
sudo curl -fsSL -o /etc/yum.repos.d/lhc4codec.repo \
https://gitlab.cern.ch/apeters/lhc4codec-bin/-/raw/master/lhc4codec-el9.repo
sudo dnf install lhc4codec lhc4codec-devel
# macOS Apple silicon
brew tap apeters/lhc4codec https://gitlab.cern.ch/apeters/lhc4codec-bin.git
brew install apeters/lhc4codec/lhc4codec
If the library is missing, CMake prints these install steps and stops. Use -DLHC4CODEC_ROOT=/prefix for a custom layout.
Or let ROOT clone https://gitlab.cern.ch/apeters/lhc4codec.git (do not set LHC4CODEC_SOURCE_DIR):
cmake -S /path/to/root -B /path/to/root-build \
-Dlhc4codec=ON \
-Dbuiltin_lhc4codec=ON
cmake --build /path/to/root-build -j
CERN GitLab credentials must work non-interactively. The commit is pinned in builtins/lhc4codec/CMakeLists.txt. If a previous configure used LHC4CODEC_SOURCE_DIR, clear it (-ULHC4CODEC_SOURCE_DIR) and remove <build>/builtins/lhc4codec-prefix before reconfiguring.
---
After each page is compressed, --verify decompresses it and compares the result with the raw input buffer (catches codec round-trip corruption). Example:
tree2ntuple --no-out --verify --jobs 1 input.root
tree2ntuple --no-out --verify --auto-codecs all input.root
--no-out writes each variant to a temp file and deletes it after measuring size. --jobs 1 runs all default formats sequentially (native_zstd, native_lzma, and the lhc4_* variants). --auto-codecs selects the Auto race (root by default, or all / zstd,beam,...). LHC4 zip/unzip failures dump page buffers under /tmp/root-lhc4-failure-<pid>/.
GetConstSubfields() was allocating a new vector on every TTree entry, which made tree2ntuple and other imports much slower than they should be. Caching the leaf-count layout once removes that bottleneck and makes the conversion run much quicker.
apeters1971
requested review from
bellenot,
dpiparo,
jblomer,
pcanal and
silverweed
as code owners
September 16, 2026 17:38
jblomer
reviewed
Sep 17, 2026
jblomer
left a comment
jblomer
left a comment
Contributor
There was a problem hiding this comment.
Thank you!
In principle looks good to me!
Regarding commits: can you split the first commit in three:
- addition of the new algorithm
- modifications to RNTuple
- the new binaries (
[rnutple|tree]2rntuple)
Can you submit the second commit as a separate PR to the master branch? I'll then merge this and rebase the lhc4codec branch.
The failure dumps are ok for now but should be removed eventually, I think. If easily possible, perhaps that functionality can be guarded by a define.
| const auto subfields = c.fRecordField->GetConstSubfields(); | ||
| const auto &offsets = c.fRecordField->GetOffsets(); | ||
| R__ASSERT(subfields.size() == c.fLeafBranchIndexes.size()); | ||
| c.fPackedLeaves.clear(); |
Contributor
There was a problem hiding this comment.
assert(c.fPackedLeaves.empty())?
Comment on lines
+947
to
+952
| if (fTypeAlias == "Double32_t") { | ||
| if (options.GetEnableColumnEncoding() && options.GetCompression() != 0) | ||
| SetColumnRepresentatives({{ROOT::ENTupleColumnType::kSplitReal32}}); | ||
| else | ||
| SetColumnRepresentatives({{ROOT::ENTupleColumnType::kReal32}}); | ||
| } |
Contributor
There was a problem hiding this comment.
I guess this could also be a separate fix to master? And here, it should use usePlainColumns.
| fSealedPages.clear(); | ||
| } | ||
|
|
||
| void ROOT::Internal::RPageSinkBuf::RColumnBuf::ValidateReadyToCommit(bool requireChecksum) const |
Contributor
There was a problem hiding this comment.
This seems unrelated to the new compression algorithm. What is the motivation to add this?
Comment on lines
+39
to
+47
| static int R__LHC4Codec = static_cast<int>(lhc4codec::Codec::Beam); | ||
| static int R__LHC4Filters = 0; | ||
| static int R__LHC4FilterFallback = 1; | ||
| static int R__LHC4FilterRle = 1; | ||
| static int R__LHC4FilterDict = 1; | ||
| static int R__LHC4AutoMinGainPct = 1; | ||
| static int R__LHC4AutoMaxLevel = 0; | ||
| static unsigned R__LHC4AutoCodecs = kLHC4AutoCodecsRoot; | ||
| static int R__LHC4LastCxLevel = -1; |
Contributor
There was a problem hiding this comment.
Perhaps add a TODO: At some point, we need to decide on settings or make it part of the level option.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull request:
Changes or fixes:
Checklist:
This PR fixes #