Repository navigation
[kv] Bump RocksDB version to 11.8.1 - #4133
Conversation
|
@platinumhamburg, I would like to work on this |
0c7a99a to
f731cb1
Compare
@rakeshkodathala Thank you for your interest. Upgrading RocksDB requires comprehensive consideration of both server-side and client-side compatibility, as well as porting all the frocksdb features currently referenced by Fluss. As of now , I have already completed all the related work. If you are interested in continuing to contribute in this area, please take a look at the repository https://github.com/fluss-contrib. Going forward, some RocksDB-related development work will likely be carried out in the corresponding forked repositories. |
fd0b756 to
7377792
Compare
wuchong
left a comment
There was a problem hiding this comment.
LGTM.
Please rebase and resolve the conflicts. Additionally, could you share a performance benchmark for Fluss KV tables after switching to fluss-rocksdb, if available?
Replace FRocksDB with fluss-rocksdbjni 11.8.1-fluss-2 and adapt the JNI package and API changes. Handle empty checkpoint files generated by newer RocksDB releases. Closes apache#4132.
Upgrade fluss-rocksdbjni to 11.8.1-fluss-3 and migrate imports to the relocated io.github.fluss_contrib.rocksdb namespace.
7377792 to
d0d665d
Compare
I benchmarked ordinary KV against main ( The test ran on a Mac mini with Apple M4 (10 cores, 16 GB RAM). An OrbStack arm64 VM hosted one coordinator, three tablet servers, and ZooKeeper. Each tablet server used a 768 MiB heap, 256 MiB maximum direct memory, 256 MiB RocksDB shared block cache, and 512 MiB shared write buffer. Every run started with fresh volumes. The workload used a 128-bucket KV table with a 5-million-key range. The client used 32 write threads, four lookup threads, a 512 MiB writer buffer, 2 MiB batches, five in-flight write requests per bucket, and 128 in-flight lookups. It covered bulk upsert, sustained upsert, point lookup, and 50/50 mixed read/write. I ran three passes, alternating the build order.
Overall, ordinary KV throughput was comparable between main and the PR across all four phases, with average differences within 1.6%. |
Replace FRocksDB with fluss-rocksdbjni 11.8.1-fluss-3 and adapt the JNI package and API changes. Handle empty checkpoint files generated by newer RocksDB releases.
Closes #4132.
Purpose
Linked issue: close #4132 4132
Brief change log
Tests
API and Format
Documentation