fix(store): keep GET /v1/partitions working on follower stores - #3232
bitflicker64 wants to merge 1 commit into
Conversation
PartitionAPI.getPartitions built each raft group's conf through PartitionEngine.getCurrentConf(), which calls jraft Node.listPeers(). jraft only allows that on the leader, so the endpoint failed with a 500 "Not leader" at the first group the store follows. Every store follows some groups once replicas > 1, so the endpoint failed cluster-wide. Move the conf into the existing isLeader() block that already guards peers and learners. Followers now report conf as null, matching peers and learners. Also add TODO comments at dead code found nearby, with no behavior change: a dead return and an unused DashResponse line in PartitionAPI, a commented-out changeShards block, an unused groupId with an unlogged error in doBlankTaskSync and a silent replicator onError in PartitionEngine, and HgStoreNodeServiceTest, whose tests are all commented out. Fixes apache#3230
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3232 +/- ##
============================================
- Coverage 41.35% 41.34% -0.01%
+ Complexity 7299 7295 -4
============================================
Files 802 802
Lines 69688 69688
Branches 9291 9291
============================================
- Hits 28816 28812 -4
- Misses 37576 37587 +11
+ Partials 3296 3289 -7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
517aa51 to
fc017f8
Compare
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: Moving setConf inside the isLeader() block fixes the follower 500 from #3230, and the new PartitionAPITest reproduces the jraft contract and runs in the store CI job. One non-blocking note: the isLeader() check and the three leader-only calls are not atomic, so a leader step-down during the request can still return 500. Evidence: read PartitionAPI.getPartitions and PartitionEngine.getCurrentConf/isLeader at fc017f8; read NodeImpl.listPeers and isLeader(false) in jraft-core 1.3.13 sources; confirmed PartitionAPITest ran and passed in each hg-store-test step of the store job log; all workflows on this head passed, only codecov/project reports -0.01%.
| raft.setConf(engine.getCurrentConf().toString()); | ||
| // jraft only lists peers and learners on the leader | ||
| if (engine.isLeader()) { | ||
| raft.setConf(engine.getCurrentConf().toString()); |
There was a problem hiding this comment.
Minor: engine.isLeader() calls raftNode.isLeader(false), which reads state without the node lock. getCurrentConf(), listPeers() and listLearners() then each take the read lock and throw IllegalStateException("Not leader") if state is no longer STATE_LEADER (jraft-core 1.3.13 NodeImpl.listPeers). If the group steps down between the check and these calls, for example during a PD leader balance, the whole GET /v1/partitions response is still a 500. This race existed before for listPeers() and the PR does not make it worse, so it does not block.
If you want the endpoint to survive leader transfers too, wrap the three calls in one try { ... } catch (IllegalStateException e) { ... } and leave conf, peers and learners null on failure, the same as the follower case. A second follower-style mock whose isLeader() returns true while the node methods throw would cover it.
Purpose of the PR
GET /v1/partitionson a Store returns 500IllegalStateException: Not leaderas soon as it reaches a raft group the Store follows.PartitionAPI.getPartitionscallsPartitionEngine.getCurrentConf()for every group, andgetCurrentConf()builds theConfigurationfromraftNode.listPeers()andraftNode.listLearners(). jraft'sNodeImplthrowsNot leaderfrom both unless the node is the leader. With more than one replica per group every Store follows some groups, so the endpoint fails on every Store in the cluster.Main Changes
PartitionAPI.getPartitions: moveraft.setConf(...)inside theengine.isLeader()block that already guardssetPeersandsetLearners. On followersconfis nownull, likepeersandlearners; all other fields (role,leader,term,logIndex, partitions and metrics) are filled as before.PartitionEngine.getCurrentConf()itself is unchanged, since its other callers (changePeers,onLeaderStart,onStartFollowing) are outside this endpoint.DashResponseline inPartitionAPI; a commented-outchangeShardsblock, an unusedgroupIdwith an unlogged error indoBlankTaskSync, and a silent replicatoronErrorinPartitionEngine; andHgStoreNodeServiceTest, whose@Testannotations are all commented out.PartitionAPITestinhg-store-node/src/test, next to the existingScanUtilTest. It mocks one follower group and one leader group. The follower'sNode.listPeers(),Node.listLearners()andgetCurrentConf()throwIllegalStateException("Not leader"), which is how jraft behaves. The test checks that both groups are listed, that the follower hasconf,peersandlearnersset to null, and that the leader still reports its conf and peers.Verifying these changes
PartitionAPITest#testGetPartitionsOnFollower, run with JDK 11:mvn test -pl hugegraph-store/hg-store-node -am -Dtest=PartitionAPITest \ -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=falseWithout the
PartitionAPIchange it fails with the error from the issue:With the change, together with the other
hg-store-nodetests (-Dtest='PartitionAPITest,ScanUtilTest,JraftMetricsTest'):mvn editorconfig:check -pl hugegraph-store/hg-store-nodepasses.I have not yet re-run this against a live 3-Store cluster. The verification above is the unit test only.
Does this PR potentially affect the following parts?
Store REST
GET /v1/partitionsnow returns 200 on a Store that follows some groups instead of 500. For those groupsconfisnullin the response. Nothing changes for groups the Store leads.Documentation Status
Doc - TODO: required documentation is pending; complete it before merging.Doc - Done: documentation is included here or linked below.Doc - No Need: no user-visible documentation is affected.Documentation files in this PR or paired hugegraph-doc PR: