HDDS-16949. BackgroundService stops permanently when getTasks() throws a RuntimeException

[smengcl]

Generated-by: Claude Code (Sonnet 5.5)

What changes were proposed in this pull request?

BackgroundService.PeriodicalTask#run called getTasks() outside any try/catch. The task is scheduled with scheduleWithFixedDelay, which silently cancels all later executions when a run throws. DirectoryDeletingService#getTasks iterates the snapshot chain without a lock, so a concurrent snapshot purge makes the iterator throw NoSuchElementException, and the service then never runs again on that OM. This is what made TestSnapshotDeletingServiceIntegrationTest#testSnapshotWithFSO time out waiting for deep cleaning.

This patch catches RuntimeException from getTasks(), logs it and retries in the next interval, so one bad cycle cannot cancel the schedule of any background service. Making the snapshot chain iteration itself race safe is left as a follow up.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16949

How was this patch tested?

Added TestBackgroundService#testServiceKeepsRunningAfterGetTasksThrows, where getTasks() throws on the first call. It fails with a timeout without the fix and passes with it. TestBackgroundService (3 tests) and checkstyle pass locally.

• 10x10 flaky-test-check on this branch: in progress (an earlier run on the previous commit, before the test only changed its thrown exception type, passed 100/100)

🤖 Generated with Claude Code

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings

• This wait condition can throw a NullPointerException due to auto-unboxing when map.get(0) is null… · New
• The test's intent is to validate that any unchecked exception from getTasks() doesn't cancel… · New

What changed in this PR

Prevents BackgroundService periodic execution from being permanently cancelled when getTasks() throws at runtime, which previously caused background services to stop silently and led to test timeouts.

Changes:

• Catch and log RuntimeException from BackgroundService.PeriodicalTask#run when calling getTasks(), then return to allow the next scheduled interval to proceed.
• Add a regression test ensuring the service continues running after an initial getTasks() failure.

File	Description
hadoop-hdds/​framework/​src/​main/​java/​org/​apache/​hadoop/​hdds/​utils/​BackgroundService.java	Wraps getTasks() in try/catch to prevent scheduled task cancellation on runtime exceptions.
hadoop-hdds/​framework/​src/​test/​java/​org/​apache/​hadoop/​util/​TestBackgroundService.java	Adds a test verifying periodic scheduling continues after getTasks() throws once.

🧠 Review effort: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jojochuang]

Approve. Catching RuntimeException from getTasks() in PeriodicalTask.run() is the right framework-level fix: scheduleWithFixedDelay stops scheduling after an uncaught throw, which matches the DDS outage when snapshot-chain iteration failed mid-getTasks.

The unit test cleanly proves the schedule survives a failed first getTasks() cycle. This is defense in depth alongside HDDS-16954 (#11500), which fixes the iterator race itself—both should land.

[priyeshkaratha]

Thanks for the patch @smengcl and changes LGTM