feat(cluster): respin instances with stalled nodes

[mkoura]

A node whose tip is older than the forecast horizon (3k/f slots) has no ledger view for the current slot and cannot forge anymore. When it happens to all nodes, the chain is halted for good; a single node in that state is usually stuck on a fork deeper than k. Tests on such instance either fail on the missing UTxOs, or hang waiting for new blocks.

Treat such instance as unhealthy, so it is respun once the running tests finish and no new tests start on it meanwhile.

The time of the last block is taken from the newest write to the node's volatile DB, which covers also the blocks downloaded during a sync. Only running nodes are checked, as tests stop nodes on purpose, and a node is measured from the start of its process at the earliest, so a node restarted after a long stop has time to catch up. A node whose volatile DB can't be found or read is not reported. The genesis parameters are cached, as the check runs under the global cluster lock.

[Copilot]

Copilot review overview

🟡 Changes recommended

Genesis parsing errors remain unhandled, and two tests need stronger mtime isolation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

• Handle TypeError and OverflowError when parsing genesis JSON · New
• Ensure test isolates new file mtime from volatile directory mtime · New
• Ensure removed-file test isolates file mtime from directory mtime · New

What changed in this PR

Adds stalled-node detection based on genesis parameters, process uptime, and volatile database activity, triggering respins for unhealthy instances.

Changes:

• Detects nodes that stop advancing.
• Integrates detection into cluster health and respin decisions.
• Adds detection and health-check tests.

File	Changes	Review notes
framework_tests/​test_cluster_nodes.py	Tests uptime and volatile DB activity.	Test directory mtimes should be reset to isolate file-mtime behavior.
framework_tests/​test_cluster_getter.py	Tests health-check integration.	No findings.
cardano_node_tests/​utils/​cluster_nodes.py	Implements stalled-node detection and state resolution.	Handle TypeError and OverflowError when parsing malformed genesis data.
cardano_node_tests/​cluster_management/​cluster_getter.py	Integrates stalled nodes into respin decisions.	No findings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.