Skip to content

Transition KVM disk-only VM snapshot to Error state on delete failure - #14283

Draft
DaanHoogland wants to merge 1 commit into
apache:4.22from
shapeblue:ghi14019-kvmVmSnapshotDeleteErrorState
Draft

DaanHoogland wants to merge 1 commit into
apache:4.22from
shapeblue:ghi14019-kvmVmSnapshotDeleteErrorState

Conversation

@DaanHoogland

Copy link
Copy Markdown
Contributor

deleteVMSnapshot in KvmFileBasedStorageVmSnapshotStrategy transitioned the snapshot to Expunging but had no exception handling, so any exception thrown by the merge/delete helpers (e.g. an agent command timeout) left the snapshot permanently stuck in Expunging. VMSnapshotManagerImpl.hasActiveVMSnapshotTasks treats Expunging as active work, so this permanently blocked reboot and further snapshot operations on the VM.

Wrap the delete body in a try/catch, mirroring the existing pattern already used by takeVMSnapshot and revertVMSnapshot in the same class: on any RuntimeException, transition the snapshot to Error via the existing Expunging -> OperationFailed state machine transition before rethrowing.

Test: KvmFileBasedStorageVmSnapshotStrategyTest#testDeleteVMSnapshotMarksSnapshotFailedWhenDeleteThrowsAfterExpungeRequested fails without this fix (no OperationFailed transition is ever fired) and passes with it.

Description

This PR...

Fixes: #14019

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 10.81081% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.05%. Comparing base (bd80951) to head (f1ffa57).

Files with missing lines Patch % Lines
...napshot/KvmFileBasedStorageVmSnapshotStrategy.java 10.81% 33 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               4.22   #14283   +/-   ##
=========================================
  Coverage     18.05%   18.05%           
- Complexity    16323    16325    +2     
=========================================
  Files          5937     5937           
  Lines        536919   536923    +4     
  Branches      65810    65810           
=========================================
+ Hits          96945    96957   +12     
+ Misses       428902   428897    -5     
+ Partials      11072    11069    -3     
Flag Coverage Δ
uitests 4.04% <ø> (ø)
unittests 19.13% <10.81%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland
DaanHoogland force-pushed the ghi14019-kvmVmSnapshotDeleteErrorState branch from 678ed74 to db74da6 Compare October 7, 2026 14:19
@DaanHoogland

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19451

@DaanHoogland

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-17124)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55423 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14283-t17124-kvm-ol8.zip
Smoke tests completed. 147 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py
test_list_system_vms_metrics_history Failure 0.25 test_metrics_api.py

@DaanHoogland
DaanHoogland requested a balanced review from Copilot October 20, 2026 07:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

No open issues 🎉

What changed in this PR

Ensures KVM disk-only VM snapshot deletion doesn’t leave snapshots stuck in Expunging when delete/merge operations throw, by transitioning snapshots to Error (OperationFailed) on failure.

Changes:

  • Wraps deleteVMSnapshot body in a try/catch to transition Expunging -> OperationFailed on runtime failures, then rethrow.
  • Adds a unit test asserting ExpungeRequested is followed by OperationFailed when delete fails after expunge is requested.
File Description
engine/​storage/​snapshot/​src/​main/​java/​org/​apache/​cloudstack/​storage/​vmsnapshot/​KvmFileBasedStorageVmSnapshotStrategy.java Adds exception handling in deleteVMSnapshot to mark snapshot OperationFailed on runtime errors during expunge.
engine/​storage/​snapshot/​src/​test/​java/​org/​apache/​cloudstack/​storage/​vmsnapshot/​KvmFileBasedStorageVmSnapshotStrategyTest.java Adds regression test verifying state transitions on delete failure after expunge request.

Review effort: Lite

…e failure

deleteVMSnapshot in KvmFileBasedStorageVmSnapshotStrategy transitioned the
snapshot to Expunging but had no exception handling, so any exception thrown
by the merge/delete helpers (e.g. an agent command timeout) left the
snapshot permanently stuck in Expunging. VMSnapshotManagerImpl.hasActiveVMSnapshotTasks
treats Expunging as active work, so this permanently blocked reboot and
further snapshot operations on the VM.

Wrap the delete body in a try/catch, mirroring the existing pattern already
used by takeVMSnapshot and revertVMSnapshot in the same class: on any
RuntimeException, transition the snapshot to Error via the existing
Expunging -> OperationFailed state machine transition before rethrowing.

Test: KvmFileBasedStorageVmSnapshotStrategyTest#testDeleteVMSnapshotMarksSnapshotFailedWhenDeleteThrowsAfterExpungeRequested
fails without this fix (no OperationFailed transition is ever fired) and
passes with it.
@DaanHoogland
DaanHoogland force-pushed the ghi14019-kvmVmSnapshotDeleteErrorState branch from db74da6 to f1ffa57 Compare October 20, 2026 13:47
@DaanHoogland

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19634

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants