Skip to content

[update] Serialize VM evacuation during compute reboot - #4200

Open
srac0 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
srac0:update/sync-vm-migration
Open

srac0 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
srac0:update/sync-vm-migration

Conversation

@srac0

@srac0 srac0 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Wait for each VM evacuation to finish before starting the next one. Asynchronous VM migration can cause issues when VMs have overlapping host constraints.

@openshift-ci

openshift-ci Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci

openshift-ci Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign archana203 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ba6dd72244b44902bc4a2fe6985d8730

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 40m 42s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 32s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 27m 56s
❌ cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000211201 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 16s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 20m 24s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 50s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 27s
✔️ cifmw-molecule-update SUCCESS in 6m 23s

@srac0
srac0 force-pushed the update/sync-vm-migration branch from b548ba1 to 17d742e Compare September 23, 2026 21:14
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/8b1192b32ca347c890737c4b0f8ed808

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 28s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 11s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 35m 00s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 47s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 21s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 18m 39s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 07s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 53s
✔️ cifmw-molecule-update SUCCESS in 5m 39s

@srac0
srac0 marked this pull request as ready for review September 29, 2026 07:54
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@srac0
srac0 force-pushed the update/sync-vm-migration branch from 17d742e to dc4c005 Compare September 29, 2026 07:56
@srac0 srac0 changed the title [update] Add guarded synchronous VM evacuation [update] Serialize VM evacuation during compute reboot Sep 29, 2026
When enabled, migrate running VMs in creation order and wait for each
migration before continuing. This prevents overlapping migrations during
post-adoption updates.

Signed-off-by: Samuel Rac <srac@redhat.com>
@srac0
srac0 force-pushed the update/sync-vm-migration branch from dc4c005 to ec90d10 Compare September 29, 2026 08:05
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/caefb2c048054a04bd70d0fc7f3550d5

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 58m 54s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 26m 37s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 22m 55s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 39m 21s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 34m 58s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 28s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 12m 28s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 54s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 51s
❌ cifmw-molecule-update FAILURE in 6m 10s

@michburk

Copy link
Copy Markdown
Contributor

recheck

Comment on lines +24 to +25
'((.status == "ACTIVE") or (.status == "PAUSED")) and
(.["OS-EXT-SRV-ATTR:host"] != $source_host)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI raised a concern over this check using the full 1 hour timeout (720 retries at 5 second intervals) if a migration fails and stays active on the pre-migration host. I'm not super familiar with the process here, but is this a valid concern? Could an early failure get stuck waiting for a full hour before failing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Micheal, I also thought about this issue, however, I haven't found an easy way to check it at the moment. So far, there were no issues with the timeout, but we can update it in the future if needed, based on production logs.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/06e8f36c91f64c8991bb452774c4a06e

❌ openstack-k8s-operators-content-provider FAILURE in 5m 24s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ podified-multinode-edpm-deployment-crc-centos-10 SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 08s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 5m 17s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 35s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 33s
✔️ cifmw-molecule-update SUCCESS in 6m 06s

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants