kvm: look RBD volumes up through librbd instead of refreshing the whole pool

[bhouse-nexthop]

Description

Starting a VM whose root disk is on RBD primary storage refreshes the entire libvirt storage pool. The cost grows with the number of volumes in the Ceph pool, and it is paid on every single VM start.

Why it happens

LibvirtStorageAdaptor.getPhysicalDisk() asks libvirt for the volume. The volume was just created by the management server, so this host's libvirt has never seen it and the lookup misses. getVolume() then refreshes the whole pool:

vol = pool.storageVolLookupByName(volName);
if (vol == null) {
    refreshPool(pool);                        // opens and stats EVERY image in the pool
    vol = pool.storageVolLookupByName(volName);
}

The existing comment in getVolume() already describes the case: "This can happen when a volume has just been created on a different host and since then the libvirt storage pool has not been refreshed." For a cluster that creates VMs continuously, that is every start.

Measured cost

Ceph pool holding 950 RBD images, measured on the hypervisor:

operation	time
list image names only	0.005 s
open + stat + close every image, serially (what a pool refresh does)	11.67 s

Running four of these at once does not slow them down (9.9 - 11.1 s each), so this is not Ceph contention. It is 950 serial round trips.

Agent-side StartCommand on a host in a 12-node cluster running short-lived VMs:

	p50	p90	max
whole StartCommand	35.2 s	68.4 s	141 s
the stall inside getPhysicalDisk	20.3 s	46.6 s	304 s

The stall is 0 when the volume happens to be cached, which is what points at the refresh rather than at fixed work.

Fix

Look RBD volumes up directly through librbd and skip libvirt entirely.

This makes getPhysicalDisk() consistent with the rest of the class. Creating, cloning, resizing, copying and deleting RBD volumes in LibvirtStorageAdaptor already use Rados/Rbd directly. Only the lookup went through libvirt.

• Applies to StoragePoolType.RBD only. Every other pool type is untouched.
• The disk path <ceph pool>/<volume uuid> is the same format the class already builds in createPhysicalDisk, createDiskFromTemplate and createDiskFromTemplateOnRBD.
• Size and virtual size come from rbd stat, matching what the class already does after converting an image into an RBD volume.

What this does not fix

getStorageStats still calls pool.refresh() explicitly, so the full walk of the pool still happens on the storage stats poll. That is a periodic background poll rather than something on the VM start path, so it is left alone here.

Connection handling

Because this runs on every VM start rather than occasionally, the rados connection is handled more carefully than at the existing call sites in this class:

	why
shutDown() in a finally	KVMStoragePoolManager.getPhysicalDisk retries a missing volume 100 times with a 3 s sleep, so leaking the client would mean 101 live librados clients for one absent volume
openReadOnly() rather than open()	the image is only stat'ed, so it should not be able to take the exclusive lock
a close helper that does not throw	Rbd.close() throws, and in a finally that would discard a successful lookup or hide the exception already on its way out
cephx key only set when the pool has a user	librados aborts the process, rather than returning an error, if rados_conf_set("key", NULL) is called. This is pre-existing at the other call sites in this class and is guarded only on the new one

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

How Has This Been Tested?

• Measured the cost of the pool refresh against the observed agent stall, as in the tables above. rbd_list on its own is 0.005 s; opening and stating all 950 images serially, which is what the refresh does, is 11.67 s.
• Ran the same enumeration four times in parallel to rule out Ceph contention: 9.9 - 11.1 s each, no slowdown.
• Checked that the new path produces the same disk path format, format and size as the libvirt path for RBD volumes.
• Full KVM plugin test suite on this branch: 676 tests, 0 failures, 1 skipped.

On the one non-obvious point, the librbd lookup reports rbd stat size as both size and virtual size, where the libvirt path reported allocation and capacity separately. These are the same number for RBD: libvirt derives allocation as obj_size * num_objs. Checked against libvirt 10.6.0 on three RBD volumes of different sizes:

Capacity:  32212254720 bytes   Allocation:  32212254720 bytes
Capacity:  32212254720 bytes   Allocation:  32212254720 bytes
Capacity: 644245094400 bytes   Allocation: 644245094400 bytes

Consumers of those two values (LibvirtGetVolumeStatsCommandWrapper, the template and volume sizes reported back by KVMStorageProcessor, and the snapshot pre-check in validateAvailableSizeOnPoolToTakeVolumeSnapshot) therefore see no change.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.94%. Comparing base (10037c8) to head (d74bf10).

Files with missing lines	Patch %	Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	42 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14188   +/-   ##
=========================================
  Coverage     17.93%   17.94%           
- Complexity    16142    16148    +6     
=========================================
  Files          5928     5928           
  Lines        535205   535247   +42     
  Branches      65501    65503    +2     
=========================================
+ Hits          95989    96028   +39     
- Misses       428286   428290    +4     
+ Partials      10930    10929    -1     

Flag	Coverage Δ
uitests	4.02% <ø> (ø)
unittests	19.01% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.