wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host - #13758
wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host#13758wido wants to merge 12 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13758 +/- ##
============================================
+ Coverage 19.78% 19.79% +0.01%
- Complexity 19992 20047 +55
============================================
Files 6371 6373 +2
Lines 575903 576323 +420
Branches 70496 70602 +106
============================================
+ Hits 113945 114102 +157
- Misses 449530 449759 +229
- Partials 12428 12462 +34
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR introduces Direct Routed (L3) guest networks where KVM hosts perform L3 routing directly to instances (no Virtual Router, no NAT, no DHCP), using ConfigDrive-only addressing and a shared link-local gateway (169.254.0.1 / fe80::1) on per-network host bridges.
Changes:
- Adds
GuestType.L3with server-side offering/network validation and a newDirectRoutedNetworkGuruthat forces NICs into host-route form (/32, /128) with link-local gateways. - Updates ConfigDrive generation to always emit
network_data.jsonfor direct-routed NICs even when DHCP/DNS services are not present. - Extends the KVM agent to create/manage per-network bridges (
brdr-<networkId>), program host routes/neighbour entries, and adapt security-group rule plumbing for routed traffic (including secondary IP handling).
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/src/main/java/com/cloud/utils/net/NetUtils.java | Centralizes link-local gateway/CIDR constants and adds IPv6 link-local gateway getter. |
| ui/src/views/offering/AddNetworkOffering.vue | Adds UI support for L3 offerings and enforces L3-specific service/provider composition. |
| ui/src/views/network/CreateNetwork.vue | Adds an L3 network creation tab. |
| ui/src/views/network/CreateL3NetworkForm.vue | New UI form for creating L3 (Direct Routed) networks. |
| ui/public/locales/en.json | Adds UI strings for L3 offering/network creation messaging. |
| tools/marvin/marvin/config/test_data.py | Adds Marvin test data for L3 offerings and L3 networks. |
| test/integration/smoke/test_l3_networks.py | Adds smoke/integration tests covering L3 network lifecycle and validations. |
| server/src/test/java/com/cloud/network/guru/DirectRoutedNetworkGuruTest.java | Unit tests for the new guru selection/design and NIC host-route forcing. |
| server/src/test/java/com/cloud/configuration/ConfigurationManagerImplTest.java | Unit tests for L3 offering validation rules. |
| server/src/main/resources/META-INF/cloudstack/server-network/spring-server-network-context.xml | Registers the new DirectRoutedNetworkGuru bean. |
| server/src/main/java/com/cloud/network/security/SecurityGroupManagerImpl.java | Ensures agent commands are sent for secondary IPs on L3 networks even when SG rules aren’t applied. |
| server/src/main/java/com/cloud/network/NetworkServiceImpl.java | Extends network creation/IPv6 checks/VLAN handling and secondary-IP orchestration for L3. |
| server/src/main/java/com/cloud/network/guru/DirectRoutedNetworkGuru.java | New guru for L3 direct routed guest networks; forces /32 + /128 form and link-local gateways. |
| server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java | Adds L3 offering validation + zone-wide IPv4 overlap checks for L3 ranges. |
| scripts/vm/network/vnet/modifymacip.sh | Adds targeted delete support for specific -4/-6 addresses (secondary IP remove). |
| scripts/vm/network/vnet/modifybrdr.sh | New script to create/delete per-network brdr-* bridges with sysctls + gateway addresses. |
| scripts/vm/network/tests/test_security_group.py | New unit tests for classic-vs-L3 security_group.py rule streams and --directrouted plumbing. |
| scripts/vm/network/tests/golden_default_network_rules.txt | Golden output for classic rule generation (used to ensure no regression). |
| scripts/vm/network/tests/golden_add_fw_framework.txt | Golden output for classic FW framework creation (ensures byte-identical behavior). |
| scripts/vm/network/security_group.py | Adds routed-aware framework/hooks and rule generation, plus --directrouted CLI plumbing. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtSecurityGroupRulesCommandWrapper.java | Passes direct-routed flag when applying SG rules for the VM’s first NIC. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtNetworkRulesVmSecondaryIpCommandWrapper.java | Threads direct-routed + apply-SG booleans into secondary-IP handling. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java | Adds --directrouted plumbing and direct-routed secondary-IP route/neigh programming via modifymacip.sh. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/BridgeVifDriver.java | Creates per-network bridges for direct-routed NICs and ensures MAC/IP script runs for those NICs. |
| engine/storage/configdrive/src/test/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilderTest.java | Adds tests for direct-routed NIC detection and network_data.json generation behavior. |
| engine/storage/configdrive/src/main/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilder.java | Always generates network_data.json for direct-routed NICs; adds NIC signature detection. |
| docs/design/direct-routed-networks.md | Adds comprehensive design document for Direct Routed networks. |
| core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java | Adds directRouted and applySecurityGroupRules flags for secondary-IP commands. |
| api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java | Sends agent update for secondary IP removal on L3 even when zone SG is disabled. |
| api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java | Allows specifying physical network ID for L3 networks. |
| api/src/main/java/com/cloud/network/Network.java | Adds GuestType.L3 and parsing support. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Network network = _networkModel.getNetwork(nic.getNetworkId()); | ||
|
|
||
| // On a Direct Routed network the host needs a route and a static neighbour entry for the | ||
| // secondary IP before it is reachable at all. That is independent of security groups, | ||
| // which are optional there and which the Instance may not be using, so the agent is told | ||
| // either way - otherwise the address would stay dark until the Instance was restarted. | ||
| boolean directRouted = Network.GuestType.L3.equals(network.getGuestType()); | ||
|
|
|
@blueorangutan package |
|
@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18739 |
|
@wido this is still marked as wip. what is the status? |
The status is that I have real hardware in a datacenter now where I’m going to deploy this code for further testing. Please keep it WIP for now as I want to validate it there. |
bf3d8a3 to
526934c
Compare
Review feedback on apache#13758: the multi-line inline comments narrating the code are not helpful there. Where a comment documents a real contract (guru design shape, bridge creation/naming, the MAC-before-IPv6 ordering, ConfigDrive as the only addressing channel, secondary-IP delivery independent of security groups, the link-local gateway constants) it now lives as Javadoc on the class, method or field it describes. Mid-method narration is removed; upstream one-line comments that this branch merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) are kept in their original single-line form. No code changes.
|
@DaanHoogland fair point on the inline comments — addressed in a911e9c. The multi-line inline blocks are gone throughout the PR: where a comment documented an actual contract it moved into Javadoc on the class/method/field it describes, the mid-method narration is simply removed, and upstream one-liners this PR merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) kept their original single-line form. That covers the three spots you flagged in ConfigurationManagerImpl and CreateL3NetworkForm.vue plus the equivalent blocks elsewhere. |
… Instances Adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack. Management server: - GuestType.L3; the guest_type column is char(32), so no schema change. - Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared), Dhcp rejected as not supported and not needed. Network mode, specifyVlan and VPC use are rejected. - DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. canHandle() selects on the offering's guest type alone; design() produces a Native broadcast domain with no isolation id. After allocation the NicProfile is forced into host-route form, which is also the signature by which the agent and ConfigDrive recognise these NICs. - createNetwork treats L3 like Shared for the subnet: explicit IP range mandatory, vlan/IP-range row created at network creation, IPv6 accepted without the /64 restriction, aclType Account. - Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide. ConfigDrive: - Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR but would leave these NICs with no addressing at all. Route generation itself is unchanged: cloud-init detects an IPv4 gateway inside 169.254.0.0/16 and sets on-link on the rendered route by itself. KVM agent: - One uplink-less bridge per network, brdr-<network id>, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding and strict rp_filter; separate bridges make isolation between networks topological rather than a filtering concern. - BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host route, regardless of the host-wide EVPN property, whose meaning is unchanged. The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.
…adcast domains
Revises the isolation model of Direct Routed (L3) networks. A network is
now direct routed when its offering's guest type is L3 AND it lives on a
dedicated physical network carrying the new isolation method ROUTED --
the network operator's explicit, zone-level opt-in. Every network gets a
broadcast domain of the new type routed://<id>, and that id names the
per-network bridge on the hypervisors: routed://5828 -> brdr-5828. The
id is a label, never an encapsulation; nothing appears on the wire.
The id lifecycle reuses the Shared network's VLAN mechanics verbatim:
with specifyVlan the operator passes the id at network creation through
the existing vlan parameter (bridge names become plannable, so host
routing policy per brdr-<id> can exist before the network does); without
it the id is allocated from the ROUTED physical network's vnet range at
creation and released on deletion. Zone-wide URI overlap checks apply.
Management server:
- BroadcastDomainType.Routed ("routed", Long).
- DirectRoutedNetworkGuru registers IsolationMethod("ROUTED") and follows
the standard guru contract (isMyIsolationMethod); design() stamps the
Routed broadcast domain and carries the routed:// URI.
- encodeVlanIdIntoBroadcastUri() maps a ROUTED physical network to
routed://<id>; the network's broadcast domain type is derived from the
URI scheme instead of hard-coded Vlan.
- The Shared-without-specifyVlan vnet allocation/release paths extend to
L3; L3 offerings may carry specifyVlan either way.
- An L3 IPv6 subnet longer than /64 is rejected at network creation:
addresses are computed with EUI-64 from subnet + MAC (the existing
setNicIp6Address()/updateNicIpv6() behaviour -- no address pool exists,
only the allocated result on the NIC), and EUI-64 needs 64 bits.
SystemVMs (CPVM/SSVM), dual-stack from the start:
- PublicNetworkGuru stamps NICs of a routed public range (created with
vlan=routed://<id>) in host-route form: /32 + 169.254.0.1, /128 +
fe80::1, Routed broadcast domain. IPv6 comes from the same EUI-64
computation as everywhere else.
- The CPVM and SSVM boot-arg builders emit eth<N>ip6, eth<N>ip6prelen
and ip6gateway, as the VR builder already did.
- common.sh marks the v4 default route onlink for link-local gateways,
and installs a static v6 default from the previously parsed-but-unused
IP6GW -- on a direct routed bridge no RA ever arrives.
KVM agent:
- BridgeVifDriver recognises direct routed NICs by broadcast type
instead of inferring from the address form, for guest and public
traffic alike, and derives the bridge from the NIC's broadcast URI.
modifybrdr.sh mechanics are unchanged; -n is now the routed id.
UI: ROUTED joins the isolation method choices (zone wizard, physical
network form). The design document records the revision and rationale;
no schema change and no data migration.
Reviewing every path the overhaul touches surfaced four fixes: - SystemVM public IPv6 is now computed directly in PublicNetworkGuru (EUI-64 from the range's ip6_cidr and the NIC MAC, then /128 + fe80::1). The previously planned ipv6Service.updateNicIpv6() path is unusable here: it is gated on the public network offering's internet protocol (never set on the system public offering) and reserves through one placeholder NIC per network -- on the shared Public network, every SystemVM would have received the same address. - Guest networks and public ranges share one routed-id space (a routed id names a bridge on every host), so collisions between them would merge L2 domains. Both directions are now rejected: a guest network cannot take an id a public range carries, and a public range cannot take an id a guest network holds. - Creating an L3 network on a physical network without the ROUTED isolation method now fails early with a clear message, instead of an opaque no-guru error deep in setupNetwork(). - A direct routed guest NIC's isolation URI now mirrors its routed:// broadcast URI instead of the misleading vlan://<tag> inherited from the Shared allocation path. GuestType.L3 itself was re-verified as still required: some thirty management-server branches (offering validation, subnet handling, zone-wide overlap checks, secondary-IP host routes, security groups) key on the guest type in places where no physical network -- and thus no isolation method -- is in scope. The guest type says what a network is; the isolation method says where it may live.
…D message CreateNetworkCmd.getPhysicalNetworkId() now accepts guest type L3 and its error message reads 'Shared, L2 or L3 only'; the test still asserted the old 'Shared or L2 only' text and failed.
…maining UI form Add a default network offering for Direct Routed (L3) guest networks, DefaultL3NetworkOffering: UserData and DNS via ConfigDrive, Security Groups enabled - the only service set validateL3NetworkOffering() permits. It is created in NetworkOrchestrator.configure() like the other default offerings, so it appears on fresh installs and on upgrades alike; until now the operator had to create it by hand. The routed id is allocated from the ROUTED physical network's range (no specifyVlan). UI: the ROUTED isolation method was selectable in the zone wizard and in the infrastructure listing's add-physical-network form, but missing from the form on a zone's Physical Networks tab - add it there. Give the VLAN, VXLAN and ROUTED options hover descriptions explaining what each isolation method does, so operators understand what they are opting into; ROUTED's spells out the Direct Routed model (no VLANs on the wire, no Virtual Router, no DHCP, addressing via ConfigDrive, L3 offerings only, KVM only).
Creating an L3 network died in NetworkOrchestrator.createGuestNetwork(): after encodeVlanIdIntoBroadcastUri() produced routed://<id>, the unconditional PVLAN overlap check handed that URI to listByPhysicalNetworkPvlan(), which rejects every URI scheme other than vlan:// and vxlan://: 'Requested URI routed://189 is not in the expected format. Expected URI Scheme as vlan://VID or vxlan://VID.' The check is meaningless for a routed id: no PVLAN network can exist on a ROUTED physical network (only L3 networks can), and the id's real overlap checks - zone-wide URI, public ranges, the vnet range - have already run earlier in the method. Skip it when the URI scheme is routed. Also reject the isolatedpvlan parameter for L3 networks explicitly, so the pvlan branch (which would build a pvlan:// URI around the routed id) is unreachable for this guest type.
CreateL3NetworkForm only supported offerings that allocate the routed id automatically; an operator-chosen id (the API's vlan parameter, like a Shared network's VLAN tag) could be given through the API alone. Add a 'Routed ID' field to the form, shown to root admins when the selected L3 offering carries specifyVlan - where the API requires the id - and hidden otherwise, since the API then rejects the parameter and allocates the id from the ROUTED physical network's range. The field's description explains what the id is: a label naming the per-network bridge (brdr-<id>) on every KVM host, unique per zone, nothing on the wire, and useful to choose in advance so host routing policy can exist before the network does.
…t range check Creating an L3 network whose offering does not carry specifyVlan failed in ConfigurationManagerImpl.createVlanAndPublicIpRange(): 'The VLAN tag 573 is already being used for dynamic vlan allocation for the guest network in zone AMS02' even though the id showed as Free. The check calls findVnet(), which matches any op_dc_vnet_alloc row with that number - taken or free - so it really asks 'does this tag lie inside a dynamic allocation range', not 'is it in use'. An auto-allocated routed id always comes from the ROUTED physical network's vnet range (commitNetwork() takes it via allocateVnet() moments earlier), so the check rejected every such network; only bypassvlanoverlapcheck=true got past it, and the rollback released the id again, which is why it kept showing as Free. Shared networks with specifyVlan=false have long been exempt from this check for exactly this reason (isSharedNetworkWithoutSpecifyVlan forces bypassVlanOverlapCheck). Give L3 networks the same exemption: expose the orchestrator's isL3NetworkWithoutSpecifyVlan() helper on NetworkOrchestrationService and OR it into the bypass condition. The operator-specified path is unchanged: with specifyVlan the given routed id still must not collide with a vnet range, as for Shared.
…EUI-64 IPv6 Deploying a SystemVM on a direct routed public range died with NullPointerException: Cannot invoke "String.toLowerCase()" because "macAddress" is null in ConsoleProxyManagerImpl/SecondaryStorageManagerImpl. In PublicNetworkGuru.getIp() the routed branch calls setRoutedRangeIpv6(), which derives the NIC's IPv6 address with EUI-64 from nic.getMacAddress() - but the MAC was only copied onto the profile at the end of the block, after the branch, so NetUtils.EUI64Address() ran against a null MAC on the freshly created NicProfile. The same ordering hid a second bug: the unconditional nic.setFormat(AddressFormat.Ip4) after the branch clobbered the DualStack format setRoutedRangeIpv6() had just set, so even with a MAC the NIC would have come out IPv4-only. Move the format/reservationId/MAC assignments above the branch chain. The vlan and vxlan paths never touch MAC or format, so they are unaffected. Add PublicNetworkGuruTest covering the routed shape (host-route IPv4, EUI-64 IPv6 from the NIC MAC, DualStack) and the classic VLAN shape as regression tests.
Starting a SystemVM on a direct routed public range failed on the KVM agent: Failed to create vnet 600: Cannot find device "br-600" ... Failed to create vlan 600 on pif: br-600. The agent's dispatch, BridgeVifDriver.isDirectRoutedNic(), checked only the NicTO's broadcast *type* enum. That type is not persisted per nic: at VM start the NicProfile is rebuilt and copies broadcastType from the *network* (NicProfile line 88), and the shared Public network's broadcast domain type is Vlan. The routed:// stamp survives only in the nic row's broadcast URI. The public NIC therefore arrived as type=Vlan, uri=routed://600: the routed branch in plug() was skipped, the Public/VLAN branch parsed 600 out of the routed URI and ran modifyvlan.sh against a nonexistent pif - and the modifymacip.sh call, gated on the same predicate, never happened at all. Guest NICs were unaffected because an L3 network row genuinely carries the Routed domain type. Recognise the routed:// URI scheme as well as the type enum - matching the predicate's own javadoc, which already described recognition by broadcast domain. One predicate serves every agent-side call site (plug, unplug, modifymacip gate, security group wrapper). Add regression tests for both recognitions and the ordinary-NIC negatives.
Review feedback on apache#13758: the multi-line inline comments narrating the code are not helpful there. Where a comment documents a real contract (guru design shape, bridge creation/naming, the MAC-before-IPv6 ordering, ConfigDrive as the only addressing channel, secondary-IP delivery independent of security groups, the link-local gateway constants) it now lives as Javadoc on the class, method or field it describes. Mid-method narration is removed; upstream one-line comments that this branch merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) are kept in their original single-line form. No code changes.
a911e9c to
655530c
Compare
… networks Deploying an Instance with securitygroupids into an L3 (Direct Routed) network failed with 'Can't create vm with security groups; security group feature is not enabled per zone' The message is misleading: the zone flag is only the first half of the gate. checkSecurityGroupSupportForNetwork() also accepts a deploy when one of the requested networks itself supports the SecurityGroup service - but that check compared the guest type against Shared only, so an L3 network carrying the SecurityGroup service (as DefaultL3NetworkOffering does) never qualified and the deploy fell through to the error. Accept GuestType.L3 alongside Shared. Every other security group touchpoint already treats L3 like Shared (offering validation, the secondary-IP paths, the security_group.py dispatch); this deploy-time gate was the one spot left behind. The zone-level securitygroupenabled flag stays untouched - it selects the legacy shared-only zone mode, which forbids public IP ranges and is exactly what a direct routed zone must not be.
|
LAB environment is working ( will share all LAB details and notes when finished ) and I have the first SystemVMs running. root@v-28-VM:~# ip route get 9.9.9.9
9.9.9.9 via 169.254.0.1 dev eth2 src 2.57.59.69 uid 0
cache
root@v-28-VM:~# ip -6 route get 2001:db8::1
2001:db8::1 from :: via fe80::1 dev eth2 src 2a00:f10:402:4:1c01:76ff:fe00:93 metric 1024 pref medium
root@v-28-VM:~#
root@v-28-VM:~# ip addr show dev eth2
4: eth2: mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 1e:01:76:00:00:93 brd ff:ff:ff:ff:ff:ff
altname enp0s5
altname ens5
inet 2.57.59.69/32 brd 2.57.59.69 scope global eth2
valid_lft forever preferred_lft forever
inet6 2a00:f10:402:4:1c01:76ff:fe00:93/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::1c01:76ff:fe00:93/64 scope link
valid_lft forever preferred_lft forever
root@v-28-VM:~#
Notice the /32 and /128 subnetmask on both IPv4 and IPv6 and the gateways on fe80::1 and 169.254.0.1 It really runs on CloudStack: root@v-28-VM:~# dmidecode memory|grep CloudStack
Product Name: CloudStack KVM Hypervisor
root@v-28-VM:~#
Aiming to have this in version 24.0 and give a demo/talk during CCC 2026. |
Description
This Pull Request adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack.
Management server
ConfigDrive
KVM agent
The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.
This implements issue #12210
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Testing is still ongoing on real hardware and this PR currently (July 2026) exists to gain initial feedback.