Skip to content

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host - #13758

Open
wido wants to merge 12 commits into
apache:mainfrom
wido:direct-routed-network
Open

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host#13758
wido wants to merge 12 commits into
apache:mainfrom
wido:direct-routed-network

Conversation

@wido

@wido wido commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Description

This Pull Request adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack.

Management server

  • GuestType.L3; the guest_type column is char(32), so no schema change.
  • Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared), Dhcp rejected as not supported and not needed. Network mode, specifyVlan and VPC use are rejected.
  • DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. canHandle() selects on the offering's guest type alone; design() produces a Native broadcast domain with no isolation id. After allocation the NicProfile is forced into host-route form, which is also the signature by which the agent and ConfigDrive recognise these NICs.
  • createNetwork treats L3 like Shared for the subnet: explicit IP range mandatory, vlan/IP-range row created at network creation, IPv6 accepted without the /64 restriction, aclType Account.
  • Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide.

ConfigDrive

  • Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR but would leave these NICs with no addressing at all. Route generation itself is unchanged: cloud-init detects an IPv4 gateway inside 169.254.0.0/16 and sets on-link on the rendered route by itself.

KVM agent

  • One uplink-less bridge per network, brdr-, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding and strict rp_filter; separate bridges make isolation between networks topological rather than a filtering concern.
  • BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host route, regardless of the host-wide EVPN property, whose meaning is unchanged.

The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.

This implements issue #12210

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Testing is still ongoing on real hardware and this PR currently (July 2026) exists to gain initial feedback.

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 35.02674% with 243 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.79%. Comparing base (a723d44) to head (7441659).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...cloud/hypervisor/kvm/resource/BridgeVifDriver.java 34.48% 32 Missing and 6 partials ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java 0.00% 38 Missing ⚠️
...om/cloud/network/guru/DirectRoutedNetworkGuru.java 50.68% 20 Missing and 16 partials ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java 3.03% 31 Missing and 1 partial ⚠️
...ain/java/com/cloud/network/NetworkServiceImpl.java 0.00% 20 Missing and 8 partials ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java 58.53% 16 Missing and 1 partial ⚠️
...ud/agent/api/NetworkRulesVmSecondaryIpCommand.java 27.27% 8 Missing ⚠️
...oud/network/security/SecurityGroupManagerImpl.java 0.00% 8 Missing ⚠️
api/src/main/java/com/cloud/network/Networks.java 22.22% 6 Missing and 1 partial ⚠️
...tack/api/command/user/vm/RemoveIpFromVmNicCmd.java 0.00% 5 Missing ⚠️
... and 8 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #13758      +/-   ##
============================================
+ Coverage     19.78%   19.79%   +0.01%     
- Complexity    19992    20047      +55     
============================================
  Files          6371     6373       +2     
  Lines        575903   576323     +420     
  Branches      70496    70602     +106     
============================================
+ Hits         113945   114102     +157     
- Misses       449530   449759     +229     
- Partials      12428    12462      +34     
Flag Coverage Δ
uitests 3.52% <ø> (-0.01%) ⬇️
unittests 21.07% <35.02%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces Direct Routed (L3) guest networks where KVM hosts perform L3 routing directly to instances (no Virtual Router, no NAT, no DHCP), using ConfigDrive-only addressing and a shared link-local gateway (169.254.0.1 / fe80::1) on per-network host bridges.

Changes:

  • Adds GuestType.L3 with server-side offering/network validation and a new DirectRoutedNetworkGuru that forces NICs into host-route form (/32, /128) with link-local gateways.
  • Updates ConfigDrive generation to always emit network_data.json for direct-routed NICs even when DHCP/DNS services are not present.
  • Extends the KVM agent to create/manage per-network bridges (brdr-<networkId>), program host routes/neighbour entries, and adapt security-group rule plumbing for routed traffic (including secondary IP handling).

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
utils/src/main/java/com/cloud/utils/net/NetUtils.java Centralizes link-local gateway/CIDR constants and adds IPv6 link-local gateway getter.
ui/src/views/offering/AddNetworkOffering.vue Adds UI support for L3 offerings and enforces L3-specific service/provider composition.
ui/src/views/network/CreateNetwork.vue Adds an L3 network creation tab.
ui/src/views/network/CreateL3NetworkForm.vue New UI form for creating L3 (Direct Routed) networks.
ui/public/locales/en.json Adds UI strings for L3 offering/network creation messaging.
tools/marvin/marvin/config/test_data.py Adds Marvin test data for L3 offerings and L3 networks.
test/integration/smoke/test_l3_networks.py Adds smoke/integration tests covering L3 network lifecycle and validations.
server/src/test/java/com/cloud/network/guru/DirectRoutedNetworkGuruTest.java Unit tests for the new guru selection/design and NIC host-route forcing.
server/src/test/java/com/cloud/configuration/ConfigurationManagerImplTest.java Unit tests for L3 offering validation rules.
server/src/main/resources/META-INF/cloudstack/server-network/spring-server-network-context.xml Registers the new DirectRoutedNetworkGuru bean.
server/src/main/java/com/cloud/network/security/SecurityGroupManagerImpl.java Ensures agent commands are sent for secondary IPs on L3 networks even when SG rules aren’t applied.
server/src/main/java/com/cloud/network/NetworkServiceImpl.java Extends network creation/IPv6 checks/VLAN handling and secondary-IP orchestration for L3.
server/src/main/java/com/cloud/network/guru/DirectRoutedNetworkGuru.java New guru for L3 direct routed guest networks; forces /32 + /128 form and link-local gateways.
server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java Adds L3 offering validation + zone-wide IPv4 overlap checks for L3 ranges.
scripts/vm/network/vnet/modifymacip.sh Adds targeted delete support for specific -4/-6 addresses (secondary IP remove).
scripts/vm/network/vnet/modifybrdr.sh New script to create/delete per-network brdr-* bridges with sysctls + gateway addresses.
scripts/vm/network/tests/test_security_group.py New unit tests for classic-vs-L3 security_group.py rule streams and --directrouted plumbing.
scripts/vm/network/tests/golden_default_network_rules.txt Golden output for classic rule generation (used to ensure no regression).
scripts/vm/network/tests/golden_add_fw_framework.txt Golden output for classic FW framework creation (ensures byte-identical behavior).
scripts/vm/network/security_group.py Adds routed-aware framework/hooks and rule generation, plus --directrouted CLI plumbing.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtSecurityGroupRulesCommandWrapper.java Passes direct-routed flag when applying SG rules for the VM’s first NIC.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtNetworkRulesVmSecondaryIpCommandWrapper.java Threads direct-routed + apply-SG booleans into secondary-IP handling.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java Adds --directrouted plumbing and direct-routed secondary-IP route/neigh programming via modifymacip.sh.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/BridgeVifDriver.java Creates per-network bridges for direct-routed NICs and ensures MAC/IP script runs for those NICs.
engine/storage/configdrive/src/test/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilderTest.java Adds tests for direct-routed NIC detection and network_data.json generation behavior.
engine/storage/configdrive/src/main/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilder.java Always generates network_data.json for direct-routed NICs; adds NIC signature detection.
docs/design/direct-routed-networks.md Adds comprehensive design document for Direct Routed networks.
core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java Adds directRouted and applySecurityGroupRules flags for secondary-IP commands.
api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java Sends agent update for secondary IP removal on L3 even when zone SG is disabled.
api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java Allows specifying physical network ID for L3 networks.
api/src/main/java/com/cloud/network/Network.java Adds GuestType.L3 and parsing support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1454 to +1461
Network network = _networkModel.getNetwork(nic.getNetworkId());

// On a Direct Routed network the host needs a route and a static neighbour entry for the
// secondary IP before it is reachable at all. That is independent of security groups,
// which are optional there and which the Instance may not be using, so the agent is told
// either way - otherwise the address would stay dark until the Instance was restarted.
boolean directRouted = Network.GuestType.L3.equals(network.getGuestType());

@kiranchavala

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18739

@DaanHoogland DaanHoogland moved this from Backlog to conflict/waiting in CloudStack Testing Aug 31, 2026
@DaanHoogland

Copy link
Copy Markdown
Contributor

@wido this is still marked as wip. what is the status?

@wido

wido commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

@wido this is still marked as wip. what is the status?

The status is that I have real hardware in a datacenter now where I’m going to deploy this code for further testing. Please keep it WIP for now as I want to validate it there.

Comment thread ui/src/views/network/CreateL3NetworkForm.vue Outdated
Comment thread ui/src/views/network/CreateL3NetworkForm.vue Outdated
Comment thread server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java Outdated
wido added a commit to wido/cloudstack that referenced this pull request Sep 7, 2026
Review feedback on apache#13758: the multi-line inline comments narrating the
code are not helpful there. Where a comment documents a real contract
(guru design shape, bridge creation/naming, the MAC-before-IPv6
ordering, ConfigDrive as the only addressing channel, secondary-IP
delivery independent of security groups, the link-local gateway
constants) it now lives as Javadoc on the class, method or field it
describes. Mid-method narration is removed; upstream one-line comments
that this branch merely extended (mandatory start/end IP, vlan-creation
cases, vnet allocation/release) are kept in their original single-line
form. No code changes.
@wido

wido commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

@DaanHoogland fair point on the inline comments — addressed in a911e9c. The multi-line inline blocks are gone throughout the PR: where a comment documented an actual contract it moved into Javadoc on the class/method/field it describes, the mid-method narration is simply removed, and upstream one-liners this PR merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) kept their original single-line form. That covers the three spots you flagged in ConfigurationManagerImpl and CreateL3NetworkForm.vue plus the equivalent blocks elsewhere.

wido added 11 commits September 7, 2026 10:50
… Instances

Adds a new guest network type in which the hypervisor performs L3 routing
for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance
receives a public IPv4 address as a /32 and/or an IPv6 address as a /128,
with a shared, host-independent gateway (169.254.0.1 and fe80::1) that
every hypervisor carries on the network's bridge. All addressing reaches
the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on
the host (FRR, BIRD, ...) advertises the addresses to the fabric and is
deliberately out of scope for CloudStack.

Management server:
- GuestType.L3; the guest_type column is char(32), so no schema change.
- Offering validation: UserData via ConfigDrive is mandatory, Dns optional
  but ConfigDrive-only, SecurityGroup permitted (now allowed for L3
  alongside Shared), Dhcp rejected as not supported and not needed.
  Network mode, specifyVlan and VPC use are rejected.
- DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the
  Shared-network address lifecycle. canHandle() selects on the offering's
  guest type alone; design() produces a Native broadcast domain with no
  isolation id. After allocation the NicProfile is forced into host-route
  form, which is also the signature by which the agent and ConfigDrive
  recognise these NICs.
- createNetwork treats L3 like Shared for the subnet: explicit IP range
  mandatory, vlan/IP-range row created at network creation, IPv6 accepted
  without the /64 restriction, aclType Account.
- Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share
  one host routing table and one fabric, so an overlap is an address
  conflict. The IPv6 vlan check was already zone-wide.

ConfigDrive:
- Network data is always generated for a direct routed NIC; the historical
  gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR
  but would leave these NICs with no addressing at all. Route generation
  itself is unchanged: cloud-init detects an IPv4 gateway inside
  169.254.0.0/16 and sets on-link on the rendered route by itself.

KVM agent:
- One uplink-less bridge per network, brdr-<network id>, created and
  removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove
  a bridge still in use). The bridge carries the gateway addresses,
  forwarding and strict rp_filter; separate bridges make isolation between
  networks topological rather than a filtering concern.
- BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs
  the existing modifymacip.sh hook per NIC to install the static neighbour
  entry and host route, regardless of the host-wide EVPN property, whose
  meaning is unchanged.

The design document, including the decision log and the verification notes
behind each choice, is added under docs/design/.
…adcast domains

Revises the isolation model of Direct Routed (L3) networks. A network is
now direct routed when its offering's guest type is L3 AND it lives on a
dedicated physical network carrying the new isolation method ROUTED --
the network operator's explicit, zone-level opt-in. Every network gets a
broadcast domain of the new type routed://<id>, and that id names the
per-network bridge on the hypervisors: routed://5828 -> brdr-5828. The
id is a label, never an encapsulation; nothing appears on the wire.

The id lifecycle reuses the Shared network's VLAN mechanics verbatim:
with specifyVlan the operator passes the id at network creation through
the existing vlan parameter (bridge names become plannable, so host
routing policy per brdr-<id> can exist before the network does); without
it the id is allocated from the ROUTED physical network's vnet range at
creation and released on deletion. Zone-wide URI overlap checks apply.

Management server:
- BroadcastDomainType.Routed ("routed", Long).
- DirectRoutedNetworkGuru registers IsolationMethod("ROUTED") and follows
  the standard guru contract (isMyIsolationMethod); design() stamps the
  Routed broadcast domain and carries the routed:// URI.
- encodeVlanIdIntoBroadcastUri() maps a ROUTED physical network to
  routed://<id>; the network's broadcast domain type is derived from the
  URI scheme instead of hard-coded Vlan.
- The Shared-without-specifyVlan vnet allocation/release paths extend to
  L3; L3 offerings may carry specifyVlan either way.
- An L3 IPv6 subnet longer than /64 is rejected at network creation:
  addresses are computed with EUI-64 from subnet + MAC (the existing
  setNicIp6Address()/updateNicIpv6() behaviour -- no address pool exists,
  only the allocated result on the NIC), and EUI-64 needs 64 bits.

SystemVMs (CPVM/SSVM), dual-stack from the start:
- PublicNetworkGuru stamps NICs of a routed public range (created with
  vlan=routed://<id>) in host-route form: /32 + 169.254.0.1, /128 +
  fe80::1, Routed broadcast domain. IPv6 comes from the same EUI-64
  computation as everywhere else.
- The CPVM and SSVM boot-arg builders emit eth<N>ip6, eth<N>ip6prelen
  and ip6gateway, as the VR builder already did.
- common.sh marks the v4 default route onlink for link-local gateways,
  and installs a static v6 default from the previously parsed-but-unused
  IP6GW -- on a direct routed bridge no RA ever arrives.

KVM agent:
- BridgeVifDriver recognises direct routed NICs by broadcast type
  instead of inferring from the address form, for guest and public
  traffic alike, and derives the bridge from the NIC's broadcast URI.
  modifybrdr.sh mechanics are unchanged; -n is now the routed id.

UI: ROUTED joins the isolation method choices (zone wizard, physical
network form). The design document records the revision and rationale;
no schema change and no data migration.
Reviewing every path the overhaul touches surfaced four fixes:

- SystemVM public IPv6 is now computed directly in PublicNetworkGuru
  (EUI-64 from the range's ip6_cidr and the NIC MAC, then /128 + fe80::1).
  The previously planned ipv6Service.updateNicIpv6() path is unusable
  here: it is gated on the public network offering's internet protocol
  (never set on the system public offering) and reserves through one
  placeholder NIC per network -- on the shared Public network, every
  SystemVM would have received the same address.
- Guest networks and public ranges share one routed-id space (a routed
  id names a bridge on every host), so collisions between them would
  merge L2 domains. Both directions are now rejected: a guest network
  cannot take an id a public range carries, and a public range cannot
  take an id a guest network holds.
- Creating an L3 network on a physical network without the ROUTED
  isolation method now fails early with a clear message, instead of an
  opaque no-guru error deep in setupNetwork().
- A direct routed guest NIC's isolation URI now mirrors its routed://
  broadcast URI instead of the misleading vlan://<tag> inherited from
  the Shared allocation path.

GuestType.L3 itself was re-verified as still required: some thirty
management-server branches (offering validation, subnet handling,
zone-wide overlap checks, secondary-IP host routes, security groups)
key on the guest type in places where no physical network -- and thus
no isolation method -- is in scope. The guest type says what a network
is; the isolation method says where it may live.
…D message

CreateNetworkCmd.getPhysicalNetworkId() now accepts guest type L3 and its
error message reads 'Shared, L2 or L3 only'; the test still asserted the
old 'Shared or L2 only' text and failed.
…maining UI form

Add a default network offering for Direct Routed (L3) guest networks,
DefaultL3NetworkOffering: UserData and DNS via ConfigDrive, Security
Groups enabled - the only service set validateL3NetworkOffering()
permits. It is created in NetworkOrchestrator.configure() like the
other default offerings, so it appears on fresh installs and on
upgrades alike; until now the operator had to create it by hand.
The routed id is allocated from the ROUTED physical network's range
(no specifyVlan).

UI: the ROUTED isolation method was selectable in the zone wizard and
in the infrastructure listing's add-physical-network form, but missing
from the form on a zone's Physical Networks tab - add it there. Give
the VLAN, VXLAN and ROUTED options hover descriptions explaining what
each isolation method does, so operators understand what they are
opting into; ROUTED's spells out the Direct Routed model (no VLANs on
the wire, no Virtual Router, no DHCP, addressing via ConfigDrive, L3
offerings only, KVM only).
Creating an L3 network died in NetworkOrchestrator.createGuestNetwork():
after encodeVlanIdIntoBroadcastUri() produced routed://<id>, the
unconditional PVLAN overlap check handed that URI to
listByPhysicalNetworkPvlan(), which rejects every URI scheme other than
vlan:// and vxlan://:

  'Requested URI routed://189 is not in the expected format. Expected
   URI Scheme as vlan://VID or vxlan://VID.'

The check is meaningless for a routed id: no PVLAN network can exist on
a ROUTED physical network (only L3 networks can), and the id's real
overlap checks - zone-wide URI, public ranges, the vnet range - have
already run earlier in the method. Skip it when the URI scheme is
routed.

Also reject the isolatedpvlan parameter for L3 networks explicitly, so
the pvlan branch (which would build a pvlan:// URI around the routed id)
is unreachable for this guest type.
CreateL3NetworkForm only supported offerings that allocate the routed id
automatically; an operator-chosen id (the API's vlan parameter, like a
Shared network's VLAN tag) could be given through the API alone. Add a
'Routed ID' field to the form, shown to root admins when the selected
L3 offering carries specifyVlan - where the API requires the id - and
hidden otherwise, since the API then rejects the parameter and
allocates the id from the ROUTED physical network's range.

The field's description explains what the id is: a label naming the
per-network bridge (brdr-<id>) on every KVM host, unique per zone,
nothing on the wire, and useful to choose in advance so host routing
policy can exist before the network does.
…t range check

Creating an L3 network whose offering does not carry specifyVlan failed
in ConfigurationManagerImpl.createVlanAndPublicIpRange():

  'The VLAN tag 573 is already being used for dynamic vlan allocation
   for the guest network in zone AMS02'

even though the id showed as Free. The check calls findVnet(), which
matches any op_dc_vnet_alloc row with that number - taken or free - so
it really asks 'does this tag lie inside a dynamic allocation range',
not 'is it in use'. An auto-allocated routed id always comes from the
ROUTED physical network's vnet range (commitNetwork() takes it via
allocateVnet() moments earlier), so the check rejected every such
network; only bypassvlanoverlapcheck=true got past it, and the rollback
released the id again, which is why it kept showing as Free.

Shared networks with specifyVlan=false have long been exempt from this
check for exactly this reason (isSharedNetworkWithoutSpecifyVlan forces
bypassVlanOverlapCheck). Give L3 networks the same exemption: expose
the orchestrator's isL3NetworkWithoutSpecifyVlan() helper on
NetworkOrchestrationService and OR it into the bypass condition.

The operator-specified path is unchanged: with specifyVlan the given
routed id still must not collide with a vnet range, as for Shared.
…EUI-64 IPv6

Deploying a SystemVM on a direct routed public range died with

  NullPointerException: Cannot invoke "String.toLowerCase()" because
  "macAddress" is null

in ConsoleProxyManagerImpl/SecondaryStorageManagerImpl. In
PublicNetworkGuru.getIp() the routed branch calls setRoutedRangeIpv6(),
which derives the NIC's IPv6 address with EUI-64 from
nic.getMacAddress() - but the MAC was only copied onto the profile at
the end of the block, after the branch, so NetUtils.EUI64Address() ran
against a null MAC on the freshly created NicProfile.

The same ordering hid a second bug: the unconditional
nic.setFormat(AddressFormat.Ip4) after the branch clobbered the
DualStack format setRoutedRangeIpv6() had just set, so even with a MAC
the NIC would have come out IPv4-only.

Move the format/reservationId/MAC assignments above the branch chain.
The vlan and vxlan paths never touch MAC or format, so they are
unaffected. Add PublicNetworkGuruTest covering the routed shape
(host-route IPv4, EUI-64 IPv6 from the NIC MAC, DualStack) and the
classic VLAN shape as regression tests.
Starting a SystemVM on a direct routed public range failed on the KVM
agent:

  Failed to create vnet 600: Cannot find device "br-600" ...
  Failed to create vlan 600 on pif: br-600.

The agent's dispatch, BridgeVifDriver.isDirectRoutedNic(), checked only
the NicTO's broadcast *type* enum. That type is not persisted per nic:
at VM start the NicProfile is rebuilt and copies broadcastType from the
*network* (NicProfile line 88), and the shared Public network's
broadcast domain type is Vlan. The routed:// stamp survives only in the
nic row's broadcast URI. The public NIC therefore arrived as
type=Vlan, uri=routed://600: the routed branch in plug() was skipped,
the Public/VLAN branch parsed 600 out of the routed URI and ran
modifyvlan.sh against a nonexistent pif - and the modifymacip.sh call,
gated on the same predicate, never happened at all. Guest NICs were
unaffected because an L3 network row genuinely carries the Routed
domain type.

Recognise the routed:// URI scheme as well as the type enum - matching
the predicate's own javadoc, which already described recognition by
broadcast domain. One predicate serves every agent-side call site
(plug, unplug, modifymacip gate, security group wrapper). Add
regression tests for both recognitions and the ordinary-NIC negatives.
Review feedback on apache#13758: the multi-line inline comments narrating the
code are not helpful there. Where a comment documents a real contract
(guru design shape, bridge creation/naming, the MAC-before-IPv6
ordering, ConfigDrive as the only addressing channel, secondary-IP
delivery independent of security groups, the link-local gateway
constants) it now lives as Javadoc on the class, method or field it
describes. Mid-method narration is removed; upstream one-line comments
that this branch merely extended (mandatory start/end IP, vlan-creation
cases, vnet allocation/release) are kept in their original single-line
form. No code changes.
@wido
wido force-pushed the direct-routed-network branch from a911e9c to 655530c Compare September 7, 2026 14:02
… networks

Deploying an Instance with securitygroupids into an L3 (Direct Routed)
network failed with

  'Can't create vm with security groups; security group feature is not
   enabled per zone'

The message is misleading: the zone flag is only the first half of the
gate. checkSecurityGroupSupportForNetwork() also accepts a deploy when
one of the requested networks itself supports the SecurityGroup service
- but that check compared the guest type against Shared only, so an L3
network carrying the SecurityGroup service (as DefaultL3NetworkOffering
does) never qualified and the deploy fell through to the error.

Accept GuestType.L3 alongside Shared. Every other security group
touchpoint already treats L3 like Shared (offering validation, the
secondary-IP paths, the security_group.py dispatch); this deploy-time
gate was the one spot left behind. The zone-level securitygroupenabled
flag stays untouched - it selects the legacy shared-only zone mode,
which forbids public IP ranges and is exactly what a direct routed zone
must not be.
@wido

wido commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

LAB environment is working ( will share all LAB details and notes when finished ) and I have the first SystemVMs running.

root@v-28-VM:~# ip route get 9.9.9.9
9.9.9.9 via 169.254.0.1 dev eth2 src 2.57.59.69 uid 0
    cache
root@v-28-VM:~# ip -6 route get 2001:db8::1
2001:db8::1 from :: via fe80::1 dev eth2 src 2a00:f10:402:4:1c01:76ff:fe00:93 metric 1024 pref medium
root@v-28-VM:~#
root@v-28-VM:~# ip addr show dev eth2
4: eth2:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1e:01:76:00:00:93 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 2.57.59.69/32 brd 2.57.59.69 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 2a00:f10:402:4:1c01:76ff:fe00:93/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::1c01:76ff:fe00:93/64 scope link
       valid_lft forever preferred_lft forever
root@v-28-VM:~#

Notice the /32 and /128 subnetmask on both IPv4 and IPv6 and the gateways on fe80::1 and 169.254.0.1

It really runs on CloudStack:

root@v-28-VM:~# dmidecode memory|grep CloudStack
        Product Name: CloudStack KVM Hypervisor
root@v-28-VM:~#

Aiming to have this in version 24.0 and give a demo/talk during CCC 2026.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: conflict/waiting

Development

Successfully merging this pull request may close these issues.

5 participants