Add ControllerGetNodeInfo RPC (alpha)

[huww98]

What type of PR is this?

/kind feature

What this PR does / why we need it:

See KEP-6011: CSI ControllerGetNodeInfo for details

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce an API-breaking change?:

Add ControllerGetNodeInfo RPC and corresponding new field in NodeGetInfo (alpha)

[jdef]

There's a lot of k8s stuff in this PR desc, probably better suited for a KEP that this PR might reference. This (CSI spec) is not a subproject of k8s, it's CO agnostic, or at least it tries to be. Appreciate the context though. Reading this through it's not immediately clear to me why this isn't solveable some other way- what alternatives were considered and why were they not sufficient? James DeFelice

…

On Sat, Mar 21, 2026, 10:27 AM 胡玮文 ***@***.***> wrote: *What type of PR is this?* /kind feature *What this PR does / why we need it*: Proposal: Add ControllerGetNodeInfo and NodeGetID RPCs Motivation Some user have strict security requirements that prohibit distributing cloud API credentials to node components. However, the current NodeGetInfo RPC requires cloud API access to retrieve: 1. *Topology information* (zone, region, supported disk categories) 2. *max_volumes_per_node* (requires querying currently attached disks that may not be managed by CSI) This proposal introduces two new RPCs to enable node registration without requiring cloud API credentials on the node side. Proposed Changes New RPCs 1. NodeGetID (Node Service) - *Input*: None - *Output*: node_id (e.g., cloud instance ID) - *Purpose*: Returns only the node identifier, which can be obtained locally (e.g., from instance metadata service) without cloud API credentials. 2. ControllerGetNodeInfo (Controller Service) - *Input*: node_id, published_volume_ids (volumes CO believes are published, including uncertain status) - *Output*: accessible_topology, max_volumes_per_node - *Purpose*: Fetches node topology and capacity information from the controller side, where cloud API credentials are already available. New Capabilities - NodeServiceCapability.RPC.GET_ID - Indicates support for NodeGetID - ControllerServiceCapability.RPC.GET_NODE_INFO - Indicates support for ControllerGetNodeInfo Kubernetes Integration The following diagram illustrates how these new RPCs integrate with Kubernetes components (proposed changes to external-attacher sidecar): sequenceDiagram box rgba(255,0,0,0.1) Node Side participant kubelet participant csi-node as CSI Node Plugin end box Controller Side participant attacher as external-attacher participant csi-ctrl as CSI Controller Plugin participant scheduler as Scheduler end attacher->>attacher: Watch CSINode/VolumeAttachment kubelet->>+csi-node: NodeGetCapabilities csi-node-->>-kubelet: GET_ID capability attacher->>+csi-ctrl: ControllerGetCapabilities csi-ctrl-->>-attacher: GET_NODE_INFO capability kubelet->>+csi-node: NodeGetID csi-node->>csi-node: Get instance ID from metadata csi-node-->>-kubelet: node_id = instance-xxx kubelet->>attacher: Annotate CSINode with node_id attacher->>+csi-ctrl: ControllerGetNodeInfo(node_id, published_volume_ids) csi-ctrl->>csi-ctrl: Query cloud APIs for topology & capacity csi-ctrl-->>-attacher: topology, max_volumes_per_node attacher->>scheduler: Update CSINode (remove annotation) Note over attacher,csi-ctrl: Volume limit reached scenario attacher->>+csi-ctrl: ControllerPublishVolume csi-ctrl-->>-attacher: RESOURCE_EXHAUSTED attacher->>+csi-ctrl: ControllerGetNodeInfo(node_id, published_volume_ids) csi-ctrl-->>-attacher: Updated max_volumes_per_node attacher->>kubelet: Update CSINode kubelet->>kubelet: Fail affected Pods Loading Why Integrate with external-attacher? 1. *Reuses existing watches*: external-attacher already watches VolumeAttachment resources, providing the list of attached volumes needed for accurate max_volumes_per_node calculation. 2. *Consistency guarantee*: The attacher can pause attach/detach operations during ControllerGetNodeInfo calls to ensure consistent results. *Note*: If the SP does not support PUBLISH_UNPUBLISH_VOLUME, the ControllerGetNodeInfo RPC can still be used to fetch static topology information at node registration time with empty published_volume_ids. Backward Compatibility - Both RPCs are marked as *alpha* (option (alpha_method) = true) - Existing NodeGetInfo RPC remains unchanged - COs can detect support via capability discovery: - If both GET_ID (node) and GET_NODE_INFO (controller) are supported, use the new flow - Otherwise, fall back to the traditional NodeGetInfo flow Required Changes in Other Components This CSI spec change requires corresponding updates in the following Kubernetes components: 1. kubelet (kubernetes/kubernetes) - Add support for calling NodeGetID RPC when the node plugin advertises GET_ID capability - When NodeGetID is used instead of NodeGetInfo, kubelet should: - Store the node_id in CSINode annotation (e.g., csi.volume.kubernetes.io/nodeid.{driver} <http://csi.volume.kubernetes.io/nodeid.%7Bdriver%7D>) - NOT populate topology and allocatable count in CSINode (leave for external-attacher) - Maintain backward compatibility: use NodeGetInfo if GET_ID is not supported 2. external-attacher (kubernetes-csi/external-attacher) - Watch for CSINode annotations indicating nodes that need ControllerGetNodeInfo - Call ControllerGetNodeInfo when: - A new node registers with only node_id (no topology) - RESOURCE_EXHAUSTED error is returned from ControllerPublishVolume - Update CSINode with topology and allocatable count from ControllerGetNodeInfo response - Coordinate with attach/detach operations to ensure consistent published_volume_ids list Security Benefits - Node components no longer require cloud API credentials - Reduced attack surface on nodes - Cloud credentials are centralized in the controller component - Meets strict security baseline requirements for sensitive environments Example Use Case (Alibaba Cloud) For Alibaba Cloud ECS: - NodeGetID returns the ECS instance ID (obtainable from instance metadata at http://100.100.100.200/latest/meta-data/instance-id) - ControllerGetNodeInfo queries: - Zone ID and Region ID via DescribeInstances API - Supported disk categories via DescribeAvailableResource API - Current disk attachments via DescribeDisks API (to detect disks not managed by CSI) - Total attachable disk count via DescribeInstanceTypes API *Which issue(s) this PR fixes*: Fixes # *Special notes for your reviewer*: *Does this PR introduce an API-breaking change?*: Add ControllerGetNodeInfo and NodeGetID RPCs (alpha) ------------------------------ You can view, comment on, or merge this pull request online at: #603 Commit Summary - a727d32 <a727d32> Add ControllerGetNodeInfo and NodeGetID RPCs (alpha) File Changes (4 files <https://github.com/container-storage-interface/spec/pull/603/files>) - *M* csi.proto <https://github.com/container-storage-interface/spec/pull/603/files#diff-e099a7ef79268152caab1065df9864ec07353456cad2199895c2c89601f23287> (94) - *M* lib/go/csi/csi.pb.go <https://github.com/container-storage-interface/spec/pull/603/files#diff-da7481db22d3f8409112d2ce13ff879eff4c225908a3cf4a3f949b4b1386890b> (2996) - *M* lib/go/csi/csi_grpc.pb.go <https://github.com/container-storage-interface/spec/pull/603/files#diff-c3ffbc06c3e8b5baaff813a111e0e6f0dadc7677a4d1ae2903197050b5931083> (74) - *M* spec.md <https://github.com/container-storage-interface/spec/pull/603/files#diff-bc6661da34ecae62fbe724bb93fd69b91a7f81143f2683a81163231de7e3b545> (143) Patch Links: - https://github.com/container-storage-interface/spec/pull/603.patch - https://github.com/container-storage-interface/spec/pull/603.diff — Reply to this email directly, view it on GitHub <#603?email_source=notifications&email_token=AAR5KLC7MOMVE475VLZREQ34R2RG5A5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTIMRYGY3DEOJUGKTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6Y3MNFRWW>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR5KLBLQBRNWKPNM4XBKWT4R2RG5AVCNFSM6AAAAACW2CHXVWVHI2DSMVQWIX3LMV43ASLTON2WKOZUGEYTENZQGUZTMOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[huww98]

There's a lot of k8s stuff in this PR desc, probably better suited for a
KEP that this PR might reference.

Yes, I will propose this to Kubernetes if it looks fine on the CSI side.
I provide them here for additional context.

what alternatives were considered and why were
they not sufficient?

There are not many alternatives actually.

The instance metadata service (e.g., http://169.254.169.254 or http://100.100.100.200) typically only provides basic info like instance-id and zone. It does not provide disk attachment limits or supported disk categories, which require cloud API calls.

The information required by NodeGetInfo is not fetchable from node with the security requirement. We have considered adding a new private CRD and a new controller in Kubernetes. The controller watches for new node, fetches the required info and store them in a CR, then node watches for the CR to get the info. But this still have major drawbacks:

• The permission to invoke Cloud API is replaced with a service account with CR read permission, which does not fundamentally solve the security issue.
• Requires more network roundtrips
  • This proposal: Cloud -> csi-controller -> external-attacher -> APIServer(CSINode) -> scheduler
  • With CRD: Cloud -> new controller -> APIServer(CR) -> csi-node -> kubelet -> APIServer(CSINode) -> scheduler
• Even harder considering the new capability to fetch max_volumes_per_node immediately after publish failure. Will need some way for node to tell controller to do an update immediately, which requires write permission, making things even worse.
• This is Kubernetes specific, not resolving the security requirement for other CO.

The core insight is: max_volumes_per_node and accessible_topology is consumed by scheduler, not node. So this piece of infomation should not appear on node for optimal security, hence this proposal.

[bswartz]

I don't see any point to the NodeGetID RPC. NodeGetInfo already returns a superset of the information.

[huww98]

I design NodeGetID as a per-node switch for this feature, so that we can upgrade controller first, then upgrade nodes gradually: if node reports GET_ID cap, we go for the new workflow, otherwise we go for the old flow. We can also safely install new CSI driver (with new sidecars) in old Kubernetes cluster this way: old kubelet don't call NodeGetID, then new sidecar should not call ControllerGetNodeInfo, even if both sidecar and CSI driver support this.

As just discussed in the meeting, an operator may choose to remove credential from node side, so that NodeGetInfo can no longer retrieve topology info. I think it would be better for SP to fail NodeGetInfo instead of returning incomplete info.

Alternative: Adding a new field in NodeGetInfoResponse (say incomplete) as the switch stated above. But for old CO (kubelet) that don't understand this field, it may silently accept the incomplete topology or capacity, leading to confusing scheduling result. Also we may need to consider what to do if node and controller returns conflicting topology.

For the periodical update scalability issue, I take the number from https://kubernetes.io/docs/setup/best-practices/cluster-large/:

No more than 150,000 total pods
No more than 300,000 total containers

if every container have a volume, we can have 300,000 volumes in the cluster (we didn't tested this large scale). Our DescribeDisks API call can return at most 500 disks per-call. The default rate limit is 1000/60s in a region. So we can actually finish the query of the whole cluster in less than 1 min. So I think this is fine.

I can see that it could be hard for SPs that want to determine topology at controller side but capacity from node side. But I haven't come up with a better idea though.

[torredil]

@huww98 This has a merge conflict now, please rebase when you get a chance. Feel free to ping when its updated and reviewable.

[torredil]

Largely lgtm, will take another look after its rebased.

[gnufied]

This is something again I mentioned in the KEP, a driver that choses to implement ControllerGetNodeInfo without implemeneting/supporting controller-publish of volumes will force users to deploy external-attacher.

So, question is - should drivers that don't support controller-publish of volume still implement this capability.

[torredil]

+1, in Kubernetes a plugin without PUBLISH_UNPUBLISH_VOLUME that advertises NODE_INFO_FROM_CONTROLLER would actually end up with no topology and no limit since external-attacher's trivial handler only exists to mark VA objects as attached and has no CSI connection.

I think the cleanest fix is to require that a plugin with GET_NODE_INFO also supports PUBLISH_UNPUBLISH_VOLUME.

[huww98]

will force users to deploy external-attacher

This is a Kubernetes issue, right? I don't see anything inconsistent in the CSI layer.

Kubernetes a plugin without PUBLISH_UNPUBLISH_VOLUME that advertises NODE_INFO_FROM_CONTROLLER would actually end up with no topology and no limit

I didn't get this. This seems also a implementation issue? I think from CSI level it is fine for CO to call ControllerGetNodeInfo without ControllerPublishVolume to retrieve real topology and attach limit.

should drivers that don't support controller-publish of volume still implement this capability

My call: go implement this if it is useful for that driver. Kubernetes will additionally check for CSIDriver.spec.attachRequired before actually using it, ensuring we have external-attacher deployed and VolumeAttachments created for the driver. No matter whether the driver supports controller-publish or not.

the cleanest fix is to require that a plugin with GET_NODE_INFO also supports PUBLISH_UNPUBLISH_VOLUME.

If we really want to go this way, I think we need to require a plugin to not advertise NODE_INFO_FROM_CONTROLLER if it does not support PUBLISH_UNPUBLISH_VOLUME. Since CO needs to check it on node side to determine which flow to go with. But I cannot find a good reason for this requirement.

[bswartz]

Requiring SPs to support also PUBLISH_UNPUBLISH_VOLUME if they want to support NODE_INFO_FROM_CONTROLLER is not a significant implementation burden. Those operations can be no-ops and the CO will work just fine (NetApp actually did this for some time).

Do we have an example of a driver (hypothetical or real) which would need the NODE_INFO_FROM_CONTROLLER capability and not need the PUBLISH_UNPUBLISH_VOLUME capability? That would be a good use case to focus on if we want to argue to not make the capability dependent.

[huww98]

Requiring SPs to support also PUBLISH_UNPUBLISH_VOLUME if they want to support

We now require SPs to support also PUBLISH_UNPUBLISH_VOLUME in the latest version of this PR.
This makes things simpler anyway. We can reconsider when we really see a use-case.

Those operations can be no-ops

Exactly. But on the other hand, that seems to mean CO can just pretend SP has a no-op publish operation if PUBLISH_UNPUBLISH_VOLUME is not advertised? Kubernetes external-attacher already doing this with its trivial handler.

Do we have an example of a driver (hypothetical or real)

I don't have a real use-case.
For hypothetical one, we are looking for network attached storage (no controller publish needed). Those drivers typically don't have volume count limitation, but topology constriant is possible.
For example. ceph-csi do support topology, it fetches Kubernetes node object for labels, which requires RBAC permission in daemonset. This step can be moved to controller side theoretically to remove credential from node. Note ceph-csi actually supports PUBLISH_UNPUBLISH_VOLUME, so it is not a direct example.

[bswartz]

From a security perspective, SPs that don't do controller publish can't achieve both (1) no sensitive secrets on the node AND (2) protection from attacks by malicious nodes, so I'm not sure what the point of moving the topology information from the node to the controller would be. They can do topology on the node like they do now.

The bigger concern is whether to run the external-attacher sidecar, because that's a noticeable burden, but to enable this feature they're going to have to run some additional sidecar, and it may as well be the external-attacher with possibly a no-op controllerpublish/unpublish implementation. Is anyone proposing a different sidecar to consume the new RPC?

[huww98]

I'm not sure what the point of moving the topology information from the node to the controller would be. They can do topology on the node like they do now.

Yes, I agree for these drivers moving NodeGetInfo to controller has little benefit.

But we have NodePublishVolumeRequest.secrets. Those secrets can be scoped to volumes scheduled to the node, and not present on file-system. It should be slightly safer than a global permission from Kubernetes projected volume mounted to csi plugin daemonset.

Is anyone proposing a different sidecar to consume the new RPC?

The published_volume_ids feature can only work from external-attacher. It needs the in-memory state of currently processing volumes to avoid race condition.
And it seems only external-attacher caring about the node_id and CSINode object. I think it is a perfect fit.

[gnufied]

IMO - I would avoid being too prescriptive here about what to do with returned published_volume_ids. It is fine to put this under an example, but I would avoid making it too prescriptive at this stage.

[huww98]

Removed "The CO MUST NOT use these volume IDs for any purpose other than slot accounting".

Other sentences are necessary IMO. SP generally needs an idea on how CO will behaviour with the IDs. And it still leaves room for implementation. It only say combine, not how.
But I'm not sure I get the meaning of "prescriptive".

I also added an example below.

[gnufied]

The CO SHALL combine these volumes with its own records (deduplicating as needed) when calculating available slots.

SHALL is basically same as MUST and I wonder if that is suitable language in CSI spec.

How about following (feel free to rephrase as you see fit):

The SP MAY additionally populate published_volume_ids with the IDs of volumes currently published to the node that count toward max_volumes_per_node, including volumes attached out-of-band, such as a root disk or manually attached volumes. The CO MAY use this information to determine effective maximum number of volumes it allows to publish on this node.

[huww98]

Also added something I learned when writing the KEP about the race condition in this round:

The SP MAY include or exclude volumes being published or unpublished during the ControllerGetNodeInfo call.
The CO MUST be resilient to that.
For ControllerGetNodeInfo calls that start after ControllerUnpublishVolume succeeds, the SP MUST exclude the volume from published_volume_ids for each affected node, unless a new publish operation for that volume on that node has since started.