v3.1.1: Error: empty List of Node VMs returned from nodeManager

[Sispheor]

Hello there. I'm trying to install the plugin on K8S

kubectl version

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

Here are the csi-provisioner log when creting a testing pvc:

I1117 13:23:29.275496       1 feature_gate.go:249] feature gates: &{map[]}
I1117 13:23:29.275827       1 csi-provisioner.go:154] Version: v3.5.0
I1117 13:23:29.275894       1 csi-provisioner.go:177] Building kube configs for running in cluster...
I1117 13:23:29.278853       1 common.go:111] Probing CSI driver for readiness
I1117 13:23:29.282864       1 csi-provisioner.go:230] Detected CSI driver csi.vsphere.vmware.com
I1117 13:23:29.282886       1 csi-provisioner.go:240] Supports migration from in-tree plugin: kubernetes.io/vsphere-volume
I1117 13:23:29.283644       1 common.go:111] Probing CSI driver for readiness
I1117 13:23:29.288698       1 csi-provisioner.go:299] CSI driver supports PUBLISH_UNPUBLISH_VOLUME, watching VolumeAttachments
I1117 13:23:29.290549       1 controller.go:732] Using saving PVs to API server in background
I1117 13:23:29.294208       1 leaderelection.go:245] attempting to acquire leader lease vmware-system-csi/csi-vsphere-vmware-com...
I1117 13:23:29.305028       1 leaderelection.go:349] lock is held by 1700221067888-8081-csi-vsphere-vmware-com and has not yet expired
I1117 13:23:29.305164       1 leaderelection.go:250] failed to acquire lease vmware-system-csi/csi-vsphere-vmware-com
I1117 13:23:29.305263       1 leader_election.go:185] new leader detected, current leader: 1700221067888-8081-csi-vsphere-vmware-com
I1117 13:24:11.895833       1 leaderelection.go:349] lock is held by 1700221067888-8081-csi-vsphere-vmware-com and has not yet expired
I1117 13:24:11.895854       1 leaderelection.go:250] failed to acquire lease vmware-system-csi/csi-vsphere-vmware-com
I1117 13:25:00.822487       1 leaderelection.go:349] lock is held by 1700221067888-8081-csi-vsphere-vmware-com and has not yet expired
I1117 13:25:00.822585       1 leaderelection.go:250] failed to acquire lease vmware-system-csi/csi-vsphere-vmware-com
I1117 13:25:54.698151       1 leaderelection.go:255] successfully acquired lease vmware-system-csi/csi-vsphere-vmware-com
I1117 13:25:54.698409       1 leader_election.go:185] new leader detected, current leader: 1700227409288-6523-csi-vsphere-vmware-com
I1117 13:25:54.698549       1 leader_election.go:178] became leader, starting
I1117 13:25:54.708374       1 reflector.go:287] Starting reflector *v1.StorageClass (1h0m0s) from k8s.io/client-go/informers/factory.go:150
I1117 13:25:54.708592       1 reflector.go:323] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:150
I1117 13:25:54.709298       1 reflector.go:287] Starting reflector *v1.PersistentVolumeClaim (15m0s) from k8s.io/client-go/informers/factory.go:150
I1117 13:25:54.709403       1 reflector.go:323] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:150
I1117 13:25:54.714388       1 reflector.go:287] Starting reflector *v1.VolumeAttachment (1h0m0s) from k8s.io/client-go/informers/factory.go:150
I1117 13:25:54.714406       1 reflector.go:323] Listing and watching *v1.VolumeAttachment from k8s.io/client-go/informers/factory.go:150
I1117 13:25:54.803235       1 shared_informer.go:341] caches populated
I1117 13:25:54.803269       1 shared_informer.go:341] caches populated
I1117 13:25:54.803274       1 shared_informer.go:341] caches populated
I1117 13:25:54.803328       1 controller.go:811] Starting provisioner controller csi.vsphere.vmware.com_vsphere-csi-controller-675dfc79fd-6hmtt_e498f0cc-43c5-4a43-a38c-12bc53f581dd!
I1117 13:25:54.803408       1 volume_store.go:97] Starting save volume queue
I1117 13:25:54.806268       1 reflector.go:287] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v9/controller/controller.go:845
I1117 13:25:54.806374       1 reflector.go:323] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/v9/controller/controller.go:845
I1117 13:25:54.806284       1 reflector.go:287] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v9/controller/controller.go:848
I1117 13:25:54.806458       1 reflector.go:323] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/v9/controller/controller.go:848
I1117 13:25:54.903835       1 shared_informer.go:341] caches populated
I1117 13:25:54.904376       1 controller.go:1359] provision "default/example-pvc" class "thin": started
I1117 13:25:54.904577       1 controller.go:860] Started provisioner controller csi.vsphere.vmware.com_vsphere-csi-controller-675dfc79fd-6hmtt_e498f0cc-43c5-4a43-a38c-12bc53f581dd!
I1117 13:25:54.904654       1 controller.go:568] skip translation of storage class for plugin: csi.vsphere.vmware.com
I1117 13:25:54.906390       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"example-pvc", UID:"e3543deb-ce1d-4b4b-a20e-36dced73d2fb", APIVersion:"v1", ResourceVersion:"15711", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/example-pvc"
I1117 13:25:55.274373       1 controller.go:1075] Final error received, removing PVC e3543deb-ce1d-4b4b-a20e-36dced73d2fb from claims in progress
W1117 13:25:55.274487       1 controller.go:934] Retrying syncing claim "e3543deb-ce1d-4b4b-a20e-36dced73d2fb", failure 0
E1117 13:25:55.274586       1 controller.go:957] error syncing claim "e3543deb-ce1d-4b4b-a20e-36dced73d2fb": failed to provision volume with StorageClass "thin": rpc error: code = Internal desc = failed to get shared datastores in kubernetes cluster. Error: empty List of Node VMs returned from nodeManager
I1117 13:25:55.275785       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"example-pvc", UID:"e3543deb-ce1d-4b4b-a20e-36dced73d2fb", APIVersion:"v1", ResourceVersion:"15711", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "thin": rpc error: code = Internal desc = failed to get shared datastores in kubernetes cluster. Error: empty List of Node VMs returned from nodeManager

In the logs of the controller I can see my datastore URLs well detected

(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:922"}
{"level":"info","time":"2023-11-17T13:28:21.26213202Z","caller":"k8sorchestrator/topology.go:230","msg":"Refreshing preferred datastores information...","TraceId":"708ddc10-7153-4686-a07c-25041dfeb632"}
{"level":"info","time":"2023-11-17T13:28:21.262724449Z","caller":"vsphere/utils.go:265","msg":"Defaulting timeout for vCenter Client to 5 minutes","TraceId":"708ddc10-7153-4686-a07c-25041dfeb632"}
{"level":"info","time":"2023-11-17T13:28:21.653155384Z","caller":"vsphere/utils.go:409","msg":"New tag manager with useragent 'k8s-csi-useragent-talos-test'","TraceId":"708ddc10-7153-4686-a07c-25041dfeb632"}
{"level":"info","time":"2023-11-17T13:28:22.020486331Z","caller":"common/authmanager.go:163","msg":"auth manager: datastoreMapForBlockVolumes is updated to map[ds:///vmfs/volumes/61a0a555-9a7c5a66-5bcb-d4f5ef1af690/:Datastore: Datastore:datastore-6893, datastore URL: ds:///vmfs/volumes/61a0a555-9a7c5a66-5bcb-d4f5ef1af690/ ds:///vmfs/volumes/61a0a582-0625fd0e-50f9-d4f5ef180ec0/:Datastore: Datastore:datastore-6882, datastore URL: ds:///vmfs/volumes/61a0a582-0625fd0e-50f9-d4f5ef180ec0/ ds:///vmfs/volumes/61a0a596-7ea7499a-d809-d4f5ef180ef0/:Datastore: Datastore:datastore-6872, datastore URL: ds:///vmfs/volumes/61a0a596-7ea7499a-d809-d4f5ef180ef0/ ds:///vmfs/volumes/62a064d3-bd672cc2-e2f8-d4f5ef1af690/:Datastore: Datastore:datastore-6894, datastore URL: ds:///vmfs/volumes/62a064d3-bd672cc2-e2f8-d4f5ef1af690/ ds:///vmfs/volumes/62a06e31-00e55a00-881d-d4f5ef180ec0/:Datastore: Datastore:datastore-6883, datastore URL: ds:///vmfs/volumes/62a06e31-00e55a00-881d-d4f5ef180ec0/ ds:///vmfs/volumes/62a075b8-02c63d90-bfc1-b47af10bde6a/:Datastore: Datastore:datastore-6873, datastore URL: ds:///vmfs/volumes/62a075b8-02c63d90-bfc1-b47af10bde6a/ ds:///vmfs/volumes/62ba2ed7-ffbb59d6-8d7f-3e591300016a/:Datastore: Datastore:datastore-2046, datastore URL: ds:///vmfs/volumes/62ba2ed7-ffbb59d6-8d7f-3e591300016a/ ds:///vmfs/volumes/62ba3703-013ff20e-69e4-3e5913000170/:Datastore: Datastore:datastore-2056, datastore URL: ds:///vmfs/volumes/62ba3703-013ff20e-69e4-3e5913000170/ ds:///vmfs/volumes/62

My setup

CPI

Installed cloud-provider-vsphere v1.28 with manifest

vsphere-cloud-config config map

 kind: ConfigMap
apiVersion: v1
metadata:
  name: vsphere-cloud-config
  namespace: kube-system
  labels:
    component: cloud-controller-manager
    vsphere-cpi-infra: config
data:
  vsphere.conf: |2
     # Global properties in this section will be used for all specified vCenters unless overriden in VirtualCenter section.
        global:
          port: 443
          # set insecureFlag to true if the vCenter uses a self-signed cert
          insecureFlag: true
          # settings for using k8s secret
          #secretName: vsphere-cloud-secret
          #secretNamespace: kube-system

        # vcenter section
        vcenter:
          my-vc:
            server: myfqdn
            user: administrator@vsphere.local
            password: very_secret
            datacenters:
              - Cluster4Lab

Providers are well added to nodes

kubectl describe nodes | grep "ProviderID"
+ kubectl describe nodes
ProviderID:                   vsphere://42243af9-0e6b-013f-7040-a81d1e29e099
ProviderID:                   vsphere://42242b67-b1f9-9521-8a18-615de9e44bc6
ProviderID:                   vsphere://4224b768-a024-878b-a952-19b21e319c2a
ProviderID:                   vsphere://42243501-69a8-b252-7024-ade709e5ed7d
ProviderID:                   vsphere://4224b3e5-32de-2581-9658-7a05765fe989

CSI

vsphere-config-secret secret (I use admin credentials to avoid any perm issue):

kind: Secret
apiVersion: v1
metadata:
  name: vsphere-config-secret
  namespace: vmware-system-csi
data:
  csi-vsphere.conf: |
    [Global]
    cluster-id = "{{ talos_cluster_name }}"
    insecure-flag = "1"

    [VirtualCenter "{{ vcenter_hostname }}"]
    datacenters = "{{ vcenter_datacenter }}"
    user = "{{ vcenter_username }}"
    password = "{{ vcenter_password }}"

Deployed manifest 3.1.1

Finally I created a storage class with my datastore URL (a local datastore on an esxi where all VMs of the cluster are placed)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: thin
parameters:
  datastoreurl: ds:///vmfs/volumes/62c22699-65fbf5a0-6453-d4f5ef3701bc/
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Created my testing PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: thin

And the PVC is not created (Error: empty List of Node VMs returned from nodeManager):

oc -n default describe pvc example-pvc
Name:          example-pvc
Namespace:     default
StorageClass:  thin
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
               volume.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                  From                                                                                                 Message
  ----     ------                ----                 ----                                                                                                 -------
  Normal   Provisioning          105s (x12 over 15m)  csi.vsphere.vmware.com_vsphere-csi-controller-675dfc79fd-6hmtt_e498f0cc-43c5-4a43-a38c-12bc53f581dd  External provisioner is provisioning volume for claim "default/example-pvc"
  Warning  ProvisioningFailed    105s (x12 over 15m)  csi.vsphere.vmware.com_vsphere-csi-controller-675dfc79fd-6hmtt_e498f0cc-43c5-4a43-a38c-12bc53f581dd  failed to provision volume with StorageClass "thin": rpc error: code = Internal desc = failed to get shared datastores in kubernetes cluster. Error: empty List of Node VMs returned from nodeManager
  Normal   ExternalProvisioning  37s (x63 over 15m)   persistentvolume-controller                                                                          Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Did I miss something in the config? Thanks for any help.

Help Me Obi-Wan Kenobi, You’re My Only Hope.

[Sispheor]

Just notice that I have this error in daemon set

Error creating: pods "vsphere-csi-node-wwtkt" is forbidden: violates PodSecurity "baseline:latest": non-default capabilities (container "vsphere-csi-node" must not include "SYS_ADMIN" in securityContext.capabilities.add), host namespaces (hostNetwork=true), hostPath volumes (volumes "registration-dir", "plugin-dir", "pods-mount-dir", "device-dir", "blocks-dir", "sys-devices-dir"), hostPort (container "vsphere-csi-node" uses hostPort 9808), privileged (container "vsphere-csi-node" must not set securityContext.privileged=true)

[Sispheor]

Fix:

kubectl label ns vmware-system-csi pod-security.kubernetes.io/enforce=privileged

All good PVC is created 🍾