This section will cover the prerequisites that need to be in place before attempting the deployment.
vSphere 6.7U3 (or later) is a prerequisite for using CSI and CPI at the time of writing. This may change going forward, and the documentation will be updated to reflect any changes in this support statement. If you are on a vSphere version that is below 6.7 U3, you can either upgrade vSphere to 6.7U3 or follow one of the tutorials for earlier vSphere versions. Here is the tutorial on deploying Kubernetes with kubeadm, using the VCP - Deploying Kubernetes using kubeadm with the vSphere Cloud Provider (in-tree).
Providing the K8s master node(s) access to the vCenter management interface will be sufficient, given the CPI and CSI pods are deployed on the master node(s). Should these components be deployed on worker nodes or otherwise - those nodes will also need access to the vCenter management interface.
If you want to use topology-aware volume provisioning and the late binding feature using
region, the node needs to discover its topology by connecting to the vCenter, for this every node should be able to communicate to the vCenter. You can disable this optional feature if you want to open only the master node to the vCenter management interface.
Virtual Machine Hardware requirements
Virtual Machine Hardware must be
version 15 or higher. For Virtual Machine CPU and Memory requirements, size adequately based on workload requirements.
VMware also recommend that virtual machines use the VMware Paravirtual SCSI controller for Primary Disk on the Node VM. This should be the default, but it is always good practice to check.
disk.EnableUUID parameter must be set for each node VMs. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.
It is recommended to not take snapshots of CNS node VMs to avoid errors and unpredictable behavior.
The following govc commands will set the disk.EnableUUID=1 on all nodes.
export GOVC_INSECURE=1 export GOVC_URL='https://<VC_IP>' export GOVC_USERNAME=VC_Admin_User export GOVC_PASSWORD=VC_Admin_Passwd
Check the connection to vCenter:
$ govc ls /datacenter/vm /datacenter/network /datacenter/host /datacenter/datastore
To retrieve all Node VMs, use the following command:
$ govc ls /<datacenter-name>/vm /datacenter/vm/k8s-node3 /datacenter/vm/k8s-node4 /datacenter/vm/k8s-node1 /datacenter/vm/k8s-node2 /datacenter/vm/k8s-master
To use govc to enable Disk UUID, use the following command:
govc vm.change -vm '/datacenter/vm/k8s-node1' -e="disk.enableUUID=1" govc vm.change -vm '/datacenter/vm/k8s-node2' -e="disk.enableUUID=1" govc vm.change -vm '/datacenter/vm/k8s-node3' -e="disk.enableUUID=1" govc vm.change -vm '/datacenter/vm/k8s-node4' -e="disk.enableUUID=1" govc vm.change -vm '/datacenter/vm/k8s-master' -e="disk.enableUUID=1"
Further information on disk.enableUUID can be found in VMware Knowledgebase Article 52815.
Upgrade Virtual Machine Hardware
VM Hardware should be at version 15 or higher.
govc vm.upgrade -version=15 -vm '/datacenter/vm/k8s-node1' govc vm.upgrade -version=15 -vm '/datacenter/vm/k8s-node2' govc vm.upgrade -version=15 -vm '/datacenter/vm/k8s-node3' govc vm.upgrade -version=15 -vm '/datacenter/vm/k8s-node4' govc vm.upgrade -version=15 -vm '/datacenter/vm/k8s-master'
Check the VM Hardware version after running the above command:
$ govc vm.option.info '/datacenter/vm/k8s-node1' | grep HwVersion HwVersion: 15
On each K8s node, set the
cloud-provider flag to
external on all nodes. This flag needs to be set in the service configuration file (usually
/etc/systemd/system/kubelet.service) but this depends on how you installed Kubernetes or the distribution you are using.
kubelet service on each node.
systemctl daemon-reload systemctl restart kubelet.service
Kubernetes manifest changes
Set taints on all nodes to allow them to be initialised by the vSphere Cloud Provider Interface, this allows them to have their
providerID populated, which creates the link between the CSI and the VM in vCenter.
On worker nodes set this taint:
kubectl taint nodes --selector='!node-role.kubernetes.io/master' node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
On master nodes set this taint:
kubectl taint nodes --selector='node-role.kubernetes.io/master' node-role.kubernetes.io/master=:NoSchedule
Install the vSphere Cloud Provider Interface
Please refer to this guide for details on installing the CPI – https://cloud-provider-vsphere.sigs.k8s.io/tutorials/kubernetes-on-vsphere-with-kubeadm.html#install-the-vsphere-cloud-provider-interface
Note: Taints needs to be set on the nodes BEFORE the installation of the CPI.
Install the vSphere CSI Driver
Please refer to this guide for details on installing the CSI Driver - https://vsphere-csi-driver.sigs.k8s.io/