Deploying vSphere CPI using Helm
The purpose of this guide is to provide the reader with step by step instructions on how to deploy the vSphere CPI on vSphere infrastructure using Helm. The reader will learn how to install and configure Helm as well as learn about basic Helm operations. At the end of this tutorial you will have a fully configured vSphere CPI running on a vSphere environment.
Review the comprehensive guide locate at Deploying a Kubernetes Cluster on vSphere with CSI and CPI. The prerequisites found in that guide also apply when deploying the CPI via Helm. This guide also assumes that you have a Kubernetes cluster up and running. If you need assistance in setting up a Kubernetes cluster, please refer to the Deploying a Kubernetes Cluster on vSphere with CSI and CPI guide for cluster setup instructions.
Helm charts has been fully deprecated since Nov 13th 2020.
The Helm Chart for vSphere CPI has been moved to this repo. It has been tested and verified working using Helm v.2.16.X and v3.0.0+. It is highly recommended that Helm v3.0.0+ be used when deploying the Helm Chart for vSphere CPI. At any point should you have additional questions regarding Helm, please visit this website for the official Helm documentation.
Setting up Helm
Before you begin, all outlined steps below are carried out on the master node only.
Option 1: From the Binary Releases
Download the latest release of Helm appropriate for your platform. At the time of writing this Helm quickstart guide, the latest release is v3.6.0. It is highly recommended using Helm v3.0.0+ when deploying the Helm Chart for vSphere CPI.
After the download is complete, unpack and install Helm v3.0.0+.
# tar -zxvf helm-v3.6.0-linux-amd64.tgz # sudo mv linux-amd64/helm /usr/local/bin/helm
Option 2: From Script
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 chmod 700 get_helm.sh ./get_helm.sh
See Install Helm for more options.
Verify that Helm installed successfully by running the following command:
Check that all nodes are tainted
Before continuing, make sure all nodes are tainted with
node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule. When the kubelet is started with "external" cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud provider initializes this node, the kubelet removes this taint.
# kubectl describe nodes | egrep "Taints:|Name:" Name: k8s-master Taints: node-role.kubernetes.io/master:NoSchedule Name: k8s-node1 Taints: node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule Name: k8s-node2 Taints: node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule Name: k8s-node3 Taints: node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule Name: k8s-node4 Taints: node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Installing the Chart using Helm 3.0+
To add the Helm Stable Charts for cloud-provider-vsphere, you can run the following command:
helm repo add vsphere-cpi https://kubernetes.github.io/cloud-provider-vsphere helm repo update
See help repo for command documentation.
Deploying vSphere CPI for Simple Configurations
If your vSphere environment contains only a single vCenter Server where the default
vsphere.conf is acceptable, this section should be sufficient for your deployment needs. You can deploy the Helm Chart for vSphere CPI using the follwing single Helm command:
# helm upgrade --install vsphere-cpi vsphere-cpi/vsphere-cpi --namespace kube-system --set config.enabled=true --set config.vcenter=<vCenter IP> --set config.username=<vCenter Username> --set config.password=<vCenter Password> --set config.datacenter=<vCenter Datacenter>
Here we use the '--set' flag to override values in a chart and pass configuration from the command line.
The following is a description of the fields used in the vsphere.conf configmap:
config.enabledshould be set to true to enable the functionality to create the configMap and secret
config.vcenterthe IP address or FQDN for your vCenter Server should be specified here
config.usernameholds the username to be used for your vCenter Server
config.passwordholds the password to be used for your vCenter Server
config.datacentershould be the list of all comma separated datacenters where kubernetes node VMs are present.
Deploying vSphere CPI for Advanced Configurations
If your vSphere environment contains multiple vCenter Servers or the default parameters contained within the
vsphere.conf must be changed, you can deploy the Helm Chart using the procedure below.
- Create a CPI configMap
- Create a CPI secret
- Deploy vSphere CPI using Helm
Create a CPI configMap
This cloud-config configmap file, passed to the CPI on initialization, contains details about the vSphere configuration. This file, which here we have called
vsphere.conf has been populated with some sample values. Obviously, you will need to modify this file to reflect your own vSphere configuration.
# tee /etc/kubernetes/vsphere.conf >/dev/null <<EOF [Global] port = "443" insecure-flag = "true" secret-name = "cpi-global-secret" secret-namespace = "kube-system" [VirtualCenter "220.127.116.11"] datacenters = "finance" [VirtualCenter "192.168.0.1"] datacenters = "hr" [VirtualCenter "10.0.0.1"] datacenters = "engineering" secret-name = "cpi-engineering-secret" secret-namespace = "kube-system" [Labels] region = "k8s-region" zone = "k8s-zone" EOF
Here is a description of the fields used in the vsphere.conf configmap.
insecure-flagshould be set to true to use self-signed certificate for login
VirtualCentersection is defined to hold property of vcenter. IP address and FQDN should be specified here.
secret-nameholds the credential(s) for a single or list of vCenter Servers.
secret-namespaceis set to the namespace where the secret has been created.
portis the vCenter Server Port. The default is 443 if not specified.
datacentersshould be the list of all comma separated datacenters where kubernetes node VMs are present.
Create the configmap by running the following command:
# cd /etc/kubernetes
# kubectl create configmap cloud-config --from-file=vsphere.conf --namespace=kube-system
Verify that the configmap has been successfully created in the kube-system namespace.
# kubectl get configmap cloud-config --namespace=kube-system NAME DATA AGE cloud-config 1 82s
Create a CPI secret
The CPI supports storing vCenter credentials either in:
- a shared global secret containing all vCenter credentials, or
- a secret dedicated for a particular vCenter configuration which takes precedence over anything that might be configured within the global secret
In the example
vsphere.conf above, there are two configured Kubernetes secret. The vCenter at
10.0.0.1 contains credentials in the secret named
cpi-engineering-secret in the namespace
kube-system and the vCenter at
192.168.0.1 contains credentials in the secret named
cpi-global-secret in the namespace
kube-system defined in the
An example Secrets YAML can be used for reference when creating your own
secrets. If the example secret YAML is used, update the secret name to use a
<unique secret name>, the vCenter IP address in the keys of
stringData, and the
password for each key.
The secret for the vCenter at
18.104.22.168 might look like the following:
apiVersion: v1 kind: Secret metadata: name: cpi-engineering-secret namespace: kube-system stringData: 10.0.0.1.username: "email@example.com" 10.0.0.1.password: "password"
Then to create the secret, run the following command replacing the name of the YAML file with the one you have used:
# kubectl create -f cpi-engineering-secret.yaml
Verify that the credential secret is successfully created in the kube-system namespace.
# kubectl get secret cpi-engineering-secret --namespace=kube-system NAME TYPE DATA AGE cpi-engineering-secret Opaque 1 43s
If you have multiple vCenters as in the example vsphere.conf above, your Kubernetes Secret YAML could look like the following to storage the vCenter credentials for vCenters at
apiVersion: v1 kind: Secret metadata: name: cpi-global-secret namespace: kube-system stringData: 22.214.171.124.username: "firstname.lastname@example.org" 126.96.36.199.password: "password" 192.168.0.1.username: "email@example.com" 192.168.0.1.password: "password"
Deploy vSphere CPI using Helm
secret have been created, deploy the Helm Chart for vSphere CPI by running the following command:
# helm install vsphere-cpi vsphere-cpi/vsphere-cpi
Verify that the CPI has been successfully deployed
You can verify the vSphere CPI deployed succesfully by listing the Helm Charts currently deployed.
# helm list
Next verify vsphere-cloud-controller-manager is running and all other system pods are up and running (note that the coredns pods were not running previously - they should be running now as the taints have been removed by installing the CPI):
# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-bq7qq 1/1 Running 0 71m coredns-fb8b8dccf-r47q2 1/1 Running 0 71m etcd-k8s-master 1/1 Running 0 69m kube-apiserver-k8s-master 1/1 Running 0 70m kube-controller-manager-k8s-master 1/1 Running 0 69m kube-flannel-ds-amd64-7kmk9 1/1 Running 0 38m kube-flannel-ds-amd64-dtvbg 1/1 Running 0 63m kube-flannel-ds-amd64-hq57c 1/1 Running 0 30m kube-flannel-ds-amd64-j7g4s 1/1 Running 0 22m kube-flannel-ds-amd64-q4zsn 1/1 Running 0 21m kube-proxy-6jcng 1/1 Running 0 30m kube-proxy-bh8kh 1/1 Running 0 21m kube-proxy-rb9xp 1/1 Running 0 22m kube-proxy-srhpj 1/1 Running 0 71m kube-proxy-vh4lg 1/1 Running 0 38m kube-scheduler-k8s-master 1/1 Running 0 70m vsphere-cloud-controller-manager-549hb 1/1 Running 0 25s
Check that all nodes are untainted
node.cloudprovider.kubernetes.io/uninitialized taint is removed from all nodes.
# kubectl describe nodes | egrep "Taints:|Name:" Name: k8s-master Taints: node-role.kubernetes.io/master:NoSchedule Name: k8s-node1 Taints: <none> Name: k8s-node2 Taints: <none> Name: k8s-node3 Taints: <none> Name: k8s-node4 Taints: <none>
Note: If you happen to make an error with the
vsphere.conf, simply delete the Helm deployment of vSphere CPI (describe in the next section below), the configMap, and the secret, then make any necessary edits to the configMap
vSphere.conf file and/or secret, and reapply the steps above.
You may now remove the
vsphere.conf file created at
Uninstall the Helm Chart for vSphere CPI
To uninstall/delete the vsphere-cpi deployment:
# Helm 3 $ helm uninstall [RELEASE_NAME]
You can delete the
secret for the vSphere CPI if they are no longer needed.