Kubernetes store sensitive information in vault as persistent storage

4 min readJun 30, 2020

Overview

In the kubernetes environment, many people may be wondering how to store sensitive information such as passwords and access tokens when trying to automate by CI/CD.

I think that there is resistance to pushing the manifest with the password written on GitHub etc. This time, we will verify the operation of secrets-store-csi-driver , which mounts the confidential information stored in vault , which is provided by hashicorp for managing confidential information, as storage on kuberntes .

Verification environment

Governors v1.14.1
secrets-store-csi-driver v0.0.11

Advance preparation

As a preliminary preparation, start up the vault server used for operation verification.

Download

Download secrets-store-csi-driver from the GitHub repository .

$ git clone https://github.com/kubernetes-sigs/secrets-store-csi-driver.git && cd secrets-store-csi-driver

Deploying a verification vault

Next, deploy the vault used for verification on kubernetes. Since this vault does not have persistent data, all the stored data will be deleted when the pod is deleted. Please use it only for verification.

$ kubectl apply -f test/bats/tests/vault.yaml

When deployed, vault pods will start as shown below.

$ kubectl get pod -l app=vault  
NAME                    READY   STATUS    RESTARTS   AGE
vault-f56478d9-9saf2   1/1     Running   1          5m7s

$ kubectl get service -l app=vault
NAME    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
vault   NodePort   10.21.21.12   <none>        8200:31899/TCP   17m

Port forward to make vault accessible from outside the kubernetes cluster.

$ kubectl port-forward vault-f56478d9-9saf2 8200:8200 &

In fact, go to vault and make sure it is up.

$ export VAULT_ADDR="http://127.0.0.1:8200"
$ export VAULT_TOKEN="root"
i$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.1.1
Cluster Name    vault-cluster-692252b2
Cluster ID      60c527f1-2e28-70c5-88fc-55afcc01ec43
HA Enabled      false

This completes the setup of the vault server used for verification.

Operation verification

After setting authentication to vault so that kubernetes can access it, deploy secrets-store-csi-driver. In this verification, defaultwe will use the Kubernetes namespace.

Creating a service account in kubernetes

First, create a service account on kubernetes for accessing vault.

$ kubectl create serviceaccount vault-auth
$ kubectl apply -f serviceaccount.yml

Set kubernetes credentials in Vault

Next, set the kubernetes credentials in the vault. First of all, get the required information such as certificates and tokens from various kubernetes resources.

$ CLUSTER_NAME="$(kubectl config view -o json | jq -r .clusters[].name)"$ SECRET_NAME="$(kubectl get serviceaccount vault-auth \
  -o go-template='{{ (index .secrets 0).name }}')"$ TR_ACCOUNT_TOKEN="$(kubectl get secret ${SECRET_NAME} \
  -o go-template='{{ .data.token }}' | base64 --decode)"$ KUBE_HOST="$(kubectl config view --raw \
  -o go-template="{{ range .clusters }}{{ if eq .name \"${CLUSTER_NAME}\" }}{{ index .cluster \"server\" }}{{ end }}{{ end }}")"$ KUBE_CACERT="$(kubectl config view --raw \
  -o go-template="{{ range .clusters }}{{ if eq .name \"${CLUSTER_NAME}\" }}{{ index .cluster \"certificate-authority-data\" }}{{ end }}{{ end }}" | base64 --decode)"

Some values are obtained from kuberntes’ config information. If multiple kubernetes clusters are registered in the config information, check if the correct kubernetes cluster information is acquired. Then kubernetesenable it in vault authentication .

$ vault auth enable kubernetes

Next kubernetes, set the previously acquired certificate and token information as the authentication information of.

$ vault write auth/kubernetes/config \
  kubernetes_host="${KUBE_HOST}" \
  kubernetes_ca_cert="${KUBE_CACERT}" \
  token_reviewer_jwt="${TR_ACCOUNT_TOKEN}"

Next, set the access policy of the confidential information used in this operation verification to vault.

$ echo 'path "secret/data/example" {
  capabilities = ["read", "list"]
}path "sys/renew/*" {
  capabilities = ["update"]
}' | vault policy write example-readonly -$ vault write auth/kubernetes/role/example-role \
  bound_service_account_names=csi-driver-registrar \
  bound_service_account_namespaces=default \
  policies=default,example-readonly \
  ttl=20m

This completes the vault credential settings. Next, example store the confidential information of the sample used in the operation verification .

$ vault kv put secret/example bar=hello

Deploy secrets-store-csi-driver

Deploy the secrets-store-csi-driver.

$ kubectl apply -f deploy/crd-csi-driver-registry.yaml
$ kubectl apply -f deploy/rbac-csi-driver-registrar.yaml
$ kubectl apply -f deploy/rbac-csi-attacher.yaml
$ kubectl apply -f deploy/csi-secrets-store-attacher.yaml
$ kubectl apply -f deploy/secrets-store-csi-driver.yaml

defaultWhen deploying to a namespace other than the namespace, you need to edit the manifest. Verify that the secrets-store-csi-driver has been deployed.

$ kubectl get sts,ds
NAME                                          READY   AGE
statefulset.apps/csi-secrets-store-attacher   1/1     16m42sNAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
daemonset.extensions/csi-secrets-store   2         2         2       2            2           <none>          219s

Mount sensitive information stored in Vault with PV or PVC

Create manifest to mount sensitive information stored in vault.

Now deploy this manifest

kubectl apply -f pv-manifest.yml

Next, create pvc-manifest

Deploy manifest

kubectl apply -f pvc-manifest.yml

Make you sure that pvc & pv are done.

$ kubectl get pvc,pv
NAME                              STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/pvc-vault   Bound    pv-vault   1Gi        ROX                           124s

NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS    REASON   AGE
persistentvolume/pv-vault         1Gi        ROX            Retain           Bound    default/pvc-vault                                 12m44s

Now we can create pod that mounts the deployed pv & pvc.

Deploy pod

kubectl apply -f nginx-pod-manifest.yml

Finally, make sure the so pod is able to mount sensitive information stored in vault, let’s check if it mounted correctly

$ kubectl exec -ti nginx-vault /bin/bash
root@nginx-vault:/# ls /mnt/vault/
exampleroot@nginx-vault:/# cat /mnt/vault/example
hello

As a result we should see example as a value of the path where sensitive Vault information is stored.

At the end

We verified the operation of mounting the data stored in vault, which is a repository for storing some confidential information, as persistent storage (PVC, PV) with kubernetes. This allow to easily read some passwords, tokens which is stored in kubernetes vault.