Kubernetes store sensitive information in vault as persistent storage

Overview

In the kubernetes environment, many people may be wondering how to store sensitive information such as passwords and access tokens when trying to automate by CI/CD.

I think that there is resistance to pushing the manifest with the password written on GitHub etc. This time, we will verify the operation of secrets-store-csi-driver , which mounts the confidential information stored in vault , which is provided by hashicorp for managing confidential information, as storage on kuberntes .

Verification environment

  • Governors v1.14.1
  • secrets-store-csi-driver v0.0.11

Advance preparation

As a preliminary preparation, start up the vault server used for operation verification.

Download

Download secrets-store-csi-driver from the GitHub repository .

$ git clone https://github.com/kubernetes-sigs/secrets-store-csi-driver.git && cd secrets-store-csi-driver

Deploying a verification vault

Next, deploy the vault used for verification on kubernetes. Since this vault does not have persistent data, all the stored data will be deleted when the pod is deleted. Please use it only for verification.

$ kubectl apply -f test/bats/tests/vault.yaml

When deployed, vault pods will start as shown below.

$ kubectl get pod -l app=vault  
NAME READY STATUS RESTARTS AGE
vault-f56478d9-9saf2 1/1 Running 1 5m7s
$ kubectl get service -l app=vault
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vault NodePort 10.21.21.12 <none> 8200:31899/TCP 17m

Port forward to make vault accessible from outside the kubernetes cluster.

$ kubectl port-forward vault-f56478d9-9saf2 8200:8200 &

In fact, go to vault and make sure it is up.

$ export VAULT_ADDR="http://127.0.0.1:8200"
$ export VAULT_TOKEN="root"
i$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.1.1
Cluster Name vault-cluster-692252b2
Cluster ID 60c527f1-2e28-70c5-88fc-55afcc01ec43
HA Enabled false

This completes the setup of the vault server used for verification.

Operation verification

After setting authentication to vault so that kubernetes can access it, deploy secrets-store-csi-driver. In this verification, defaultwe will use the Kubernetes namespace.

Creating a service account in kubernetes

First, create a service account on kubernetes for accessing vault.

$ kubectl create serviceaccount vault-auth
$ kubectl apply -f serviceaccount.yml

Set kubernetes credentials in Vault

Next, set the kubernetes credentials in the vault. First of all, get the required information such as certificates and tokens from various kubernetes resources.

$ CLUSTER_NAME="$(kubectl config view -o json | jq -r .clusters[].name)"$ SECRET_NAME="$(kubectl get serviceaccount vault-auth \
-o go-template='{{ (index .secrets 0).name }}')"
$ TR_ACCOUNT_TOKEN="$(kubectl get secret ${SECRET_NAME} \
-o go-template='{{ .data.token }}' | base64 --decode)"
$ KUBE_HOST="$(kubectl config view --raw \
-o go-template="{{ range .clusters }}{{ if eq .name \"${CLUSTER_NAME}\" }}{{ index .cluster \"server\" }}{{ end }}{{ end }}")"
$ KUBE_CACERT="$(kubectl config view --raw \
-o go-template="{{ range .clusters }}{{ if eq .name \"${CLUSTER_NAME}\" }}{{ index .cluster \"certificate-authority-data\" }}{{ end }}{{ end }}" | base64 --decode)"

Some values ​​are obtained from kuberntes’ config information. If multiple kubernetes clusters are registered in the config information, check if the correct kubernetes cluster information is acquired. Then kubernetesenable it in vault authentication .

$ vault auth enable kubernetes

Next kubernetes, set the previously acquired certificate and token information as the authentication information of.

$ vault write auth/kubernetes/config \
kubernetes_host="${KUBE_HOST}" \
kubernetes_ca_cert="${KUBE_CACERT}" \
token_reviewer_jwt="${TR_ACCOUNT_TOKEN}"

Next, set the access policy of the confidential information used in this operation verification to vault.

$ echo 'path "secret/data/example" {
capabilities = ["read", "list"]
}
path "sys/renew/*" {
capabilities = ["update"]
}' | vault policy write example-readonly -
$ vault write auth/kubernetes/role/example-role \
bound_service_account_names=csi-driver-registrar \
bound_service_account_namespaces=default \
policies=default,example-readonly \
ttl=20m

This completes the vault credential settings. Next, example store the confidential information of the sample used in the operation verification .

$ vault kv put secret/example bar=hello

Deploy secrets-store-csi-driver

Deploy the secrets-store-csi-driver.

$ kubectl apply -f deploy/crd-csi-driver-registry.yaml
$ kubectl apply -f deploy/rbac-csi-driver-registrar.yaml
$ kubectl apply -f deploy/rbac-csi-attacher.yaml
$ kubectl apply -f deploy/csi-secrets-store-attacher.yaml
$ kubectl apply -f deploy/secrets-store-csi-driver.yaml

defaultWhen deploying to a namespace other than the namespace, you need to edit the manifest. Verify that the secrets-store-csi-driver has been deployed.

$ kubectl get sts,ds
NAME READY AGE
statefulset.apps/csi-secrets-store-attacher 1/1 16m42s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.extensions/csi-secrets-store 2 2 2 2 2 <none> 219s

Mount sensitive information stored in Vault with PV or PVC

Create manifest to mount sensitive information stored in vault.

Now deploy this manifest

kubectl apply -f pv-manifest.yml

Next, create pvc-manifest

Deploy manifest

kubectl apply -f pvc-manifest.yml

Make you sure that pvc & pv are done.

$ kubectl get pvc,pv
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/pvc-vault Bound pv-vault 1Gi ROX 124s

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pv-vault 1Gi ROX Retain Bound default/pvc-vault 12m44s

Now we can create pod that mounts the deployed pv & pvc.

Deploy pod

kubectl apply -f nginx-pod-manifest.yml

Finally, make sure the so pod is able to mount sensitive information stored in vault, let’s check if it mounted correctly

$ kubectl exec -ti nginx-vault /bin/bash
root@nginx-vault:/# ls /mnt/vault/
example
root@nginx-vault:/# cat /mnt/vault/example
hello

As a result we should see example as a value of the path where sensitive Vault information is stored.

At the end

We verified the operation of mounting the data stored in vault, which is a repository for storing some confidential information, as persistent storage (PVC, PV) with kubernetes. This allow to easily read some passwords, tokens which is stored in kubernetes vault.

 by the author.

--

--

--

DevOps Consultant. I’m strongly focused on automation, security, and reliability.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Down The Rabbit Hole Alice:

Getting to know RSpec, Just a Little Bit

Reading and Writing to the Database

Arduino and Visual Studio Code: Getting Started

Andela Boot Camp: Day 3

Running a Play app with Docker locally and debugging with IntelliJ Idea

Nec C++ Quiz — 0007

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Maciej

Maciej

DevOps Consultant. I’m strongly focused on automation, security, and reliability.

More from Medium

Kubernetes Clusters with “kind”

Filesystem Layouts Supported by Kubernetes

Kubernetes, rke2, containerd, Elasticsearch, limits.conf , Increasing the # of open files.

Ansible Role to Configure K8S Multi Node Cluster over AWS Cloud.