Build from scratch Grafana and Prometheus on Minikube

Introduction

In this post we will take care build Prometheus, Grafana from scratch, the verification environment is minikube.

Environment

Let’s start

Launch vagrantbox with minikube and installed Prometheus/Node_esporter/Grafana

C:\tmp
λ git clone https://github.com/spy86/minikube-grafana-prometheus.git
Cloning into 'minikube-grafana-prometheus'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (35/35), done.
Receiving oal 45 (delta 12), reused 34 (delta 8), pack-reused 0 eceiving objects: 71% (32/45)
Receiving objects: 100% (45/45), 11.14 KiB | 1.39 MiB/s, done.
Resolving deltas: 100% (12/12), done.
C:\tmp
λ cd minikube-grafana-prometheus\
C:\tmp\minikube-grafana-prometheus (main -> origin)
λ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'mmichal/ubuntu16_04'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'mmichal/ubuntu16_04' version '1.1.20210331' is up to date...
==> default: Setting the name of the VM: minikube-grafana-prometheus_default_1624560364272_87145
==> default: to use the vagrant version plugin, config.version.url must be set
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: You are trying to forward to privileged ports (ports <= 1024). Most
==> default: operating systems restrict this to only privileged process (typically
==> default: processes running as an administrative user). This is a warning in case
==> default: the port forwarding doesn't work. If any problems occur, please try a
==> default: port higher than 1024.
==> default: Forwarding ports...
default: 80 (guest) => 80 (host) (adapter 1)
default: 443 (guest) => 443 (host) (adapter 1)
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
[default] A Virtualbox Guest Additions installation was found but no tools to rebuild or start them.
Reading package lists...
Building dependency tree...
Reading state information...
dkms is already the newest version (2.2.0.3-2ubuntu11.8).
linux-headers-4.4.0-186-generic is already the newest version (4.4.0-186.216).
linux-headers-4.4.0-186-generic set to manually installed.
The following packages were automatically installed and are no longer required:
libyaml-0-2 python-cffi-backend python-crypto python-cryptography
python-ecdsa python-enum34 python-httplib2 python-idna python-ipaddress
python-jinja2 python-markupsafe python-paramiko python-pkg-resources
python-pyasn1 python-setuptools python-six python-yaml sshpass
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
Copy iso file C:\Program Files\Oracle\VirtualBox\VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
Mounting Virtualbox Guest Additions ISO to: /mnt
mount: /dev/loop0 is write-protected, mounting read-only
Installing Virtualbox Guest Additions 6.1.16 - guest version is 5.0.18
Verifying archive integrity... All good.
Uncompressing VirtualBox 6.1.16 Guest Additions for Linux........
VirtualBox Guest Additions installer
Copying additional installer modules ...
Installing additional modules ...
VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules. This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup <version>
VirtualBox Guest Additions: or
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Building the modules for kernel 4.4.0-186-generic.
update-initramfs: Generating /boot/initrd.img-4.4.0-186-generic
VirtualBox Guest Additions: Running kernel modules will not be replaced until
the system is restarted
An error occurred during installation of VirtualBox Guest Additions 6.1.16. Some functionality may not work as intended.
In most cases it is OK that the "Window System drivers" installation failed.
Unmounting Virtualbox Guest Additions ISO from: /mnt
Got different reports about installed GuestAdditions version:
Virtualbox on your host claims: 5.0.18
VBoxService inside the vm claims: 6.1.16
Going on, assuming VBoxService is correct...
Got different reports about installed GuestAdditions version:
Virtualbox on your host claims: 5.0.18
VBoxService inside the vm claims: 6.1.16
Going on, assuming VBoxService is correct...
Got different reports about installed GuestAdditions version:
Virtualbox on your host claims: 5.0.18
VBoxService inside the vm claims: 6.1.16
Going on, assuming VBoxService is correct...
Restarting VM to apply changes...
==> default: Attempting graceful shutdown of VM...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
default: /vagrant => C:/tmp/minikube-grafana-prometheus
==> default: Running provisioner: shell...
default: Running: C:/Users/Admin/AppData/Local/Temp/vagrant-shell20210624-7368-cgktq9.sh
default: WARNING:
default: apt
default:
default: does not have a stable CLI interface.
default: Use with caution in scripts.
default: Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
default: Hit:2 http://us.archive.ubuntu.com/ubuntu xenial InRelease
default: Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
default: Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [1,648 kB]
default: Get:5 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
default: Get:6 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [2,049 kB]
default: Get:7 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [1,159 kB]
default: Get:8 http://us.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [1,525 kB]
default: Get:9 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [380 kB]
default: Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [785 kB]
default: Get:11 http://us.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [482 kB]
default: Get:12 http://security.ubuntu.com/ubuntu xenial-security/universe i386 Packages [665 kB]
default: Get:13 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [1,219 kB]
default: Get:14 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [225 kB]
default: Get:15 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [1,086 kB]
default: Get:16 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [358 kB]
default: Get:17 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [11.3 kB]
default: Get:18 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe i386 Packages [10.9 kB]
default: Fetched 11.9 MB in 14s (827 kB/s)
default: Reading package lists...
default: Building dependency tree...
default: Reading state information...
default: 48 packages can be upgraded. Run 'apt list --upgradable' to see them.
default: WARNING:
default: apt
default:
default: does not have a stable CLI interface.
default: Use with caution in scripts.
default: Reading package lists...
default: Building dependency tree...
default: Reading state information...
default: The following packages were automatically installed and are no longer required:
default: libyaml-0-2 python-cffi-backend python-crypto python-cryptography
default: python-ecdsa python-enum34 python-httplib2 python-idna python-ipaddress
default: python-jinja2 python-markupsafe python-paramiko python-pkg-resources
default: python-pyasn1 python-setuptools python-six python-yaml sshpass
default: Use 'sudo apt autoremove' to remove them.
default: The following additional packages will be installed:
default: liblzo2-2 squashfs-tools
default: Suggested packages:
default: zenity | kdialog
default: The following NEW packages will be installed:
default: liblzo2-2 snapd squashfs-tools
default: 0 upgraded, 3 newly installed, 0 to remove and 48 not upgraded.
default: Need to get 21.3 MB of archives.
default: After this operation, 103 MB of additional disk space will be used.
default: Get:1 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 liblzo2-2 amd64 2.08-1.2 [48.7 kB]
default: Get:2 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 squashfs-tools amd64 1:4.3-3ubuntu2.16.04.3 [105 kB]
default: Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 snapd amd64 2.48.3 [21.1 MB]
default: dpkg-preconfigure: unable to re-open stdin: No such file or directory
default: Fetched 21.3 MB in 42s (501 kB/s)
default: Selecting previously unselected package liblzo2-2:amd64.
default: (Reading database ...
default: (Reading database ... 5%
default: (Reading database ... 10%
default: (Reading database ... 15%
default: (Reading database ... 20%
default: (Reading database ... 25%
default: (Reading database ... 30%
default: (Reading database ... 35%
default: (Reading database ... 40%
default: (Reading database ... 45%
default: (Reading database ... 50%
default: (Reading database ... 55%
default: (Reading database ... 60%
default: (Reading database ... 65%
default: (Reading database ... 70%
default: (Reading database ... 75%
default: (Reading database ... 80%
default: (Reading database ... 85%
default: (Reading database ... 90%
default: (Reading database ... 95%
default: (Reading database ... 100%
default: (Reading database ...
default: 69406 files and directories currently installed.)
default: Preparing to unpack .../liblzo2-2_2.08-1.2_amd64.deb ...
default: Unpacking liblzo2-2:amd64 (2.08-1.2) ...
default: Selecting previously unselected package squashfs-tools.
default: Preparing to unpack .../squashfs-tools_1%3a4.3-3ubuntu2.16.04.3_amd64.deb ...
default: Unpacking squashfs-tools (1:4.3-3ubuntu2.16.04.3) ...
default: Selecting previously unselected package snapd.
default: Preparing to unpack .../snapd_2.48.3_amd64.deb ...
default: Unpacking snapd (2.48.3) ...
default: Processing triggers for man-db (2.7.5-1) ...
default: Processing triggers for mime-support (3.59ubuntu1) ...
default: Setting up liblzo2-2:amd64 (2.08-1.2) ...
default: Setting up squashfs-tools (1:4.3-3ubuntu2.16.04.3) ...
default: Setting up snapd (2.48.3) ...
default: snapd.failure.service is a disabled or a static unit, not starting it.
default: snapd.snap-repair.service is a disabled or a static unit, not starting it.
default: Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
default: 2021-06-24T18:48:41Z INFO Waiting for automatic snapd restart...
default: microk8s (1.20/stable) v1.20.7 from Canonical* installed
default: microk8s is running
default: high-availability: no
default: datastore master nodes: 127.0.0.1:19001
default: datastore standby nodes: none
default: addons:
default: enabled:
default: ha-cluster # Configure high availability on the current node
default: disabled:
default: ambassador # Ambassador API Gateway and Ingress
default: cilium # SDN, fast with full network policy
default: dashboard # The Kubernetes dashboard
default: dns # CoreDNS
default: fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
default: gpu # Automatic enablement of Nvidia CUDA
default: helm # Helm 2 - the package manager for Kubernetes
default: helm3 # Helm 3 - Kubernetes package manager
default: host-access # Allow Pods connecting to Host services smoothly
default: ingress # Ingress controller for external access
default: istio # Core Istio service mesh services
default: jaeger # Kubernetes Jaeger operator with its simple config
default: keda # Kubernetes-based Event Driven Autoscaling
default: knative # The Knative framework on Kubernetes.
default: kubeflow # Kubeflow for easy ML deployments
default: linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
default: metallb # Loadbalancer for your Kubernetes cluster
default: metrics-server # K8s Metrics Server for API access to service metrics
default: multus # Multus CNI enables attaching multiple network interfaces to pods
default: portainer # Portainer UI for your Kubernetes cluster
default: prometheus # Prometheus operator for monitoring and logging
default: rbac # Role-Based Access Control for authorisation
default: registry # Private image registry exposed on localhost:32000
default: storage # Storage class; allocates storage from host directory
default: traefik # traefik Ingress controller for external access
default: Enabling DNS
default: Applying manifest
default: serviceaccount/coredns created
default: configmap/coredns created
default: deployment.apps/coredns created
default: service/kube-dns created
default: clusterrole.rbac.authorization.k8s.io/coredns created
default: clusterrolebinding.rbac.authorization.k8s.io/coredns created
default: Restarting kubelet
default: DNS is enabled
default: Enabling Kubernetes Dashboard
default: Enabling Metrics-Server
default: clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
default: clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
default: rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
default: Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
default: apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
default: serviceaccount/metrics-server created
default: deployment.apps/metrics-server created
default: service/metrics-server created
default: clusterrole.rbac.authorization.k8s.io/system:metrics-server created
default: clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
default: clusterrolebinding.rbac.authorization.k8s.io/microk8s-admin created
default: Metrics-Server is enabled
default: Applying manifest
default: serviceaccount/kubernetes-dashboard created
default: service/kubernetes-dashboard created
default: secret/kubernetes-dashboard-certs created
default: secret/kubernetes-dashboard-csrf created
default: secret/kubernetes-dashboard-key-holder created
default: configmap/kubernetes-dashboard-settings created
default: role.rbac.authorization.k8s.io/kubernetes-dashboard created
default: clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
default: rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
default: clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
default: deployment.apps/kubernetes-dashboard created
default: service/dashboard-metrics-scraper created
default: deployment.apps/dashboard-metrics-scraper created
default:
default: If RBAC is not enabled access the dashboard using the default token retrieved with:
default:
default: token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
default: microk8s kubectl -n kube-system describe secret $token
default:
default: In an RBAC enabled setup (microk8s enable RBAC) you need to create a user with restricted
default: permissions as shown in:
default: https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
default: The registry will be created with the default size of 20Gi.
default: You can use the "size" argument while enabling the registry, eg microk8s.enable registry:size=30Gi
default: Enabling default storage class
default: deployment.apps/hostpath-provisioner created
default: storageclass.storage.k8s.io/microk8s-hostpath created
default: serviceaccount/microk8s-hostpath created
default: clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
default: clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
default: Storage will be available soon
default: Applying registry manifest
default: namespace/container-registry created
default: persistentvolumeclaim/registry-claim created
default: deployment.apps/registry created
default: service/registry created
default: configmap/local-registry-hosting configured
default: The registry is enabled
default: Enabling Ingress
default: ingressclass.networking.k8s.io/public created
default: namespace/ingress created
default: serviceaccount/nginx-ingress-microk8s-serviceaccount created
default: clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
default: role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
default: clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
default: rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
default: configmap/nginx-load-balancer-microk8s-conf created
default: configmap/nginx-ingress-tcp-microk8s-conf created
default: configmap/nginx-ingress-udp-microk8s-conf created
default: daemonset.apps/nginx-ingress-microk8s-controller created
default: Ingress is enabled
default: Added:
default: - microk8s.kubectl as kubectl
default: Client Version: v1.20.7-34+984a1cd176537e
default: Server Version: v1.20.7-34+984a1cd176537e
==> default: Running provisioner: shell...
default: Running: inline script
default: Create Namespace
default: namespace/monitoring created
default: Apply configuration for node-exporter
default: persistentvolumeclaim/node-exporter-pvc-proc created
default: persistentvolumeclaim/node-exporter-pvc-sys created
default: persistentvolume/node-exporter-pv-proc created
default: persistentvolume/node-exporter-pv-sys created
default: daemonset.apps/node-exporter created
default: service/node-exporter created
default: Apply configuration for kube-state-metrics
default: serviceaccount/kube-state-metrics created
default: clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
default: clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
default: service/kube-state-metrics created
default: deployment.apps/kube-state-metrics created
default: Apply configuration for prometheus
default: configmap/prometheus-config created
default: persistentvolume/prometheus-pv created
default: serviceaccount/prometheus created
default: clusterrole.rbac.authorization.k8s.io/prometheus created
default: clusterrolebinding.rbac.authorization.k8s.io/prometheus created
default: statefulset.apps/prometheus created
default: service/prometheus created
default: Apply configuration for grafana
default: persistentvolume/grafana-pv created
default: persistentvolumeclaim/grafana-pvc created
default: service/grafana created
default: deployment.apps/grafana created
default: ingress.networking.k8s.io/grafana-ingress created
default: Done !!!
default: Open browser and go to http://grafana.minikube.local
C:\tmp\minikube-grafana-prometheus (main -> origin)
λ

After a few minutes we will be able to log into the machine and check if all our components have been deployed correctly.

C:\tmp\minikube-grafana-prometheus (main -> origin)
λ vagrant ssh
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-186-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
vagrant@vagrant:~$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
vagrant Ready <none> 3m39s v1.20.7-34+984a1cd176537e
vagrant@vagrant:~$ sudo kubectl get ns
NAME STATUS AGE
kube-system Active 3m46s
kube-public Active 3m46s
kube-node-lease Active 3m46s
default Active 3m46s
container-registry Active 2m52s
ingress Active 2m50s
monitoring Active 2m49s
vagrant@vagrant:~$ sudo kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/grafana-55c84d4895-5gmr2 1/1 Running 0 2m47s
pod/node-exporter-xww84 1/1 Running 0 2m45s
pod/kube-state-metrics-6db99c899d-djh62 1/1 Running 0 2m51s
pod/prometheus-0 1/1 Running 0 2m49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/node-exporter ClusterIP None <none> 9100/TCP 2m53s
service/kube-state-metrics ClusterIP None <none> 8080/TCP,8081/TCP 2m51s
service/prometheus ClusterIP 10.152.183.176 <none> 9090/TCP 2m49s
service/grafana ClusterIP 10.152.183.186 <none> 3000/TCP 2m48s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 1 1 1 1 1 <none> 2m54s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 2m47s
deployment.apps/kube-state-metrics 1/1 1 1 2m51s
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-55c84d4895 1 1 1 2m47s
replicaset.apps/kube-state-metrics-6db99c899d 1 1 1 2m51s
NAME READY AGE
statefulset.apps/prometheus 1/1 2m49s
vagrant@vagrant:~$

As you can see all components are deployed correctly. Then let’s add an entry to our hosts file which is located in C:\Windows\System32\drivers\​etc\hosts

192.168.123.123 grafana.minikube.local

Now open browser and type http://grafana.minikube.local/ and enter the default user/password which is (admin,admin)

Now it’s time to Prometheus + Grafana integration. Select Add Data Source, then Select Prometheus in Choose data source type and enter the information in the screen below and click Save & Test

Now we need import dashboard which we can fin in link below:

Select Import from the button on left side

Then we need upload file, all settings for dashboard are below, if we finish click Import.

After correct import, we should see the dashboard with the metrics.

DevOps Consultant. I’m strongly focused on automation, security, and reliability.