Monitor Kubernetes Cluster using Prometheus and Grafana

In this blog, we will learn how to monitor metrics for performance analysis, and also how to monitor and manage the real-time cost of Kubernetes resources using Prometheus and Grafana

Prerequisite

  1. Functional Kubernetes Cluster configured
  2. Kubernetes 1.16+
  3. Helm 3+
  4. Kubernetes’ command-line tool, kubectl, which is the command-line interface for running commands against Kubernetes clusters.
  5. Helm should be installed. We will install helm on windows in this article

Monitoring in Kubernetes

We will configure our Kubernetes cluster to get core metrics, such as CPU and memory.

We will deploy Prometheus and Grafana on our Kubernetes cluster. We will learn how to monitor a Kubernetes service with Prometheus and use Grafana dashboards to visualize cluster and application metrics.

Agenda:

  • Install Helm on Windows
  • Deploying Prometheus using Helm charts
  • Access the url using port-forward
  • Monitoring metrics using Grafana dashboards

Install helm

On windows, you need to first install chocolate and then using chocolate you can install the helm

Run PowerShell as an administrator and run the below command

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
Image for post
Image for post

Now run the command prompt as an administrator and check if chocolate is installed

chocochoco install kubernetes-helm
Image for post
Image for post
Image for post
Image for post

We have successfully installed helm on our windows machine. You can skip this if helm is already installed on your machine.

Deploying Prometheus using Helm charts

Now we will deploy Prometheus on a Kubernetes cluster using Helm charts:

Make sure that you have a kubernetes cluster is already running.

  1. Firstly, we will add the helm repository which is required
  2. Update the repository
helm repo add stable https://charts.helm.sh/stablehelm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo update
Image for post
Image for post
Image for post
Image for post

You can then run helm search repo prometheus-community to see the charts.

helm search repo prometheus-community
Image for post
Image for post

Let’s install the Prometheus chart. You can find more details about this chart here

helm install prometheus prometheus-community/kube-prometheus-stack
Image for post
Image for post

Check the status

kubectl get pods -l "release=prometheus"kubectl get pods
Image for post
Image for post

This chart will install additional charts as well:

  • Node-exporter on all the three nodes which are in your cluster
  • High Available Grafana
  • High Available Prometheus
  • High Available Alert Manager
  • Operator
  • Kube State Metrics

Now you have Prometheus installed with the bundle of components required to operate it on a Kubernetes environment.

Access the Prometheus url on the browser

Now lets verify our installation and try to access it on browser using port-forward

kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 8000:9090
Image for post
Image for post

MONITORING METRICS USING GRAFANA DASHBOARDS

Grafana is an open source analytics and monitoring solution. By default, Grafana is used for querying Prometheus

kubectl get svc
Image for post
Image for post

Port Forwarding

Create a port forwarding to access the Grafana UI using the kubectl port-forwardcommand. This command will forward the local port 8000 to port 3000 which is the default port of a Grafana pod:

Get the pod name using kubectl get pods

kubectl port-forward kube-prometheus-stack-1606233825-grafana-598d4d4bd6-r7pp5 8000:3000
  1. Open it on the browser
http://localhost:8000/login
Image for post
Image for post

Log in using admin as the username and prom-operatoras the password:

Import dashboard

Once you are able to login to Grafana successfully you can try exploring using the default dashboard which are provided by Grafana

Add Prometheus DataSource

With this helm chart Prometheus data source will be added by default. You can verify as shown below:

Click on Setting ->datasources

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Now we will create a dashboard which shows us all the pod details like CPU, memory, storage etc.

Grafana provides lot of dashboards which we can directly import in our Grafana instance and use it.

In this example, we will use this dashboard

Import the dashboard

Click on + icon -> Import and the save it

Image for post
Image for post

This is how the dashboard will look like and provide all the metrics for your pods.

Image for post
Image for post

Troubleshooting

When you will uninstall this chart and install it again you may see the below error

Failed d to install CRD crds/crd-alertmanagerconfigs.yaml: unable to recognize “”: no matches for kind “CustomResourceDefinition” in version “apiextensions.k8s.io/v1”

This is because cred’s didn’t get deleted with the uninstall step. Refer here for more details

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

Conclusion:

We have successfully learnt how to monitor the Kubernetes cluster using Prometheus and visualize the dashboard using Grafana.

Written by

Devops Automation Enginneer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store