Data Analytics with Prometheus and Grafana for Kubernetes

This walks through the setup for Prometheus and Grafana and an example that may be useful for troubleshooting connectivity issues within your cluster.


Please finish the Container Networking Tutorial, but do not clean up your cluster.

In order to install Prometheus and Grafana, please follow these steps.

$ cd install/cluster/
$ vagrant ssh kubeadm-master
[vagrant@kubeadm-master ~]$ cd contiv-*
[vagrant@kubeadm-master contiv-1.2.0]$ cd install/k8s/configs/
[vagrant@kubeadm-master configs]$ pwd
[vagrant@kubeadm-master configs]$ sudo -i
[root@kubeadm-master ~]# cd /home/vagrant/contiv-1.2.0/install/k8s/configs
[root@kubeadm-master configs]# cp prometheus.yml /var/contiv/prometheus.yml
[root@kubeadm-master configs]# exit
[vagrant@kubeadm-master configs]$  kubectl create -f contiv-prometheus.yml
clusterrole "prometheus" created
serviceaccount "prometheus" created
clusterrolebinding "prometheus" created
replicaset "contiv-prometheus" created
service "prometheus" created

[vagrant@kubeadm-master configs]$ kubectl create -f contiv-grafana.yml
clusterrole "grafana" created
serviceaccount "grafana" created
clusterrolebinding "grafana" created
replicaset "contiv-grafana" created
service "grafana" created

[vagrant@kubeadm-master configs]$ cd

Check if Contiv's stats_exporter is running as a sidecar container in the netmaster and netplugin kubernetes pods.

[vagrant@kubeadm-master configs]$ kubectl get pods -n=kube-system | grep 'netplugin\|netmaster'
contiv-netmaster-425jp                   3/3       Running   0          2d
contiv-netplugin-75dxx                   2/2       Running   0          2d
contiv-netplugin-zhbrd                   2/2       Running   0          2d

[vagrant@kubeadm-master configs]$ kubectl describe pod -n=kube-system contiv-netmaster-425jp | grep stats
    Image:          contiv/stats
    Image ID:       docker-pullable://

[vagrant@kubeadm-master configs]$ kubectl describe pod -n=kube-system contiv-netplugin-75dxx | grep stats
    Image:          contiv/stats
    Image ID:       docker-pullable://

[vagrant@kubeadm-master configs]$ kubectl describe pod -n=kube-system contiv-netplugin-zhbrd | grep stats
    Image:          contiv/stats
    Image ID:       docker-pullable://

Check if stats_exporter exposes metrics for Prometheus to scrape.

[vagrant@kubeadm-master configs]$ curl localhost:9004/metrics

[vagrant@kubeadm-master configs]$ curl localhost:9005/metrics

Chapter 1: Prometheus and Grafana

Prometheus is an open source monitoring platform that is useful for visualizing time series data. You can monitor and get alerts for your cluster through Prometheus. You can find out more about Prometheus here.

Grafana is an open source data visualization platform which further enhances the user experience. In this tutorial, you will learn how Grafana can visualize the data stored in Prometheus and how to run some useful queries in the Grafana dashboard. For more information on Grafana, click here.

Let's explore Prometheus and Grafana.

Prometheus and Grafana are running as NodePorts which are exposed on the host IP. Since they are both configured to run on the master node, let's find out the IP of the master node.

[vagrant@kubeadm-master ~]$ kubectl describe nodes kubeadm-master

We can see that its IP is

We can also find out what port Prometheus is exposed at.

[vagrant@kubeadm-master ~]$ kubectl get svc -n kube-system
NAME         CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
grafana     <nodes>       3000:32701/TCP   25m
kube-dns       <none>        53/UDP,53/TCP    58m
prometheus   <nodes>       9090:32700/TCP   25m

Open a browser and navigate to This will open up the Prometheus UI.

Let's check and make sure that Prometheus is able to fetch data from all the nodes. At the top click on Status and in the drop down menu click on Targets. You should see a list of targets and that they are UP and green. If it says UNKNOWN it is likely because the environment or prometheus is still configuring.

Go back to Graph from the top tool bar. Click on the drop down menu that says -insert metric at cursor- and pick any metric. Then click on Execute and Graph right below the Execute button.

Now let's move to Grafana.

Grafana is exposed at port 32701, so let's navigate to in our browser. The Grafana UI will open up and prompt you for a username and password. Username is "admin" and password is "admin". We will be working with Grafana for the remainder of this tutorial because its UI is more user friendly.

After you have logged in, you will see an option to Add data source. Click on that. Fill out the information as shown in the following screenshot.

Add Data Source

Click on Add and Save & Test.

Click on the Grafana logo in the top left of the screen and go to Dashboard and click on New. Click on Graph. Click on "Panel Title" and then click on "Edit".

Chapter 2: Basic Ping Example

Now go back to your terminal which has the cluster running. You should already have a contiv-net network setup as part of the Networking Tutorial as well as the contiv-c1 and contiv-c2 pods up and running.

[vagrant@kubeadm-master ~]$ netctl net ls
Tenant   Network      Nw Type  Encap type  Packet tag  Subnet        Gateway    IPv6Subnet  IPv6Gateway  Cfgd Tag
------   -------      -------  ----------  ----------  -------       ------     ----------  -----------  ---------
default  default-net  data     vxlan       0 
default  contiv-net   data     vxlan       0 
default  contivh1     infra    vxlan       0 

[vagrant@kubeadm-master ~]$ kubectl get pods
contiv-c1   1/1       Running   0          24s
contiv-c2   1/1       Running   0          9s

Now let's go back to the Grafana UI and add the following four queries.


Go back to the terminal and set up a ping between the two pods.

[vagrant@kubeadm-master ~]$ kubectl exec -it contiv-c1 sh
/ # ping

Now if we go back to the Grafana Dashboard, we should see that the number of packets transmitted increases, so our basic ping works.

Basic Ping

The way the data is displayed shows a monotonically increasing counter. This may not be the most ideal way to display your data, in which case you can also graph deltas. idelta() is a query function that takes a range vector of the metric you want to graph. A range vector is a list of datapoints over a period of time. For example, you can graph the received bytes over a one day span with the following query.


As you go through the rest of the Networking and Policy Tutorials, you can keep experimenting with different queries and viewing graphs on Grafana.


For more query functions, find them here.

Improvements or Comments

This tutorial was developed by Contiv engineers. Thank you for trying out this tutorial. Please file a GitHub issue if you see an issue with the tutorial, or if you prefer improving some text, feel free to send a pull request.