Last modified October 21, 2016

Monitoring with Prometheus and Grafana

Prometheus is an open-source monitoring solution that includes the gathering of metrics, their storage in an internal time series database as well as querying and alerting based on that data.

It offers a lot of integrations incl. Docker, Kubernetes, etc.

Prometheus can also visualize your data. However, in this recipe we include another open-source tool, Grafana, for the visualization part, as it offers a more powerful and flexible way to generate visuals and dashboards.

Deploying Prometheus and Grafana

The following command will set you up with all neccesary components and some first dashboards to check out.

kubectl apply --filename https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests-all.yaml

Checking Prometheus

Wait a bit for all the pods to come up. Then Prometheus should be ready and running. We can check the Prometheus targets at https://api.<cluster-id>.k8s.gigantic.io/api/v1/proxy/namespaces/default/services/prometheus:9090/targets

Prometheus Targets

Note: The above URL uses your Kubernetes API to proxy to the service. As the API is guarded with your credentials, you need to set them up in your system (and/or browser). We do not recommend to set up an Ingress for Prometheus at this time, as it currently does not support any kind of authentication and thus your cluster would be open to everyone.

Checking Grafana

Now that we know Prometheus is up and running we can check for Grafana.

There’s an Ingress set up for Grafana, so it should be available at https://http://grafana.monitoring.<cluster-id>.k8s.gigantic.io/

You can user the default admin (admin:admin) user for your first login. You should this admin user to reflect your desired username, your email, and a secure password ASAP!

Note: If persistent storage is not set up in your cluster, changes like the above will be reset to defaults if the Grafana Pod gets rescheduled. You would need to set them again after that.

Changing the admin

You can change the default admin user at http://grafana.monitoring..k8s.gigantic.io/admin/users/edit/1

Grafana Datasource

Please note, that you need to update the password and the user data (username, email, etc.) separately with the respective update buttons below each section.

Check out your dashboards

You can now checkout the included dashboards, e.g. the Cluster Monitoring Overview.

Grafana Import Dashboard

Note: If persistent storage is not set up in your cluster, the preset datasource and dashboards will vanish if the Grafana Pod gets rescheduled. To get them back run:

kubectl --namespace=monitoring delete job grafana-import-dashboards
kubectl --namespace=monitoring create --filename https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests/grafana-import-dashboards-job.yaml

Next Steps

Next, you should get into the Grafana and Prometheus documentations to get to know the tools and either build your own dashboards or extend the samples from above.

You can also check out grafana.net for some more example dashboards and plugins.

You might also want to set up some alerting.


You can collaborate on this recipe on GitHub.