In Chapter 2, Creating Kubernetes Clusters, we learned how to create Kubernetes clusters in different environments, experimented with different tools, and created a couple of clusters.
Creating a Kubernetes cluster is just the beginning of the story. Once the cluster is up and running, you need to make it sure it is operational, all the necessary components are in place and properly configured, and that enough resources are deployed to satisfy the requirements. Responding to failures, debugging, and troubleshooting is a major part of managing any complicated system, and Kubernetes is no exception.
The topics we will cover include the following:
- Monitoring with Heapster
- Performance analytics with Kubernetes dashboard
- Central logging
- Detecting problems at the node level
- Troubleshooting scenarios
At the end of this chapter you will have a solid understanding of the various options to monitor Kubernetes clusters, how to access logs, and how to analyze them. You will be able to look at a healthy Kubernetes cluster and verify everything is OK. You will also be able to look at an unhealthy Kubernetes cluster and methodically diagnose it, locate the problems, and address them.
Heapster is a Kubernetes project that provides a robust monitoring solution for Kubernetes clusters. It runs as a pod (of course), so it can be managed by Kubernetes itself. Heapster supports Kubernetes and CoreOS clusters. It has a very modular and flexible design. Heapster collects both operational metrics and events from every node in the cluster, stores them in a persistent backend (with a well-defined schema) and allows visualization and programmatic access. Heapster can be configured to use different backends (or sinks, in Heapster's parlance) and their corresponding visualization frontends. The most common combination is InfluxDB as backend and Grafana as frontend. The Google Cloud Platform integrates Heapster with the Google monitoring service. There are many other less common backends, such as the following:
- Log
- InfluxDB
- Google Cloud monitoring
- Google Cloud logging
- Hawkular-Metics (metrics only)
- OpenTSDB
- Monasca (metrics only)
- Kafka (metrics only)
- Riemann (metrics only)
- Elasticsearch
You can use multiple backends by specifying sinks on the commandline:
--sink=log --sink=influxdb:http://monitoring-influxdb:80/
cAdvisor is part of the kubelet, which runs on every node. It collects information about the CPU/cores usage, memory, network, and filesystems of each container. It provides a basic UI on port 4194, but, most importantly for Heapster, it provides all this information through the kubelet. Heapster records the information collected by cAdvisor on each node and stores it in its backend for analysis and visualization.
The cAdvisor UI is useful if you want to quickly verify that a particular node is set up correctly, for example, while creating a new cluster when Heapster is not hooked up yet.
Here is what it looks like:
