Bad configuration is an umbrella term. Your Kubernetes cluster state is configuration; your containers' command-line arguments are configuration; all the environment variables used by Kubernetes, your application services, and any third-party services are configuration; and all the configuration files are configuration. In some data-driven systems, configuration is stored in various data stores. Configuration issues are very common because, usually, there aren't any established good practices to test them. They often have various fallbacks (for example, search path for configuration files) and defaults, and the production-environment configuration is different to the development or staging environment.
At the Kubernetes cluster level, there are many possible configuration problems, as follows:
- Incorrect labeling of nodes, pods, or containers
- Scheduling pods without a replication controller
- Incorrect specification of ports for services
- Incorrect ConfigMap
Most of these problems can be addressed by having a proper automated deployment process, but you must have a deep understanding of your cluster architecture and how Kubernetes resources fit together.
Configuration problems typically occur after you change something. It is critical, after each deployment or manual change to the cluster, to verify its state.
Heapster and the dashboard are great options here. I suggest starting from the services and verifying that they are available, responsive, and functional. Then, you can dive deeper and verify that the system also operates within the expected performance parameters.
The logs also provide helpful hints and can pinpoint specific configuration options.