High availability is important to set up. If you go to the trouble of setting up high availability, it means that there is a business case for a highly available system. It follows that you want to test your reliable and highly available cluster before you deploy it to production (unless you're Netflix, where you test in production). Also, any change to the cluster may, in theory, break your high availability without disrupting other cluster functions. The essential point is that, just like anything else, if you don't test it, assume it doesn't work.

We've established that you need to test reliability and high availability. The best way to do it is to create a staging environment that replicates your production environment as closely as possible. This can get expensive. There are several ways to manage the cost:

• Ad hoc HA staging environment: Create a large HA cluster only for the duration of HA testing
• Compress time: Create interesting event streams and scenarios ahead of time, feed the input, and simulate the situations in rapid succession
• Combine HA testing with performance and stress testing: At the end of your performance and stress tests, overload the system and see how the reliability and high-availability configuration handles the load