During a deployment, whether it is for your microservices or something else, you are inevitably going to have to start and stop services and applications as a part of your deployment process. I am assuming here you have a managed, continuous deployment process, or, at the very least, not a manual process (yes, I've seen big clients manually promote across environment by hand picking change sets!).
In this chapter we will learn:
- How to create a deployment monitor microservice
- How to create messages specifically for this microservice
- How to handle events during a deployment
- How to tell if a deployment has taken too much time
Now, let's talk a little about what a typical deployment scenario might look like:
When a deployment starts, one or more services and applications affected in that environment (QA, staging, production, and so on) will have to be stopped to be updated. When this happens, the system is basically inoperable for the time period of the deployment.
Let me ask you this. How many folks can remember a deployment process that failed for one reason or another, and the next morning you came in to work only to find nothing working? By developing a deployment monitor microservice, we are attempting to mitigate this and other problem scenarios we might encounter. First and foremost, we are going to stop monitoring health checks for our system because there won't be any status being reported! The last thing we want to do is to start sending health alerts for our microservices not responding when the lack of response is known and valid.
Sending alerts and messages is a very important part of what this ecosystem does, and a huge value add to tell upper management. But be careful to avoid being the boy who always cried wolf. One day, the wolf will be there, and no one will be listening! I would advise sending messages to yourself until everything is perfected. While you are testing and perfecting the microservice, you should not send any email messages outside your own development arena. Even though I always tried to believe that transparency will be very beneficial, everyone wants transparency until everything becomes transparent! So don't send alert messages outside your close knit group until you are fully ready to explain what the root cause is and what the corrective action is. Otherwise, it's just another non-intuitive error message upper management is seeing and not understanding.
In talking about deployment processes that fail, we all know that there are times where a deployment fails or hangs for various reasons. The more your continuous development process becomes automated, the more likely it is you will encounter a deployment that did not complete correctly, for a plethora of reasons. In the microservice world, this is even more so than with applications, as our microservices support starting and stopping. What do we do if we are asked to stop when we are in the middle of processing messages? This and other questions need to be answered for us to be able to implement the microservice correctly. So, a good practice is that when the deployment monitoring microservice receives a message that a deployment has started, start a timer for a specific interval, say, 15 minutes. As soon as we receive the DeploymentStopMessage, stop the timer. If the 15 minutes elapses and we haven't yet received our DeploymentStopMessage, then we know there is a problem and we can begin to handle it, and hopefully mitigate the problem as well. Ultimately, we will need to send an alert email to notify users that the deployment is experiencing problems. I would recommend waiting until you gather as much information as you can so that your alert message is complete, intuitive, and is seen as valuable by all.