If we do not change direction soon we will end up where we are going.
—Dr. Irwin Corey
It is not enough just to refactor your database schemas within your development and integration sandboxes. You must also deploy the changes into production. The way that you do so must reflect your organization’s existing deployment process—you may need to improve this existing process to reflect the evolutionary approach described in this book. You will likely discover that your organization already has experience at deploying database changes, but because schema changes are often feared in many environments, you will also discover that your experiences have not been all that good. Time to change that.
The good news is that deploying database refactorings is much safer than deploying traditional database schema changes, assuming, of course, you are following the advice presented in this book. This is true for several reasons. First, individual database refactorings are less risky to deploy than traditional database schema changes because they are small and simple. However, collections of database refactorings, what you actually deploy, can become complex if you do not manage them well. This chapter provides advice for doing exactly that. Second, when you take a Test-Driven Development (TDD) approach, you have a full regression test suite in place that validates your refactorings. Knowing that the refactorings actually work enables you to deploy them with greater confidence. Third, by having a transition period during which both the old and new schemas exist in parallel, you are not required to also deploy a slew of application changes that reflect the schema changes.
To successfully deploy database refactorings into production, you need to adopt strategies for the following:
• Effectively deploying between sandboxes
• Applying bundles of database refactorings
• Scheduling deployment windows
• Deploying your system
• Removing deprecated schema
Chapter 1, “Evolutionary Database Development,” described the idea that you need a series of sandboxes in which to implement, test, and run your systems. As you see in Figure 4.1, each project team has a collection of developer sandboxes and possibly even their own corresponding demo sandbox. The preproduction test sandbox is shared between teams, as is the production environment. You can also see that there are deployment gates between the various sandboxes. It should be relatively easy to deploy database refactorings from a developer’s workstation into your shared project-integration sandbox because the impact of a mistake is fairly low: You only affect the team. It should be a little more difficult to deploy into your preproduction testing environment(s) because the impact of a mistake is much greater: Not only could your system be unavailable to the testers, it could also cause other systems within these environments to crash, thereby affecting other teams. Deploying into your production environment is often a rigorous process because the potential cost of a mistake is quite high because you could easily impact your customers.
To deploy into each sandbox, you will need to both build your application and run your database management scripts (the change log, the update log, and the data migration log, or equivalent, described in Chapter 3, “The Process of Database Refactoring”). The next step is to rerun your regression tests to ensure that your system still works—if it does not, you must fix it in your development sandbox, redeploy, and retest. The goal in your project-integration sandbox is to validate that the changes made by the individual developer (pairs) work together, whereas your goal in the preproduction test/QA sandbox is to validate that your system works well with the other systems within your organization.
It is quite common to see developers promote changes from their development sandboxes into the project-integration sandbox several times a day. As a team, you should strive to deploy your system into at least your demo environment at least once an iteration so that you can share your current working software with appropriate internal project stakeholders. Better yet, you should also deploy your system into your preproduction test environments so that it can be acceptance tested and system tested, ideally at least once an iteration. You want to deploy regularly into your preproduction environment for two reasons. First, you get concrete feedback as to how well your system actually works. Second, because you deploy regularly, you discover ways to get better at deployment—by running your installation scripts on a regular basis, including the tests that validate the installation scripts, you will quickly get them to the point where they are robust enough to deploy your system successfully into production.
Modern development teams work in short iterations; within an agile team, iterations of one or two weeks in length are quite common. But just because a team is developing working software each week, it does not mean that they are going to deploy a new version of the system into production each week. Typically, they will deploy into production once every few months. The implication is that the team will need to bundle up all the database refactorings that they performed since the last time they deployed into production so that they can be applied appropriately.
As you saw in Chapter 3, the easiest way to do this is just to treat each database refactoring as its own transaction that is implemented as a combination of data definition language (DDL) scripts to change the database schema and to migrate any data as appropriate, as well as changes to your program source code that accesses that portion of the schema. This transaction should be assigned a unique ID, a number or date/timestamp suffices, which enables you to put the refactorings in order. This allows you to apply the refactorings in order, either with a handwritten script or some sort of generic tool, in any of your sandboxes as you need.
Because you cannot assume that all database schemas will be identical to one another, you need some way to safely apply different combinations of refactorings to each schema. For example, consider the number-based scheme depicted in Figure 4.2. The various developer databases are each at different versions. Your database is at version 813, another at 811, and another at 815. Because your project-integration database is at version 811, we can tell that your database schema has been changed since the last time you synced up with the project-integration environment, that the second developer has not changed his version of the schema and has not yet obtained the most recent version (809 is less than 811), and that the third developer has made changes in parallel to yours. The changes must have been made in parallel because neither of you have promoted your changes to the integration environment yet (otherwise, that environment would have the same number as one of the two databases). To update the project-integration database, you need to run it for changes starting at 812; to update the preproduction test database, you start at change script 806; to update the demo environment, you start at change script 801; and to update the production database, you start at change number 758. Furthermore, you might not want to apply all changes to all versions—for example, you may only bring production up to version 794 for the next release.
One way to think about this is that at the beginning of development for a new release of your system, you start a new stack of database refactorings. Throughout your development efforts, you keep adding new schema changes to the stack, and sometimes remove some of them that you decide to back out of. At the end of the development of that release, you baseline the stack, accepting it as the bundle of schema changes for the current release.
A deployment window, often called a release window, is a specific point in time in which it is permissible to deploy a system into production. Your operations staff will likely have strict rules regarding when application teams may deploy systems. It is quite common to have rules such as application teams only being allowed to deploy new releases on Saturday evening between 2 a.m. and 6 a.m., and bug fixes between 4 a.m. and 6 a.m. on other evenings. These deployment windows are typically defined to coincide with periods of reduced system activity. Furthermore, they may have rules about when database schema changes are allowed to be made—for example, only on the third Saturday of each month. Smaller organizations, particularly those without many development projects underway, may choose to have deployment windows once every few months.
The implication is that your team will not be allowed to deploy your system into production whenever you want, but instead it must schedule deployment into a predefined deployment window. Figure 4.3 captures this concept, showing how two project teams schedule the deployment of their changes (including database refactorings) into available deployment windows. Sometimes there is nothing to deploy, sometimes one team has changes, and other times both teams have schema changes to deploy. The deployment windows in Figure 4.3 coincide with the final deployment from your preproduction test environment into your production environment in Figure 4.1.
You will naturally need to coordinate with any other teams that are deploying during the same deployment window. This coordination will occur long before you go to deploy, and frankly, the primary reason why your preproduction test environment exists is to provide a sandbox in which you can resolve multisystem issues. Regardless of how many database refactorings are to be applied to your production database, or how many teams those refactorings were developed by, they will have first been tested within your preproduction testing environment before being applied in production.
The primary benefit of defined deployment windows is that it provides a control mechanism over what goes into production when. This helps your operations staff to organize their time, it provides development teams with target dates to aim for, and sets the expectations of your end users as to when they might receive new functionality.
You generally will not deploy database refactorings on their own. Instead, you will deploy them as part of the overall deployment of one or more systems. Deployment is easiest when you have one application and one database to update, and this situation does occur in practice. Realistically, however, you must consider the situation in which you are deploying several systems and several data sources simultaneously. Figure 4.4 depicts a UML activity diagram overviewing the deployment process. You will need to do the following:
1. Back up your database. Although this is often difficult at best with large production databases, whenever possible you want to be able to back out to a known state if your deployment does not go well. One advantage of database refactorings is that they are small, so they are easy to back out of on an individual basis, an advantage that gets lost the longer you wait to deploy the refactorings because there is a greater chance that other refactorings will depend on it.
2. Run your previous regression tests. You first want to ensure that the existing production system(s) are in fact up and running properly. Although it should not have happened, someone may have inadvertently changed something that you do not know about. If the test suite corresponding to the previous deployment fails, your best bet is to abort and then investigate the problem. Note that you also need to ensure that your regression tests do not have any inadvertent side effects within your production environment. The implication is that you will need to be careful with your testing efforts.
3. Deploy your changed application(s). Follow your existing procedures to do this.
4. Deploy your database refactorings. You need to run the appropriate schema change scripts and data migration scripts to your data sources.
5. Run your current regression tests. After the application(s) and database refactorings have been deployed, you must run the current version of your test suite to verify that the deployed system(s) work in production. Once again, beware of side effects from your tests.
6. Back out if necessary. If your regression tests reveal serious defects, you must back out your applications and database schemas to the previous versions, in addition to restoring the database based on the backup from Step 1. If the deployment is complex, you may want to deploy in increments, testing each increment one at a time. An incremental deployment approach is more complex to implement but has the advantage that the entire deployment does not fail just because one portion of it is problematic.
7. Announce the deployment results. When systems are successfully deployed, you should let your stakeholders know immediately. Even if you have to abort the deployment, you should still announce what happened and when you will attempt to deploy again. You need to manage your stakeholders’ expectations—they are hoping for the successful deployment of one or more systems that they have been waiting for, so they are likely interested to hear how things have gone.
A database refactoring has not been truly deployed until you have removed the deprecated schema from production. When the transition period has ended, the deprecated schema and any scaffolding code, such as triggers to keep different versions of the schema synchronized, must be removed. Because the transition period may be several years, because that is how long it will take to update all the programs accessing the database, you need to have a process in place to manage these changes. The easiest approach, as described in Chapter 3, is to simply have specified dates (perhaps once a quarter) on which a transition period can end. The implication is that not only will you bundle up database schema improvements to apply them all at one time, you will also bundle up the removal of deprecated schema and apply those changes all at once.
We cannot say this enough: You must test thoroughly. Before removing the deprecated portions of the schema from production, you should first remove it from your preproduction test/QA environment and retest everything to ensure that it all still works. After you have done that, apply the changes in production, run your test suite there, and either back out or continue as appropriate.
Not only do you need to implement database refactorings, you also need to deploy them into production; otherwise, why do them at all? By having separate sandboxes, you can safely implement and test your refactorings to get them ready to be deployed. By deploying development versions of your system into your preproduction test environment on a regular basis, you improve and validate your deployment scripts long before you need to apply them within a production environment. Although you may develop working software on a regular basis, sometimes weekly, you generally will not release it into production that often. Instead, you will bundle up your database refactorings and deploy a collection of them all at one time during a predefined deployment window. Your application may not be the only one deploying into production during a given deployment window; therefore, you may need to coordinate with other teams to deploy successfully.