Chapter 7. Best Practices of Mobile DevOps
The pace of development and release for mobile apps is more rapid than previous forms of software and therefore your team must employ practices that wring every bit of inefficiency out of the develop-and-release lifecycle. Those practices that accelerate your team’s delivery are generally called “DevOps.” This chapter goes into detail covering the topic of DevOps best practices, and specifically how to apply them for mobile app projects.
What Is DevOps?
DevOps is an enterprise capability for continuous software delivery that enables organizations to seize market opportunities and reduce time to customer feedback. DevOps provides an approach for continuous delivery of software-driven innovation.
By applying lean and agile principles across the software delivery lifecycle, DevOps helps organizations deliver a differentiated and engaging customer experience, achieve quicker time to value, and gain increased capacity to innovate (Figure 7.1).
It is important to remember that the DevOps principles span the entire software delivery lifecycle. The spirit of DevOps is expanded collaboration among all stakeholders, not only between development and operations, but also among lines of business, suppliers involved in software delivery, and consumers themselves. In other words, the DevOps lifecycle is the software supply chain. All tools, processes, and stakeholders fall under its umbrella.
Some Definitions
The list of terms defined below will be used in this chapter:
• Continuous Integration—The practice in which software developers continuously, or frequently, integrate their work with that of other members of the development team. Continuous integration does not imply that every “change” to code is releasable or even deployable, but rather that the change did not cause an unforeseen breakage of functionality. The extent to which this is proven is determined by how much validation is built into the continuous integration process. The fact that code “compiles” is one measure of success. Successful passing of unit tests, or even better, some type of integration smoke tests, is a better indicator that the change did not cause a break.
• Continuous Delivery—The practice of automating the deployment of software to integration, testing, staging, production, etc. environments. Automation provides a repeatable machine-driven set of steps that reduces the tremendous amount of risk involved when humans must execute deployment processes. In addition to the removal of risk, continuous delivery provides a much faster deployment process in a time when speed-to-market continues to be a significant business driver.
• Continuous Testing—The practice of inserting testing into the continuous integration and delivery automated processes. The automated testing inserted is dependent on the type of validation desired. For example, unit testing as part of an automated build process (continuous integration) can give instant feedback on the usability of the build. Only builds that pass unit testing are valid for moving to the next step in the pipeline. Another example would be the execution of a series of smoke tests after an application has been deployed (continuous deployment). Only applications that can be initialized and that pass a series of basic functional tests are worthy of more detailed preproduction testing.
• Continuous Monitoring—The practice of applying monitoring to servers, services, data, hardware, software, and users in all environments of the deployment pipeline. If benchmarks are established during QA testing, for example, differences can be noticed earlier and performance tweaks applied before reaching production.
• Build and Deployment Pipeline—The practice of establishing (and automating as much as possible) the steps and validation gates required to push a version of an application through its necessary environments, ultimately producing a production release.
• Release Planning—A critical business function that outlines the release roadmaps, project plans, and delivery schedules as well as end-to-end traceability across these processes.
The IBM DevOps Solution
Driving innovation throughout your business takes more than new technology tweaks. Real innovation demands a different approach; a software delivery lifecycle integrated into the culture of your organization that unites trusted, dependable, and secure multi-platform applications with continuous software delivery. Enterprises that master advanced practices of software delivery achieve better business performance than those that do not. Traditional software delivery approaches that address only a single phase of the lifecycle are not sufficient. They simply shift the bottlenecks creating unnecessary waste and delays. The IBM DevOps solution addresses culture, process, and tools integration across the software delivery lifecycle from ideation to delivery. This bridges the divide among key stakeholders to establish a powerful and essential enterprise capability for continuous software delivery that leverages lean and agile principles removing waste, reducing time to customer feedback, and accelerating software delivery.
DevOps is even more important in the context of enterprise mobile applications. Mobile applications are sometimes the one and only system of interaction a customer desires. A relatively short but bad user experience can lose that customer for good.
The IBM DevOps solution can be broken down into the following topics:
Plan and Measure
All too often, the minutia of the software delivery lifecycle takes on a life of its own and before long the business value of a long running project begins to be questioned. Soon the point of no return has been reached and the project lives on to save face. To combat this common experience, organizations should establish a culture that strives to constantly reduce waste and increase business value by managing the end-to-end lifecycle. This can be achieved by continuously planning, measuring, and bringing business strategy and customer feedback into the development lifecycle. As the marketplace works today and with the business climate we face, organizations need to insure that they “do the right things” and focus on activities where they will gain the most value, and strive to “do things right” within the boundaries of organizational strategy and business objectives. Employing lean principles will help teams start small, identifying the outcomes and resources needed to test the business vision and value adapting and adjusting with agility.
Develop and Test
Keeping a competitive advantage requires the continuous innovation of ideas and the ability to translate them into products and services that bring value to your customers. Collaborative development and continuous testing support the evolution of a business idea into a high-quality software solution by applying lean principles, facilitating collaboration among all stakeholders, and striking the optimal balance between quality and time to market.
By establishing a solid development and test strategy, companies can ensure that their teams stay productive, remove unnecessary project risk, and slash the cost of software delivery.
Release and Deploy
The bread and butter of DevOps is the automation of the release and deployment processes. Tremendous business value can be achieved by automating these processes that drive down costs by eliminating errors and rework, speed time to market by the simple fact that machines can perform repeatable deployment tasks faster than humans, and reduce risk by delivering higher quality application releases with increased compliance and end-to-end transparency.
The hard costs of software release and deployment can be easily measured and applying a DevOps strategy to this aspect of the software delivery lifecycle provides a quantifiable business value return-on-investment.
Monitor and Optimize
The customer feedback loop is a key component in insuring that the right solution is delivered to the customer. Improved visibility and continuous feedback across the development lifecycle will help you exceed service levels. Monitoring solutions offer enterprise-class, easy-to-use reporting that helps developers and testers understand the performance and availability of their application while optimizing capabilities. The goal is to maximize the value of every customer visit and ensure that more transactions are completed successfully.
DevOps Best Practices
As has been already mentioned, DevOps is a series of principles applied across the software delivery lifecycle. The following is a list of prescriptions that can be applied universally and that all contribute back to the goals of DevOps:
Plan and Track Everything
The planning aspect of agile development is well-documented. A maniacal focus on planning is a direct attack against risk. DevOps brings to the table tools and techniques that promote faster processes (automation) and quicker customer feedback. Therefore, it is even more important that you have at your fingertips the information needed to make quick course corrections. Following DevOps principles will bring information back to the decision makers quickly. Therefore, the decision makers need to be prepared to react. The more data you have on the status of everything (see the next section) will allow those direction changes to occur with confidence.
But it takes effort and tooling to track things. It needs to be easy for a tester to submit a defect so that it gets into the backlog of things to fix. It needs to be easy for an analyst to inject new requirements into the process so they get evaluated, prioritized, and available for the next sprint. Planning and tracking takes effort that sometimes requires a dip in productivity until the benefits are realized. However, having a long range attitude towards planning and tracking will pay off.
Dashboard Everything
One of the tenets of DevOps is faster, or even more desirable, instant feedback. Faster and more intelligent decisions are required in order to seize opportunities and reduce time to customer feedback. Much of the decision-making data required to move forward faster is already present in numerous development tools, app store ratings, systems management tools, etc. Presenting this information at the fingertips of the decision makers is important.
Dashboarding data can be a difficult task. For example, if a release has made it to Pre Prod Staging, what type of information is required to determine if the release is ready for production? The obvious answer is the test results of the test cases that have been run on the release. But there may be more information needed, such as the features that are implemented in the release. Even though it may pass the necessary tests, it may not have the required functionality. This requires access to data from much earlier in the lifecycle which is typically not available on a deployment dashboard. Having access to all relevant data for a release (requirements, defects, test results, deployment environment readiness, etc.) is required.
Customer feedback is also very important. Mobile app stores today have easy to use rating systems for applications. Mobile app users are very accustomed to rating the apps they use after a very short user experience. A dashboard that provides a live look at an application’s ratings is a key deciding factor in how fast you rollout your next release. A bad customer experience that produces a bad rating is cause to accelerate that next release to recapture the customer base. Having this data as part of a dashboard is critical to mobile application success.
Version Everything
This prescription should be taken literally. Things that are versioned can participate in a repeatable and automated process in a way that ensures the integrity of the outcome. Version control systems are made to handle the authorization and authentication of users and the versioned content that they store. This ensures that if we repeat the same automated process, we should get the same results if the content provided to the automation has not changed. Versioning everything also allows anyone, not just those in the know, to be able to recreate an outcome.
Mostly everyone is familiar with version controlling code. But there are many other elements that participate in the software delivery lifecycle that should not be left uncontrolled. This list should include software configuration, test scripts, data definitions, data loads, and data change scripts, build and deployment scripts, and even system configurations. In this day and age of cloud solutions, the system patterns that are used to create environments should also be versioned.
Automate Everything
Automation is about tools, and it provides the engine that allows the benefits of DevOps to be realized. All of the cloud computing, continuous integration, and continuous delivery value comes from automating manual processes to increase speed and insure consistency. Organizations should continually strive to increase their automation.
Automation is in various stages of maturity across the software delivery lifecycle. Test automation has been around for years and has been a holy grail for testing organizations. More too often forgotten in the testing arena is the need for test data automation as well. Test automation only works if you can insure that the test data needed for the test cases is automatically set (or reset). Build and deployment automation has been in the hands of script writers forever. There are now enterprise-class continuous integration and continuous deployment solutions that finally provide the environment for the reusability and scalability of scripts that work. Cloud solutions are beginning to provide infrastructure automation. The ability for a multi-node complex environment to be spun up quickly and correctly in a matter of minutes or hours is now a reality.
Combining all of this automation provides a new reality—the ability to provide a new complex environment running a complex multi-tiered application in a matter of hours in what used to take weeks or months. Automation provides a unique ability to react to change faster than ever before.
Test Everything
The more automation that you create, the greater the need for testing. Testing provides the information needed to pass through gates in your delivery processes. A dashboard should not only tell if a build was successful, but also if that build is functional by observing the results of smoke tests. A successful deployment should not end at the “application is deployed,” but instead after the verification that the application passes some basic functionality tests. An environment in a cloud infrastructure should not only just be provisioned, but some type of testing should be provided to insure that the infrastructure is ready for deployment.
The testing community for years has done a good job of defining test categories (functional, integration, performance/load, user interface standards compliance, etc.) and there are numerous test automation solutions out there to reduce the strain on testing resources. DevOps strives to get smart about testing.
Monitor Everything
Monitoring software systems and servers is nothing new to the operations community. However, if the only focus is on production systems, you miss the ability to capture valuable information earlier in the lifecycle. Early detection of performance degradation may prevent an embarrassing rollout in production when your customers get to have their “first impression.”
Reducing time to customer feedback can also be expressed in terms of operations feedback. If the operations team can observe a server’s throughput statistics during a performance test run, it may provide valuable feedback that can stabilize a production deployment. This is even more important to mobile applications. Understanding bottleneck points in a mobile application during user acceptance testing is much more valuable than learning with real customers. Monitoring in preproduction environments is the key. Building in monitoring capabilities from the start will allow monitoring to take place at any level of testing.
Mobile DevOps Challenges
Enterprise mobile application development presents some unique challenges to implementing a DevOps strategy. The following is a list of mobile specific DevOps challenges:
Fragmented Platforms
Mobile application fragmentation can refer to the ever changing landscape of mobile platforms to target as well as supporting the ever growing versions of a single platform (such as Apple iOS or Google Android platforms). Fragmentation can also refer to differences in device platforms (screen sizes as an example), different programming environments, and even different app stores. Either way you look at it, fragmentation presents a challenge to the mobile application development space. Since this is a chapter on DevOps, we will not discuss the pros and cons of targeting various platforms or how many versions of Android to support, but instead address the general problem of supporting more than one target infrastructure.
Finding solutions to this problem can begin with your development environment. The silver bullet of “build once run everywhere” does not look like it will be here anytime soon. However, there are solutions that can get you closer. IBM Worklight provides a development environment that provides mechanisms to separate common code from platform-specific needs. At a minimum this prevents your need for duplicate code. However, you still must be conscious of what is platform specific and what is not.
Automation can be a large part of a solution for fragmentation. Build automation insures that you are pulling platform-specific content from the correct location when building deployment packages. If left to a human, the potential for error is magnitudes greater. Deployment automation insures that the correct application packaged for a particular app store gets to that app store. Again, the more left to the human the bigger the potential for error.
Mobile Applications Front a Complex Enterprise Back Office
The introduction of the concept of systems of engagement (SOE) by Geoffrey Moore1 has helped to highlight the shift in thinking of enterprise systems. System of Record (SOR) have been the focus of IT shops for years. Basically, SOE refers to the transition from current enterprise systems designed around discrete pieces of information (“records”) to systems which are more decentralized, incorporate technologies which encourage peer interactions, and which often leverage cloud technologies to provide the capabilities to enable those interactions. Mobile applications definitely fall into the category of a system of engagement. Mobile applications are decentralized and encourage peer interactions to say the least (Figure 7.2).
1 http://www.aiim.org/~/media/Files/AIIM%20White%20Papers/Systems-of-Engagement-Future-of-Enterprise-IT.ashx
The cadence of mobile application releases presents a challenge to up-to-now standard IT practices. Customers expect their mobile applications to be regularly updated providing new features and fixing bugs. Corporate backend systems, on the other hand, require lots of processes and approvals and are typically not updated that often. This presents numerous challenges.
• Availability of back office systems for frequent testing—The frequency of mobile application releases forces the enterprise to make available the necessary back office systems that interconnect with the mobile application. This isn’t always feasible. This then creates the need for back office service simulation. The ability to connect to a simulated service to perform basic functional tests makes the frequent mobile release cycle possible. It eliminates the need for valuable human and technical resources and allows the necessary mobile application testing to occur.
• Necessary updates—The introduction of mobile functionality will often cause the need for an update to backend functionality. However, as was mentioned above, backend systems are typically not updated frequently enough to support the ever changing mobile capabilities provided by the mobile application. This forces the need for solving these problems in not the most idealistic architectural way and sometimes forces a “creative solution” in the mobile layer that should be made in the back office system.
App Stores Add Additional Asynchronous Deployment Step
Continuous deployment prescribes that you automate the deployment of applications to each environment. However, mobile applications typically have the app store sitting between a newly built application version and it being deployed to a device. This is an asynchronous step that has to be taken into account. This can even be more challenging in the iOS space as the time and process of having a new version of an application ready for use in the app store is out of your hands.
Security, Code Signing, and Keystores
Security is probably more of a concern for mobile applications than any class of applications before it. The speed of adoption and the lack of control present unique security challenges for mobile applications. And the risk is as great for the user as it is for the enterprise. Insuring that a user is installing the approved banking application instead of a malicious imposter is a legitimate public concern. Code signing and the like have been introduced as a way to alleviate this concern, but it presents a challenge to the DevOps processes. The process of validating certificates adds complexity to the testing process and must be accounted for.
Testing
Testing is always a challenge. However, great strides have been made in the IT space to automate testing and to interconnect the tester with the broader software development toolset. Mobile applications put an additional strain on the testing process. Mobile platform fragmentation and the growing list of supported devices have made browser compatibility testing seem trivial. Simulators and emulators also have a place in mobile testing but they typically fall short in both speed and functionality to the real device. Testing on the device is a must, but it becomes a challenge to capture application and device data when bugs are found. It is important to gather as much device data as you can to help developers fix the issue. Also, submitting bug information itself is a challenge when testing on the device. Testers are used to having their test tool at their fingertips to be able to submit bugs when they are found. If a bug occurs while the field tester is at home with a device, the ability to submit the bug becomes a challenge.
Mobile DevOps Best Practices
There are numerous best practices that have emerged when implementing a DevOps strategy. The following is a list that attempts to illustrate how they are applied to mobile applications and the challenges to overcome:
Practice Continuous Integration/Delivery and Automate Builds and Deployments
The mobile build and delivery pipeline might look like Figure 7.3. A few of the challenges listed above can be seen in this illustration. The different target platforms (and platform fragmentation) all must be taken into account during the build process. Deployments to different environments are also a unique challenge for mobile applications as different stages of testing require different deployment targets. For example, early on for smoke testing, you might utilize an emulator and rely on a particular SDK for that deployment. In later environments, you need to deploy to a controlled set of users through an app store and the deployment process will be different. Release deployments might also include signing and/or the use of keys and the need to add some manual steps.
Mobile applications should require the same continuous integration rigor that you would apply to back office applications. Remembering that DevOps is all about faster feedback, continuous integration strives to insure developers get fast feedback on whether their changes have broken the build or caused tests to fail. By implementing continuous integration for mobile applications as well, these same benefits can be realized. The more automation you can include in your continuous integration strategy, the better (see below). Adding testing to your continuous integration process (JUnit, OCUnit, etc.) is even better. Taking advantage of automated testing solutions as part of continuous integration increases the value of the feedback.
It is safe to say that continuous integration is a valuable component to any mobile DevOps strategy. Continuous delivery, while more challenging for mobile applications, should also be included in your strategy. With the myriad of target platforms and variables that are no doubt part of each deployment process, an automated strategy for continuous delivery is critical. For example, deploying directly to an emulator or a device cloud may be OK at the early stages of testing, but using an actual app store (virtual or real) is most likely required when closer to release. The challenge of continuous delivery for mobile applications is, however, most likely worth the investment. Wasting time debugging deployment errors is a drastic hit to the fast and frequent delivery requirements of most mobile applications.
Test Each Build
Testing each build of the mobile app is an absolute requirement (Figure 7.4). Inserting as much testing and feedback into your continuous delivery processes is the key. However, testing mobile applications typically required numerous manual testers testing the application on different emulators or devices. However, today there are automated mobile application testing solutions out there (IBM Rational Test Workbench, IBM AppScan, Appium, Robotium) that can be integrated into your continuous delivery process. For example, if a new mobile build could be pushed to a device and a series of automated functional tests could be run with the results collected during the deployment process, you could save yourself from wasting time on builds that may pass unit tests but fail on a category of devices.
Simulate Backend Services to Expand Testing Environment Availability
As mentioned above, a significant challenge to mobile DevOps is that most enterprise applications are not stand-alone applications that maybe only rely on phone capabilities, but are most likely a key “system of engagement” that relies heavily on existing corporate business processes. Enterprise mobile applications participate in and contribute to a significant backend workload on both business processes and enterprise data. Therefore, it is important to insure that the functionality is right. One of the tenets of DevOps is that your testing environments are as close to production-like as possible. In order to make this happen, you must have a strategy to “enable” as many services as possible for testing mobile applications. This can be challenging, especially for those services that have a cost associated with them such as subscription services or services hosted by a mainframe. Another challenge is that the service you are relying on may not be ready or available yet.
You can attack this problem by simulating backend services. Obviously you get your best value if your simulated service behaves as close to the real service as possible. As testing environments are architected and designed, you determine what surrounding services are available. Typically it is difficult to provide all enterprise services for all levels of testing. Using service simulation allows you to provide more robust testing environments to meet the demands of the ever-changing mobile application (Figure 7.5).
From a DevOps perspective, it is also important that these simulated services are wired into the continuous delivery automation. This usually involves two details. First, you must wire the simulated service into the application as it is deployed to an environment. This may mean that configuration files are modified to alter service end points as a step in the delivery automation. The other detail involves insuring that the simulated service is up and running. This can also be added as part of the deployment automation (Figure 7.6).
Monitor Deployed Application and Backend Server Performance
Monitoring has traditionally been thought of as an operations problem. And therefore application performance issues are seldom caught until production or staging. The earlier you insert monitoring into your delivery lifecycle the sooner these types of problems can be captured (Figure 7.7). If server metrics are observed to be different during early QA testing, you have a much better chance of identifying and fixing the problem before it impacts customers.
Mobile applications impact performance in multiple areas. First of all a mobile application has an impact on the handset itself. A mobile application that performs poorly on a user’s handset will no doubt get little use and be promptly removed. Thorough handset resource monitoring during handset testing can help to prevent a bad user experience. Gathering handset resource statistics during testing will help developers fully understand the impact of their code. Secondly, with the mobile application being the “system of engagement” for corporate business processes, they have a direct impact on backend servers, services, and data. Mobile applications should force the reevaluation of service-level agreements to insure that backend systems can handle the additional payloads that a global mobile user base will create. Monitoring during early testing stages will provide data to operations that they can compare to production systems to potentially identify future impacts. And maybe most important of all, monitoring of user sentiment can also help to alleviate a launch disaster. This is discussed in more detail below.
Centralize Governance of Provisioning Profiles, Certificates, and API Keys
Provisioning profiles, certificates, and API keys are in place to insure that the user or caller to a service, API, etc. is valid and not malicious. As organizations establish connectivity to app stores, devices, or APIs throughout the development lifecycle, different certificates may be used. As you move your application to production, you must insure that you are using the right keys.
The first step in getting this right is to establishing a centralized repository with appropriate security and governance to hold these valuable artifacts. For example, if your application needs to access a public API for part of its functionality, the process of registering for the use of the API typically produces a key or certificate that you must present during the API access at runtime. The key is given during the registration process by the API owner and along with it is typically some type of usage license agreement. By storing the key in a controlled repository, you can insure that the key is only used by authorized applications and authorized developers.
Secondly, insert into your build and deployment automation processes steps to fetch the correct keys and include them into applications and/or configuration files. By automating this process you can insure that the correct key is used at the right time by the right application.
Use a Private App Store to Test Deployment Devices
Deploying your mobile application to Apple’s iTunes or Google Play is the final step in deploying your application to your ultimate end users if you are developing publicly consumable applications. This is not the time to test your deployment process to an app store. This process should be exercised and perfected in a private app store first. While this is not the forum to discuss the available private app store capabilities or contractual issues, it is important to stress the need for one. Again, one of the tenets of DevOps is to make each environment in your deployment pipeline as close to production as possible. And therefore insuring that you have practiced mobile application deployments to an app store is important for as many test environments as you have.
Convert Real User Feedback to Enhancements
Mobile applications that are distributed through an app store have the added benefit of capturing user sentiment as part of the app store experience. Mobile users are very accustomed to and expect the ability to provide feedback via rating the app in the app store. Development organizations have a unique and blatant feedback mechanism that needs to be taken advantage of. DevOps attempts to increase feedback loops to shorten delivery cycles and improve customer results. By translating real user feedback directly into enhancement requests (including the average rating or “number of stars”) you will directly impact the delivery schedule and solidify the prioritization of changes.
Summary
Mobile applications are now a mainstay in most every industry and have had a dramatic effect on software development strategies as the term “mobile first” indicates. Taking a mobile first approach to development has put new a burden on traditional software practices necessitating the need for new thinking. Mobile applications are bringing the enterprise closer than ever to the customer and the demands of the customer are greater and more impactful than ever before.
DevOps has come along to provide the approaches and tools to help tackle this new challenge. By applying the mobile DevOps principles described above to process improvement efforts, organizations can reap the benefits of the fast moving mobile marketplace.