The decision to go live is very dependent on the quality of end-to-end testing and user/organizational readiness, as mentioned earlier. The following are some experiences that I would like share in this area:
- Once, I was in room full of executives, making the decision about pulling the trigger on a new system. Everyone was under pressure from the CEO to say, ''We are ready''. However, most of them were not ready. They did not have enough time to go through the testing due to a lack of staff, but everyone said, ''yes''. (There was a fear of getting fired; this was way back in 2009 when the economy wasn't doing well). I failed to push back as well. Any guess as to what happened next? The customer went live, and it was very painful to stabilize them. But, lesson learnt!
- A similar situation occurred again, a couple of years later. Of course, I was smarter this time. The CIO called for a meeting to check the readiness on the project. Everyone said they were ready (the CIO was driving the dates very hard, and again there a was fear of getting fired). It was my turn—I bravely stood up and said, "No", handing over a list of areas I wasn't comfortable with and which needed more testing. The CIO called for another meeting to understand better what was needed to finish those areas and decided not to go live. We ended up extending the schedule by six weeks based on what was on the list. The CIO thanked me (and still continues to) for standing up and challenging the decision to go live based on the bugs that were reported/fixed in those six weeks.
- On another project, I was involved in the executive reviewer capacity, where I challenged their readiness, but the CEO did not want to listen. I told them it was their call, and we would support the release if they signed a liability waiver, as my team was not comfortable (due to lack of testing from the business team) with them going live. When we gave them a piece of paper to sign, the CEO chose to reconsider his decision. The customer ended up delaying the release by four weeks. The CEO who was not very happy when he received the push back, but now he feels thankful to my team for "watching his back".
There are more instances like these that I can share. The point is that you need to think about the client and the impact on their business. As a consultant, you are their advocate, and you need to protect the customer from hurting themselves (even though it's not what they wanted to hear, you are doing it in their best interest). This is the time to utilize the relationships and the respect you have earned from the customer to protect them. Don't be shy.
It is even trickier when you have to stand up for someone else's deliverables. For example, say the customer owns certain deliverables internally, and those are not production ready. You need to request the delay due to their internal deliverables, as you don't want to project to fail due to specific areas.
Saying that you need more testing is easy. The tough part is to decide how much more time you need. You won't get such an opportunity again. Thorough planning needs to be done to identify all the pieces that are incomplete and to put together a plan to come up with a realistic date. Many project managers fail in this exercise; just hitting the snooze button and delaying this by a few weeks may cost you a job eventually.
Picking realistic dates that will work for the business is important. You don't want to perform an ERP go-live right before or during the peak period for the business. challenges from the go-live will have a severe impact on the business. There are many examples of companies going out of business due to an ERP go-live during or just before the busy holiday season.
Part of your go-live planning must address the business contingency planning. Conduct a pre-mortem session to brainstorm areas that may go wrong, and to find ways in which the team can reduce the likelihood or mitigate the business impact if the issue occurs. Review the critical business functions that are important for the organization, and develop contingency plans to run the business if you did not have the computer systems in place momentarily. The goal of this exercise is to plan for the unknowns that may come your way. The following are a few examples to help with the brainstorming process:
- Third-party considerations: You have SLAs for next-day deliveries to the customers; work with your shipping carriers, and have them stand by to schedule a delayed pick up, in case you need it.
- Inventory levels: You have a great dependency on planning and the stock levels; consider beefing up your inventory prior to go-live.
- Additional workforce considerations: Look at adding temporary staff or approve overtime for areas that have changed the most or processes that would need more hand holding. I quote one of my customers, "If I was to make a mistake on spending here, I would rather make it by spending more than less".
- Additional technology resources to support the go-live: You may have a lot of things uncovered during go-live. It is like having an insurance policy: it's good to have it but it's better if you do not have to use it.
- Communication team: Have them stand by in case you need to communicate with the outside parties (customers and vendors) or even internally. You may not want to let the customers know ahead of time about the ERP release, as they would consider it as an upcoming glitch and go somewhere else.
- Cash flows: It may take a little longer to get paid for a few weeks (or months) after go-live due to system or training issues. You need to have an additional line of credit available in case you need it.
- Key processes and proactive planning: Identify the key processes and their first occurrence to provide some hand holding and validation. For example, after processing the checks for the first time, ensure that you can validate them against the checks that passed testing. The first time you are ready to start invoicing, try a few orders first and verify the results before you open the flood gates of batching hundreds of orders. In case of files that are supposed to be sent out (like EDI or positive pay files for the bank), verify those that were sent and accepted/processed faultlessly at the receiving end.
- Going back to the previous system after a few days or few hours into using the new system: Once you move to new ERP system, it may not be possible to go back to the previous system. It is not easy to perform a reverse migration from new platform to legacy. Make sure everyone understands that once you are live, there is no going back. Everybody is in it together and issues need to be resolved on the new system. This helps in avoiding unproductive discussion of going back to the previous system after going live.
- Running parallel: It seems like an easy solution for business contingency planning. However, it may not be practical unless you are staffed high and the transaction volume is not high.
- Running parallel usually adds more burden and stress on your staff while they are trying to deal with new system. In general, it would cause more issues/noise (as users would make more mistakes under stress) than helping.
- If you had to go for running both the systems in parallel, the amount of time to run parallel should be kept to a minimum.
- Very often, running parallel is looked at as a replacement for inadequate testing; you think you are better off not doing testing in production (and hoping everything would be fine).
- There is sometimes a belief that running parallel helps validate the new processes against the old system processes. This is a fallacy, as part of the point of implementing a new system is to improve the processes that may fundamentally change the way you do business. So it is like trying to compare apples to oranges.
- Release validation: As mentioned earlier, going back to the previous system or running parallel is not easy. How can you verify that there are no critical issues as part of the release itself? The following are are some ideas:
- Set aside a good amount of time in your release plan to allow for the release validation.
- Once the business has signed off data migration in the production environment, they can start the release validation testing. You don't want to start creating new transactions until data validation is complete, else the numbers won't tie.
- Identify the key end-to-end processes, and processes by functional areas to be verified; keep the step-by-step validation scripts ready.
- It is ideal if you can hold certain transactions from the previous day. For example, let the AP team enter real vendor invoices, perform a check run based on what's due, orders that were received from the customers during system downtime, incoming EDI transactions, and so on. The goal is to have good samples and real transactions to verify the system behavior.
- In one of my projects, performance was a concern. The customer involved a good part of their sales team to validate the order entry process after the initial validation was completed.
- This process enables you to identify any remaining bugs (due to setup or code push issues) and resolve them or make a cautious decision to go-live or rollback.
- Once you are live, there is no going back.
These are some ideas based on past experiences. You need to review these with the business leaders and determine what is applicable to your business to make appropriate arrangements.
The following are some technical tips to keep in mind while planning for the go-live:
- Have tools like Dynamics Perf and Perf Monitor installed and configured in the production environment ahead of time. You can use them when needed.
- Set up an additional AOS server for troubleshooting. This is helpful when you want to troubleshoot a specific process by a developer with the business user. Provide an icon for connecting this AOS to the business user (an AXC file would connect to this AOS). That way, the developer can take traces and the like in an isolated fashion only for this user. You do not have to turn tracing on for all other users, which would slow down the environment and give you a lot of noise in the traces. This would reduce the effort needed to reproduce the issue in the non-production environment and help find a quicker resolution.
- It is not uncommon to see a high number of security requests come through in the first few days after the go-live (for example, changing user roles, giving them additional roles, setting up new users, allowing more access, and so on); dedicate resources to handle such requests. Once the initial security issues have been resolved, you may want to start looking at what you would need to pass a system audit and ways in which you can start taking away the privileges that may not be needed.
- Provide a heads up to the Microsoft Support team for large releases, more importantly if you are an early adopter of a specific feature or release. Try to get additional consulting support for the go-live. The onshore and offshore models are very helpful to keep the ball rolling round the clock on critical issues.
- Consider engaging the Microsoft Premier Support team to perform a proactive health check prior to the release, and to be available on standby during/and for a few days after release. This is especially important if you have a high transaction-volume environment.
- Another checkpoint is to ensure that backups are being taken and indexed and other SQL maintenance jobs are scheduled.
- Use the latest kernel version for AOS servers. Updating the latest Kernel version does not need much effort for deployment and testing. However, it would help you get to a more stable state.
- Check if there are any hotfixes that have been released recently and which should be considered. You can check the hotfixes in the Lifecycle Services portal or you can check if any cumulative update has been released recently. Review the list of hotfixes that are critical. This is very important if you are an early adopter of a new version/module.
- Refresh the data into the non-production environment for troubleshooting. This would help troubleshoot and fix issues quickly. You need to do this with caution though. Make sure that you have captured the steps (preferably automated) needed for scrubbing the environment specific connections. I have seen situations where these steps were missed during refresh and the test environment charged real credit cards or printed real shipping labels, connected to real instances of tax calculation software, and so on. The Test Data Transfer tool can be a better solution for this instead of a data restore. Using the Test Data Transfer tool, you can exclude specific tables containing the system configuration data. Refer to Chapter 5, Data Migration – Scoping through Delivery to learn more about the capabilities of the Test Data Transfer tool.