Quality, budget (schedule and resources), and scope are the fundamental constraints on every ERP project. Most of the time, when the scope is increased and the budget stays constant, the quality gets compromised. One of the biggest mistakes that people end up making, especially on fixed-bid contracts, is that they reduce the testing budget when there is budget pressure.
I have heard this several times and would like to call it out. In many post-release postmortem meetings, I would hear, ''If I was to do this again, I would spend a hundred thousand dollars more on testing.'' You have the opportunity to do it right and not having to regret it later. Let's review the different phases of testing and their importance/execution:
- Test plan
- Unit testing and feature testing
- System integration testing
- User acceptance testing
- End-to-end testing
Similar to testing, training is another key aspect for the successful implementation of an ERP system. Ensuring that the users are comfortable with the new platform and understand the new business processes and their new role in the organization is essential for attaining a good working platform. For example, usually, the finance department is involved in journal entries for re-classing entry errors and playing a tactical role within the organization. In their new world, after the project implementation, they will be managing exceptions and reviewing key KPIs/trends. Training needs to be delivered to support this cultural shift.
The process of unlearning the old practices and learning new ways of doing things may take several iterations. Hence, training and an evaluation of adopting of that training are very essential. We will discuss the following important aspects of training in this chapter:
- Putting together a training plan
- Training preparation
- Change management
- Training tools
While discussing this topic on testing and training, our focus will NOT be on generic areas; we will talk about them at a higher level and focus on the Dynamics AX specific elements.
In this chapter we will learn the following:
- Key considerations for testing a Dynamics AX solution
- Training plan and execution
Testing is the process of validating the system and processes to meet the business requirements. It includes testing the custom as well as the standard features, along with the migrated data, integrations, reports, and security aspects of the solution. It is an area that is most often underestimated and, as a result, hampers the success of your project.
A very common misconception is that testing starts after the development phase is over. The primary goal of testing is to provide feedback on the product as soon as possible. Identifying any issues in the requirements phase prevents them from becoming a part of the design. Similarly, identifying any issues in the design phase prevents them from being coded. The cost of fixing a defect depends on the phase where it has been detected; the cost of fixing a defect in the early phases of SDLC (Software Development Life Cycle) is much lower than in the later phases. The farther you go with the backlog of testing/validation, the more debt you carry on the project. Mostly, such a debt gets unmanageable and it becomes hard to predict/commit to the schedule. Remember, you are not the government to carry high debts—the more you add to it or the longer you carry it, the worse it gets.
The following are some guidelines to keep in mind when planning for the testing phase:
- During the planning phase, create a test plan to define the scope, resources, and tools to be used for the testing, and to identify how bugs will be tracked. Establish the criteria for defining S1/S2, P1/P2 bugs depending on the business criticality (severity and priorities).
- Dedicate QA resources for each area in the similar way we do for the functional analysts and developers. You need them to start on the project right from the beginning to understand the requirements and the design that is being put in place. Good QA resources will have very valuable inputs in the design phase, and they can watch out for design gaps.
- Identify the external resources that need to be engaged during testing. For example, testing with banks for checks/electronic payments, positive pay files, EDI trading partners (customers/vendors, and any other parties to whom you send/receive data like D&B (credit), third-party invoice printing, and so on. Start engaging them as early as possible, and align their schedule into the project plan.
- Plan for performance and load testing in addition to functional testing. performance and load testing is addressed in detail in Chapter 10, Performance Tuning.
- The automation of testing is used in very large Dynamics AX deployments that can afford to invest in the automation of testing scripts, and where you need to verify the generic processes repeatedly during the development and stabilization phases. Many such customers end up investing in test automation after going live and once they have stable processes defined to reduce the rework in test automation.
Building test scenarios and cases are important for executing a good test plan. The following tips will help you in developing effective testing scenarios:
- Prepare test scenarios and test cases while the development is in progress. Review test cases with the business analysts and the business SMEs, as applicable.
- Align test scenarios to the business scenarios that are put together by the business team.
- The goal should to be identify and document each scenario in detail in the form of test cases rather than staying at a very high level. If you don't document the test cases, there is a high chance of missing them during execution. If you were to hire an army of temp staff to perform the testing, they would need enough details to execute all the test cases based on the documentation.
- Every test case must have the following sections:
Test-case ID
Test scenario
Title
Prerequisites
Role(s)
Steps
Expected result
- Maintain a traceability matrix with the number of requirements, function specifications, technical specifications, test scenarios, and test-case ID.
- Identify the test data to be used and the specific deviations in data to maximize the coverage of your testing. Say, for example, if a company has four product lines and all are sold differently, you will need to have scenarios that address each product line.
Unit testing is the standalone testing of customizations, usually performed by the developers to ensure that an individual customization is working as expected. Although in software engineering terms, unit testing typically refers to automated testing, the developers can run and execute manual tests to ensure the completeness of the feature.
It is important for developers to perform unit testing to ensure that the feature developed is working correctly. Most of the time, it is seen that the development environment does not have the right data set, and this is often used as an excuse for not testing the feature in the development environment. Plan on having appropriate configuration and transaction data in the development environment to encourage early unit testing.
Unit testing provides many benefits that include finding bugs earlier, providing a safety net of tests for changes that are made later, and improving design. Over the long term, unit testing improves customer satisfaction and developer productivity.
Function testing, also known as feature testing, is standalone testing performed during development by the QA resources or business analysts. As each feature is gets developed, engage the QA resources and business analysts to use test cases for testing it. Engage the business team members to review individual features that will help reduce surprises during the UAT phase.
Performing feature testing in the development phase will validate that the requirements are being met by that functionality. Testing at the features level enables a fast turnaround on defects, which improves the efficiency of the development process.
Start testing the security roles at this stage. Ensure that the roles defined can use the intended functionality. You can use the Security Development Tool for security role testing without using any additional test accounts. It allows you to test a newly created or modified security role, duty, or privilege from the development workspace using the security permissions associated with that security artifact. The security development tool can be downloaded from the Lifecycle Services portal.
Testing the integrations with other systems that have been developed is just as important as the features and functional testing of the product itself. The following are some points to think about when planning and executing system integration testing:
- System integration testing is performed across applications to verify the seamless flow of information. By this time, all individual features should have been tested by the QA and business analyst resources.
- You will uncover integration issues across the streams, like data migration and reporting, or across applications during this phase.
- The go-live plan should be used to migrate and verify the data in the system integration test environment.
- All individual applications must be tested independently and made ready for integration testing. You will need integrated environment across applications to perform this testing.
- Carry out a QA team exercise for going through the UAT test cases (including user roles) to validate readiness and a smoother UAT phase.
The goal of User Acceptance Testing (UAT) is to engage the users across business groups who will be using the new system to run the business. This is an opportunity to provide them with hands-on experience for learning the new system as well. The more testing that the users perform, the more comfortable they will be using the new system.
There is quite a bit of planning required to perform a successful UAT. Just throwing the business users into a room with computers and test scripts is not going to get you the results of an effective UAT. The following details should be considered when planning for UAT:
- Plan multiple rounds of testing, scheduled a few weeks apart to fix issues. It is not uncommon to find a few pieces missing once the business starts looking at the solution. The goal should be to fix everything in between both the cycles so that the business does not experience the same issues or the test cases are not blocked due to those issues.
- Prepare a list of all the batch jobs, their frequency, business/IT owners, dependencies, the path in Dynamics AX, batch group, and so on. This needs to be well documented. All the batch jobs should be scheduled, results validated, and errors/execution should to be monitored.
- Engage the DBAs and the IT operations team to monitor UAT environment, like they do in production. Similar to the business getting ready to use their new system, DBAs and IT support teams need to get familiar with their new toy for troubleshooting.
- Set up and verify the security roles assignment, access to the UAT environment, and reports for all the users. Do not use the system administrator roles for testing during UAT.
- You need to have the full data migration completed in the UAT environment. Reconcile the migrated data as a first step in UAT, before you start making changes in the environment. The Go Live plan should be in place and be used for UAT data migration. Refer to Chapter 5, Data Migration— Scoping through Delivery for additional tricks on data migration testing.
- Fix the test cases and their sequencing to ensure the correct flow. For example, you start with the data migration validation and then move on to customer/product creation, to order processing, shipping, invoicing, processing returns, commission reports, financial postings and financial statements, tax reporting, inventory value reports, and so on.
- You need to have the environments locked down to a few folks from IT (only the operations team responsible for deployments and the business analysts directly supporting the business users during testing). No changes should be promoted without going through a change control/formal release process.
- The people who run the business should be engaged to verify the system; the team should have cross-functional knowledge and knowledge case scenarios. For example, your top performing, most brilliant sales talent pool needs to be involved in testing the order entry system. They will know all the different scenarios and gotchas from the current system, and they can help you break the system.
- Avoid relying on the temporary staff for testing; you need FTEs to review your new world. Engage the temporary staff in backfilling the FTE jobs to run the day-to-day business tasks, not in reviewing the future of the company.
- Encourage the business to bring in as many real examples as possible. For example, the AP can bring in a day's worth of a stack of invoices for processing, running a check run on the migrated Open AP and newly created AP invoices to review the results and real customer orders for order entry. This will help verify credit limits, customers/products, on-hand inventory migration, and the related scenarios.
- The key messages that should be put across during the UAT kickoff are as follows:
- Finding bugs is the goal of performing UAT. If you find them in UAT, that's a great thing. Don't get frustrated because you've find issues and thus, get stuck in testing.
- Focus on first verifying all the critical business scenarios before getting into exception scenarios that won't happen frequently. Follow the 80-20 rule to define focus. This is also a good time to remind everybody about the goals for the project.
- Review the reports from previous testing and communicate any open areas.
- Communicate the schedule for testing and re-testing.
- Cover the tools/process to be used for logging bugs, triage, and communication after fixing the bugs.
- Set the sign off and exit criteria (communicate upfront that they need to sign off at the end of it).
- Provide templates for logging the bugs (capture screenshots, provide a reference to the test case, the step that failed, description of the issue being reported, any input file used for uploads, business impact, and so on for every issue that is being submitted by the users). Users need to be educated on bug-tracking tools and the overall triage process. The more information you have, the less time will be required for the development team to analyze and fix the issues.
During the execution of UAT, consider the following points:
- Track the testing progress along with the test cases that passed/failed. Publish reports on progress, bugs reported, and resolved bugs (for re-testing).
- Actively manage blocking issues. You need to stay on top of the issues that are blocking testing of certain areas; try to be creative in finding workarounds to continue testing.
- Issue a triage and managing issue list. Have multiple reviews with the team every day for issue statuses and resolutions. Set daily meetings with the business leaders to discuss issues and provide updates on the progress made. You need to hear their firsthand feedback on the issues being experienced.
- Ensure that the formal release process is defined and validations are performed to verify that the release has not broken the environment. This will ensure that precious testing time is not lost due to the broken UAT environment, which is especially important when you have multiple applications integrated with Dynamics AX.
- Track dependencies between test cases. You may have a dependency between the test cases that will need coordination among the different business groups for testing. For example, when a sales order is created, you need to verify with the warehouse to ship it, and then the AR can see the invoice and collect against it.
- On larger projects with a multi-location roll out, it is a good idea to execute testing at a central location. However, you should also perform some testing locally. For example, we had the warehouse and all other users gathered inside the HQ for testing for one of our clients. They tested the shipping labels and the product labels, and it was all okay at the HQ. However, when testing was done in the warehouse, it was terribly slow. As the print was sent from the terminal server at the HQ to a network printer in the warehouse, the network latency was making the process very slow. Upon making some tweaks with the help of the network team and some code changes, we were able to get it working in the warehouse as well.
- Poor analysis and design for complex areas will get exposed in UAT and cause a lot of rework/continuous break-fix. Identify such critical areas and put in dedicated resources to get extra focus on such critical path items. In one of my large Dynamics AX implementations, a focus team was defined for testing and fixing the revenue recognition and deferral scenarios. It was one the most complex parts of the project, and was dependent on many other processes like correct product and customer setup, order entry with different combinations of products and the way billing frequency was chosen by the customer, order entry and CRM integrations, invoice distribution and rounding of totals, and so on. Every time one scenario was fixed, another was broken in deferrals; issues in the upstream processes like order entry impacted the testing of deferrals functionality. The focus group helped track this subproject with additional visibility and helped fix issues faster.
- If complex features and processes that are dependent on several other upstream features are not tested until the later part of UAT, it leaves you with very little time to test the complex areas. Creating focus groups early on, or having an additional round of testing focused on such features, can help reduce the exposure.
- Testing with the system admin role in UAT will force you to go live with a large number of users in a system admin role. This will cause issues post go-live and call for testing rework once you have the roles defined. One customer insisted on continuing testing with the system admin role and went live with many system admins. Someone accidently (I am sure it was unknowingly) unchecked two check boxes in Production (Post Physical inventory to GL and the Post Financial inventory GL on Inventory Model group). No inventory transactions were posted to the GL for two weeks until the issue was identified by the finance team. It was a project by itself to retroactively come up with GL entries, posting of accrual entries for several months until all the POs were invoiced, and so on. Would you like to be in a similar situation?
A successful UAT is one where the business can show that they are comfortable with the application features, and thorough testing has been done with good involvement of the business users across areas. The key deliverable of UAT is the business sign off on the User Acceptance Testing (UAT) and test results. There may be cases where items do fail, but the team agrees to a conditional sign off. Track any bugs that are critical for going live as a part of this conditional sign off. Most importantly, all areas should have been tested by now. There is a difference between knowing the open issues and being unable to test specific areas due to open issues.
In addition to UAT, you need another round of testing to verify the end-to-end execution of a business process. The key difference between UAT and end-to-end testing is that UAT is more focused on validating individual business processes, while end-to-end is focused on validating all of them together once each of them have been stabilized and tested.
You need to have the testing of individual features completed and all the areas to be stable to truly start end-to-end testing. In reality, sometimes you end up making some exceptions, but this is not ideal.
Pick a selected core group for end-to-end testing. Everyone involved needs to know the end-to-end business flow. Usually, the finance team has a bigger role to play here as they have a visibility into all the parts of the organization.
Plan for at least two rounds of end-to-end testing with some time in between to fix the bugs.
Define the exit and success criteria prior to getting into end-to-end testing (such as 100 percent test execution, more than 95 percent pass rate, no more than five critical bugs open, and so on).
Similar to UAT, you need to publish reports on the test results and follow a triage process. Areas that are blocked during testing need to be unblocked and tested again. Assess whether you have met the exit criteria and review with the executives.
The following are few examples of areas that you should focus on during end-to-end testing:
- The goal of end-to-end testing is to simulate real business. Right from data migration to new product and customer creation, using this data for placing orders, fulfillment, invoicing, receiving cash, reverse logistics, transactions using migrated data, verify reporting, and so on. Come up with all the key business scenarios that should be tested:
- Customer invoicing: The timing and accuracy of invoicing customers is such a critical business function because it has a direct impact on the customer and on the cash flow of the company. On the other hand, invoicing is a downstream function—you have a dependency on products, customers, tax, fulfillment processes, and so on—which must work correctly before you can produce the invoices.
- Commission reporting: As commission reporting has an impact on the paychecks of the sales floor, you need to verify the accuracy of the commission reports with migrated orders and invoices. It should be a top priority, as you want the sales team to trust the system and focus on selling rather than tracking their orders on spreadsheets for an expected commission or worrying whether they'll be paid. Commission reporting could be even trickier for orders shipped in the previous system, and you might have to pay a commission upon receiving customer payments.
- Inventory costing and valuation: Each customer has a different way of using weighted average, FIFO, and other inventory costing methods. It impacts the P&L, their bottom line, how executives are compensated, the inventory value on the balance sheet, and so on. Efforts need to be put in during UAT and end-to-end testing to validate that inventory costing is done according to the needs of the company and understood by the financial controllers and the rest of the stakeholders.
- General ledger postings: You need to verify the posting for each type of transaction, and run month-end reconciliation reports (to verify that the general ledger and sub ledger are in balance).
- Key reports: Identify the key reports that are important to run the business, and validate the data based on the transactions that were processed in end-to-end testing.
- Engage domain experts during end-to-end testing like tax auditors for tax integration testing. They will be able to put together a great test plan and execute through unique scenarios to ensure that you have configured the system correctly. I have seen great examples of bugs that the tax auditors discovered (I don't think the internal team may have found them even after a few weeks of release).
Training drives the successful adoption of the new system and processes. The learning capacity of the audience and the amount of changes being introduced to them dictate the amount of time you need to spend on training and re-training. The more people you have up to speed on the new processes and system, smaller the chances of them making mistakes, and the volume of support calls will be highly reduced. Ultimately, this results in a smoother adoption of the new system.
The ERP project is an opportunity for organizations to get people up to speed on end-to-end processes and training them on cross functional areas. If you have a great system designed but people are not able to use it, can you call it a success?
Put together a training plan that covers the following points:
- Understanding the audience: How quickly are your users likely to catch up with the changes? The training plan needs to be defined accordingly to support their transition.
- Trainers: Consultants train the trainer - the business super users or internal business analysts. Hopefully, multiple rounds of CRP will help the business users and internal business analysts get up to speed on the system in order to be trainers.
- Scope of training / areas to be trained: You need to account for both the system and the process changes. There are three usual cycles of the training process—UAT training, end user training, and post go-live (training for areas that are struggling).
- Logistics: This includes factors like meeting rooms/travel, centralized versus location-specific, and so on.
- Training schedule and timing: Timing is key. In some areas, you may need to train the users multiple times to ensure that they are comfortable. On the other hand, areas that have not changed much may need light training, close to going live, to ensure that the users don't forget.
- Training assessment: This pertains to the ways and methods that you will use to get a feedback on the training process.
- Retraining: Build retraining into the plan, and modify it based on the feedback.
- Training material and user manuals: Reviewed this with the business SMEs. It may come in different forms. For example, checklists, Visio for business processes, documents with screen shots, recorded videos, mapping between the old versus new world, and/or a combination of multiple methods. The development of the training materials should be agreed to at the beginning of the project, in the planning phase, so that the appropriate time and resources needed are built into the plan.
- Signing off: Define the sign off process and the criteria for training sign off. It is one of the major considerations for go-live.
The usual human psychology is to resist change. In the earlier chapters, we talked about minimizing the process changes along with the major releases to focus on the system side of implementation. You will still be left with a good amount of change management due to the new system. Change management may shift jobs or workload from one department to another. The project managers of an ERP project often end up utilizing a lot of political power to fight such battles for getting the right decisions made. Even though you may have won the battle for driving the change, you may have lost a few partners that you needed to champion the project. Hence, leaving the major business process changes for the transformation phase of the project (post go-live) can be wiser and a better way to manage the scope.
Training is a good opportunity to help prepare people for the change. The more training you provide, the higher the confidence that the users will have in embracing the change, and you will receive less pushback.
A lot of preparation goes into executing a smooth and effective training. This preparation includes validating system readiness, verifying the roles, putting together multiple forms of training materials, creating and maintaining a stable training environment with valid data, and so on. We will discuss these aspects in the next section.
Ensure that the system is ready and stable enough (testing complete) prior to training a larger audience. You also need to gauge the business readiness for training and help them prepare. The following are some tips to do so:
- If a smaller group is being trained prior to UAT, the expectation may be different, as the system has not been tested as yet, and you may want to communicate the known open issues. However, when training larger groups, try to do it post UAT when the system is stable enough and the processes have been finalized.
- Create a forum for the users to participate in and get more hands on experience from training through go-live. Arrange Lunch n Learn or other such team activities that will encourage more practice. From my experience, business leaders that encourage their teams for extra practice after training, and take the initiative to drive it, will have a lot less issues to deal with post release.
- Have a process to capture and respond to bugs/queries that are raised in the training. Most likely, you will find some critical items that were not known before.
Let every user be configured with their to be production security role. Avoid the use of the system admin role during training. Of course, roles should have been tested prior to doing this.
Use the business process flows at the beginning of every session. Give a ten-thousand feet high walkthrough of the overall business process and of the piece that you plan to show before getting into application and details.
Engage the Business SMEs or internal business analysts to do the training or to assist in the delivery of the training:
- To train others, you need to get up to speed first; the person that learns the most is the trainer. The trainer approach will ensure that the business SMEs or internal business analysts have got up to speed well enough. It will reduce the dependency on the consulting team post go-live, and internal resources can be your tier 1 support.
- The internal SMEs can reference the current process/systems during training and will help in delivering the training.
- Once I had the controller of an organization deliver AP training; he was able to speak their language and relate to the screens of the existing system. It helped the users in mapping their old versus new world easily, and the training was very well received.
Training manuals and user guides are good references for the users to look at post training. An ideal team for developing the training manuals should include business super users, technical writers, and business analysts. Distribute them in a medium that the users are comfortable with—putting check lists at desks, posters in the building, or binders or electronic formats on shared drives are the commonly used methods.
In each cycle of training, use the training manuals, and make corrections based on the feedback. The training manuals will stay as living documents and can help on board the new employees, process documentation.
There are a few very powerful tools available to build training manuals for Dynamics AX, which are described in the following section:
Users can use the Task Recorder in Dynamics AX to quickly document a business process or task for training or other purposes. You can use the Task recorder to create videos or documents in Microsoft Word, Microsoft PowerPoint, and Microsoft Visio. If you are using the Task Recorder in advanced mode, you can also capture additional metadata that can be packaged and then uploaded to the business process modeler in Microsoft Dynamics Lifecycle Services. The file that you upload includes cross-functional flowcharts and activities that you can modify to identify the business requirements and generate implementation artifacts.
Be sure to have a script to follow for the process that you plan to document, so that you can avoid unnecessary clicks while the recording is on. Once you are done with the recording, generate a word document and clean up the extra steps that you do not want in the document.
In Lifecycle Services, you can use the business process modeler to create, view, and modify the business-process libraries and flowcharts for Microsoft Dynamics AX. Business process modeler helps you align your Dynamics AX processes with industry-standard processes, as described by American Productivity and Quality Center (APQC). There are more than a thousand business processes that are available, and you can tweak them as per your needs. As referenced in the earlier chapters, business process modeler can be used right from the Gap/Fit analysis phase of the project to track all customizations. You can convert the process flows in Microsoft Word or Visio for use in training.
The Microsoft Dynamics AX Help enables you to add new documentation, update existing documentation, and add entries to the table of contents. To customize the documentation, you add one or more files to the Help server.
You can press F1 to get the help from any form. Microsoft ships the product with the help documentation and the content can be added or modified according to your requirements.
This is a good feature in Dynamics AX. Users can personalize their screens such as add/hide fields. However, while training the users, you need to remind them about the side-effects of personalization which are as follows:
- If you are on a support call with the helpdesk team, they may not be seeing the same information as you are seeing on your screen.
- It may vanish with the system updates or during troubleshooting. One of the first troubleshooting steps from Microsoft is to delete the personalization if the user is facing issues. Advise the users to document the screens that were personalized or save their personalization, so those can be added back quickly when lost.
Having a stable training environment is important for successful training. A lot of time will be wasted in training if the training environment is not in a good working order. Look at the following tips to keep in mind when managing your training environment:
- You need to treat it like production; many people will be using it at the same time and you want it to be stable while the training is going on or when the users are practicing after training.
- Keep it updated with the latest code and data. Have a communication plan for any downtimes for deployments to ensure that the users are aware.
- Have it available for the users to practice after training.
- It should share an integrated environment with the other applications. For example, if you plan to use Dynamics CRM in Production for order entry and integrate it with Dynamics AX for fulfillment, ensure that you have the training instance of the CRM connected to the Dynamics AX training environment. This will ensure an end-to-end training experience for the users.
Note
Example of issues from poor training
One of my customers on-boarded the temporary staff just before the go-live to help with the returns processing. They knew that the returns volume was going to be high for the first two weeks, as the returns were put on hold for a couple of weeks prior to go-live (to avoid any in-process returns that would have to be migrated). Also, the returns team had to go through two different systems for a period of time to verify the original purchase in the old system and enter the RMA in Dynamics AX. Due to lack of training, the temporary staff made a lot of mistakes. One of them input the per unit return cost instead of putting the extended amount on the RMA. This impacted the inventory cost, their pricing (was dependent on inventory cost), commissions, and so on for every product that had issues in RMA processing. The overall system-wide impact was negative, including commissions/paychecks for a lot of sales reps.