I get a lot of questions each month about user acceptance testing (UAT) - what it is, how to perform it, which (if any) tools to use, and a variety of other questions. The purpose of this article is to build on a base of past articles and continue to build a knowledge base and lessons learned. Who knows? One day all of these articles may turn into a book!
The keys presented in this article are not hidden from view, just often overlooked by people. If you ignore them, you will experience problems and less than effective user acceptance testing. If you heed them, you will open new doors to involving your users in testing.
User Acceptance Testing (UAT) Defined - User acceptance testing is the validation that a system or application will meet user needs in the operational or business environment.
This article can be applied from multiple perspectives:
Key #1 - Understand that UAT is Performed at the Worst Possible Time in the Project
Most UAT efforts happen at the end of the project because this is when the entire system is assembled or installed. Until the end of the project, users may be able to test parts of the system or application, but not the system as a whole. This is bad because the end of the project is the worst time to find and fix major problems. Each problem found and fixed in system testing or UAT which has been in the system since requirements has a 10x cost factor had it been found or fixed in requirements or design. This is due to the ripple effect that the fix may require in other areas of the system.
The solution is to involve users throughout the project from the very beginning. When users are providing input to user requirements, they can also be defining acceptance criteria and can be involved in requirement reviews and inspections.
Key #2 - Base the Test on Real-world Conditions, Not User Requirements
This is one of the most controversial things I teach about UAT, but if you don't base the test on real-world conditions you are missing the point of UAT. There are two sides to testing - verification and validation. Verification is testing based on specifications and requirements. Validation is testing based on real-world operational conditions. You need to test from both perspectives. Validation is necessary because requirements have defects. In many cases, the requirements are not available, such as in the case of vendor-developed software.
Key #3 - Understand Your Users
UAT is not a "one-size fits all" activity. Some user groups will not have the motivation, time or skills to design and perform an adequate test. Some user groups will be able to perform a very extensive test.
One solution is to profile the affected user groups. I often categorize users as:
Key #4 - Involve Your Users
Some organizations perform UAT with surrogate users - people who take the role of users but who are not the actual people in the field that will eventually use the application. The risk with this is that when the system is deployed, the real users will find problems the surrogates didn't consider. I only recommend this approach when the actual users are unavailable or unwilling to participate in the test. Even then, the risks are high.
The solution I advise is to at least hold limited review sessions with the actual users. Complete review sessions which examine items such as user requirements in detail are even better. Also, have contingency plans in place when unexpected problems are found during UAT. If users are unwilling or unable to participate in the project, raise this situation as a risk in the project status reports.
Key #5 - Match the Intensity of the Test to the Relative Risk and the Skills of the Users
Not every project requires extensive testing. However, for those projects that control high levels of assets or affect personal safely, extensive validation is required. Users and others on the project may question the need for defined test cases and test scripts, but when viewed in the light of project and business (or operational) risks, the time and resources spent in effective testing are resources well spent.
To match testing to the risk, perform a risk assessment that can be quantified and documented. Just guessing at the level of risk is not good enough to explain after a critical failure why you thought something was a low risk of failure. The risk assessment should indicate which system and business areas are the most exposed to risk. This allows test resources to be allocated where they will have the greatest impact to detect defects that may have severe negative consequences.
Key #6 - Plan the Test in Advance
There are three levels of test planning:
Some people prefer to use informal and random approaches to testing, which can find obvious defects but will often miss the more deeply embedded defects that are often the most troublesome. Another problem with a lack of test planning is that you never quite know for sure when you have completed the test. A test plan contains measurable objectives and pass/fail criteria. Detailed test plans also make it possible for one person to design the test and for many people to perform it. Finally, detailed test plans give you a way to recreate a test if you have to perform it again.
Once again, the extent of your test planning effort should be reasonable in light of the relative risks.
Key #7 - Review Your Test Plans
Test plans can have errors just like any other type of project documentation. UAT plans can be reviewed by the UAT team, a Quality Assurance (QA) team or facilitator, the project manager, or any other people with knowledge of testing and the project.
User acceptance testing is not trivial or easy. UAT can be one of the most critical and risky types of test on a project, which means that a great deal of care should be taken when planning, executing and evaluating the results of UAT. These keys of UAT have worked for other organizations in planning and performing UAT and they can work for yours as well.