Test scripting has been a popular way to build testware since the advent of interactive software. With Graphical User Interfaces and web-based interfaces, the use of test scripts should be carefully considered. In some cases, test scripts may apply very well to object and component-based interfaces, while in other cases they may not apply well at all. This article examines the nature of test cases and how to determine the best vehicle for designing a test in the web-based environment.
Understanding the Components of a Web Test Case
Test terminology has reached the point where people use a variety of terms in non-standard ways. One of the most inconsistently used test terms is that of a "test case." There are several definitions I could use for test cases, but the one I like the best is used in IEEE Standard 829-1998. In this definition, a test case is comprised of a test condition, an expected result, and a procedure for performing the test case.
I like this definition because it implies that test cases can be performed either in combination with other test cases or in isolation. This is an important distinction when testing web interfaces, which allow a user to perform actions in a random and unpredictable manner.
Modular test cases are like objects which can be performed in isolation but often have the requirement of interfacing with other test cases to achieve a particular objective. The important concept to remember is that the sequence of performing test cases may be unpredictable from the test designer's perspective, but can be critical in manifesting a defect.
How to Determine When to Use Test Cases vs. Test Scripts
When deciding to use test cases vs. test scripts, its much like deciding to take a guided tour of a museum or browse around on your own. Much depends on how much time you have, your prior knowledge of what you're viewing and the freedom you would like to have during the experience.
Likewise, in the decision of using test scripts vs. test cases, much depends on the context of the application you’re testing. This could include the tester’s level of knowledge of the application, the amount of freedom the user is intended to have in using the web application and the amount of time available for testing.
The decision to use test cases versus test scripts depends on:
- The level of predictability of how a user will interact with a web interface
- The importance of sequence in the user’s correct performance of a task
- The degree of freedom a user is intended to have in interacting with a web interface
- The importance of documenting a test of a specified sequence
- The intensity of the test
The main benefit of a test script is that it predefines a procedure to follow in performing a test. This can also be its greatest curse. Sometimes you want the randomness of user actions. At the same time, you want to know in advance the conditions to be tested and how they should behave. This is the classic tradeoff between test scripts and test cases.
Just because you choose to use a test script in testing one area of your application does not imply you must use test scripts for testing everything in an application. You may choose to only create test scripts for the definable processes that are critical and that apply well to scripting. For everything else you could decide to use test cases.
The Sources of Web Test Cases
Many web testers describe the process of testing web applications as trying to hit a moving target. Just when you think you have things in place, they change. In fluid development environments like the web, software development happens quickly and functionality can change just a quickly. Sometimes the testers know what the web application requirements are, but many times the requirements tend to take shape over time as developers and the customer/users interact.
Before we can define a test case, we need to know how the web application is intended to behave. There are many sources of test cases in a typical web application and I will examine the ones that give you the best results with the least amount of effort in test case design and execution.
Web Test Case Components
The structure of test cases is one of the things that stays remarkably the same regardless of the technology being tested. The conditions to be tested may differ greatly from one technology to the next, but you still need to know three basic things about what you plan to test:
ID #: This is a unique identifier for the test case. The identifier does not imply a sequential order of test execution in most cases. The test case ID can also be intelligent. For example, the test case ID of ORD001 could indicate a test case for the ordering process on the first web page.
Condition: This is an event that should produce an observable result. For example, in an e-commerce application, if the user selects an overnight shipping option, the correct charge should be added to the total of the transaction. A test designer would want to test all shipping options, with each option giving a different amount added to the transaction total.
Expected Result: This is the observable result from invoking a test condition. If you can’t observe a result, you can’t determine if a test passes or fails. In the previous example of an e-commerce shipping option, the expected results would be specifically defined according to the type of shipping the user selects.
Procedure: This is the process a tester needs to perform to invoke the condition and observe the results. A test case procedure should be limited to the steps needed to perform a single test case.
Pass/Fail: This is where the tester indicates the outcome of the test case. For the purpose of space, I typically use the same column to indicate both "pass" (P) and "fail" (F). In some situations, such as the regulated environment, simply indicating pass or fail is not enough information about the outcome of a test case to provide adequate documentation. For this reason, some people choose to also add a column for "Observed Results."
Defect Number Cross-reference: If you identify a defect in the execution of a test case, this component of the test case gives you a way to link the test case to a specific defect report.
Sample Business Rule
A customer may select one of the following options for shipping when ordering products. The shipping cost will be based on product price before sales tax and the method of shipment according to the table below.
If no shipping method is selected, the customer receives an error message, "Please select a shipping option." The ordering process cannot continue until the shipping option has been selected and confirmed.
| Cost | Less than $20 | $20 – $29.99 | $30 – 39.99 | $40 and greater |
| Shipping Method | ||||
| Overnight | $12.00 | $16.00 | $22.00 | $27.00 |
| 2-day air | $8.00 | $8.00 | $4.00 | $0 |
| USPS | $6.00 | $6.00 | $6.00 | $0 |
The table below shows how the shipping options of an e-commerce application could be independently tested.
| ID # | Test Condition | Expected Result | Procedure | Pass/ Fail |
Defect ID # |
| 1. | Customer orders products totaling less than $20 and selects overnight shipping. | Shipping charge of $12 is applied. | 1. Customer accesses secure ordering page.
2. Customer adds items to shopping cart totaling less than $20. 3. Customer selects "overnight shipping" as delivery option. 4. Customer confirms items in shopping cart, shipping options, and transaction total. |
||
| 2. | Customer orders products totaling less than $30 and greater than $20 and selects overnight shipping. | Shipping charge of $16 is applied. | 1. Customer accesses secure ordering page.
2. Customer adds items to shopping cart totaling $29.99. 3. Customer selects "overnight shipping" as delivery option. 4. Customer confirms items in shopping cart, shipping options, and transaction total. |
Table 1 - Test Case Examples
Things to Notice in This Example
1. Specific test data is not embedded in the case.
The test designer did not specify "1 copy of Gone With The Wind (paperback), 1 CD of Hootie and the Blowfish (sale priced at $3.99) and one poster of Aniken Skywalker. Why? Because in these test cases the items ordered only affect the observed result as a total, not individually.
There’s another reason to keep specific data separate from the test case. That is to reduce the amount of maintenance to the test cases should either the test procedure or test data change.
Consider this situation. We design 1,000 test cases with the only difference being the test data. Then, the web developer calls and informs you that an additional step needs to be added to the process. Normally it would not be a big deal to make a procedural change. It becomes a big deal, however, when the change is multiplied times 1,000! In keeping the data and the test case separate, we would only need to change one test case and probably not change the test data at all.
At this point you might be wondering where the test data reside and how to know the expected results. There are a variety of ways to create and maintain test data, but it could be as simple as creating a PC-based database or spreadsheet. The test data can then be obtained manually through a cross-reference, can be imported into an automated test tool, or can be entered into the application using a home-grown scripting procedure.
2. There is no sequence implied or necessary for the test cases.
The important thing is that all of the conditions are tested, not necessarily the sequence. Consider the scenario where a customer selects a certain form of shipping and then decides to select a less expensive option. While it is possible to script such a scenario, the possible combinations of such scenarios can be overwhelming and not very profitable when it comes to a return on your testing investment.
3. The order of the columns is up to you.
I like the format shown in the test case examples, but you might like to see the test procedure before the expected result. The important thing is to define a format and stay consistent. Consistency in items such as test cases and test scripts helps to avoid confusion during both test design and test execution.
Test Case Design Strategies for Web Applications
In the search for the best way to test your web application, you will likely have several possible approaches. It's difficult sometimes to decide which test case development approach will eventually work best, but here are some common test case strategies. The intent of this discussion is to focus on how to develop testware using a variety of approaches, many of which have been effective in other technologies and are still useful in testing web-based applications.
To arrive at the approach that works best for you, you will need to consider:
- The type of application to be tested.
For example, if the application is very procedural, test scripts may be a primary means for designing your tests.
- The nature of the intended users
If you are testing the usability of an e-commerce application, you will need to consider that the eventual customers will not have the benefit of training and detailed written procedures. This would be an example of when to use test cases that give the tester a great deal of flexibility in testing.
- The need to repeat the test
Although there is a degree of error inherent with manual testing, well-defined test cases and test scripts give you at least some trail to follow if you ever need to repeat the test. A test script is probably the best manual vehicle to document a trail of actions, but you can also document a process with test cases ordered sequentially by using a test matrix.
- The level of relative risk
The higher the degree of risk, the more rigorous the test should be. In high-risk applications, the test needs to be documented at a high level of detail. This implies the use of well-defined test cases and test scripts.
- The degree of change
If the web application is changing rapidly and continuously, your investment in documented test cases and test scripts may prove to be overwhelming. While automated test tools can provide some relief, there are still issues in actually getting the tool integrated into your web development processes. Even with test automation, the job of test case development and defining the process to be tested is still largely a manual effort for many people.
Each of the following approaches to design web-based test cases has its strengths and weaknesses.
Test Scripts
Test scripts became popular when mainframe applications went interactive. These procedural "green screen" types of interfaces are typically very predicable in nature and the user is not given much latitude in how they can interact with the application. These procedural user interfaces gave rise to written, or manual, test scripts that told the user what to enter in each screen field, then to go to the next field, etc.
Then came the Graphical User Interface (GUI). With the advent of the GUI came the promise, and sometimes the ability, of the user to work according to their own preferences and not necessarily to a rigid procedure. The impact on test planning is that it may impossible to predict all the ways a user may perform a process while using a GUI.
Web pages tend to be visually-oriented and strive to deliver the same type of user flexibility as a client/server GUI. In addition to the web browser GUI, a typical web page may contain:
- Hyperlinks
- COM objects
- Active-X controls
- JAVA applets
- And many other functions that are accessed by "point and click" user actions.
You might be asking, "But what about GUI automated test tools? Don't they use recorded test scripts as the primary means for testing?"
Yes, they do. Each script follows one path through a process, or on a smaller level, each script describes a sub-process in a modular manner. Modularity does a lot to help reduce the large numbers of test scripts required for some GUIs, but you can still only describe so many combinations of possible actions before you run out of time, people or money.
The issue is whether or not you need to predefine a sequential order of actions for testing a particular process. Since a test script describes sequence, actions and expected results, the level of interface "randomness" affects the effectiveness of test scripting. Before we decide to eliminate test scripts as a tool for testing web applications, there are some important considerations:
- Is the process to be performed in a web application predictable or random?
Some processes delivered via a web-based application are very predictable. The user points and clicks on objects, but there are clearly defined steps that must be completed before the process is considered complete.
In the case of a predictable process delivered via a web application, test scripts can be used effectively. An example of this is ordering a product from a web site. Although the customer clicks on the "next" and "back" buttons, they must still enter required information and follow a defined path to ensure a correct order. Sure, exceptional situations will occur that also need to be anticipated and tested. These might include a lost connection, a timeout, or the customer aborting the order.
- Can the users of the web application be profiled?
If you can categorize users by how they approach and use a web application, then you can predict usage patterns. One approach to profiling user patters is to observe a diverse group of typical users or customers and look for common paths taken by them.
I have found that in some cases you can define the most commonly taken paths through a web application with less than a dozen test scripts.
- Can the web application design be simplified?
One of the reasons software, web-based or otherwise, is difficult to test is that it is often too complex to test completely. Sometimes a few basic design simplification decisions help reduce the complexity of testing.
Test scripting can involve a variety of forms and methods. I've seen about as many test script standards as I have seen testers. To illustrate, here is a sample manual test script format. This test script format is designed to handle tests that are isolated to one page in the web application, or that span multiple pages of the application as well as other system processes over a given period of time.
Moving Into Automation
The reason I have laid the foundation in manual test cases and test scripts is because much test analysis requires getting the manual process documented first. The main point in automating tests is the ability to repeat them later. The best thing you can do to facilitate repeatable tests is to plan them to be repeatable. This implies thinking about modularity and keeping scripts free of embedded data except where absolutely essential. In other words, let's take time at the beginning and define the problem before we immediately start solving it.
Once you have the manual concepts mastered and once you have some idea of the processes and cases you want to automate, then you are ready to start thinking about automating the test. There are also other things to consider, such as naming conventions, test script management, etc.
Actually, this is where we arrive at where we started this discussion: using test cases to design a modular test. Once we can define the triggering events (test conditions) and have a way to observe the outcome (expected results), then we can design and perform a test, whether it be manual or automated.
An exciting advancement in test automation is that some of the tools are able to generate test cases automatically from the web interface. That's a step in the right direction, but there is still a need to define tests that span interfaces, such as in the case of ordering a product. It's the integration of functions that give test designers their greatest challenge.
Related Articles
Identifying the Challenges You Face in Testing Web Applications
