William E. Perry and I have published our second book together, called "Testing Dirty Systems." We define a dirty system as one that may lack any of the following:

1. Complete documentation
2. Full understanding of requirements
3. Design/Architecture is not done in a standardized format and/or
4. The defect density of the software is unknown

The following information is found in our new book.

The key to testing a dirty system is knowing how to be a "testing archeologist." You will be digging through system artifacts and trying to piece together a view of the system that allows you to build a test plan. Your goal is to plan and perform a test that is more than just guesswork.

Fortunately, there are tools and techniques that help in the process of peeling back the layers of the unknown system structure and function. After diagnosing a dirty system's functionality and structure, then developing a test plan to validate the system, it's time to test the system. It is during test execution of a dirty system that unexpected functions are discovered and strange things happen that require further investigation.

All of the work performed in the diagnostic and test planning steps will prove to be very valuable in building a basis for test execution. Although surprises may arise, the prior research and planning will help greatly in predicting and isolating defects.

Much like an exterminator knows where to find certain kinds of pests due to the knowledge of where they thrive, you can also become an expert software bug exterminator by identifying common breeding ground for categories of software bugs.

The 20 Most Common Software Problems

After over 30 years of combined software defect analysis performed by ourselves and colleagues, we have identified 20 common software problems. These common software problems appear in a wide variety of applications and environments, but are especially prone to be seen in dirty systems.

1. Incorrect calculations - This is seen in functions such as financial and date calculations. The key determinant is whenever mathematical functions and mathematical operators are involved.

2. Incorrect data edits - This is when the software does not apply existing data edits correctly. For example, a data edit may be coded to prohibit the entry of the day of the month greater than "31", but does not allow for the month. This would allow the entry of February 30 and other invalid dates.

3. Ineffective data edits - This is when data edits are in place and working correctly, yet still fail to prevent incorrect data from being entered into the system. An example of this is an alphanumeric address field that allows spaces to be entered before any numbers or letters in the address. Therefore, when searches or sorts are performed on the address field, the search or sort may not find the intended address.

4. Incorrect coding/implementation of business rules - This refers to the one of the most common sources of software problems - the mistakes that occur between what is intended to be developed or implemented and what is actually delivered. These defects can be traced back to incorrect, missing, or vague system requirements specifications, or to the misinterpretation of requirements specifications. If you are asking, "What specifications? What requirements?", the incorrect coding or implementation of business rules is probably a common problem for you.

5. Inadequate software performance - This refers to slow system response times and transaction throughput rates.

6. Confusing or misleading data - This means that the data shown to users may be correct, but the users might not fully understand how to interpret the data. This is not a trivial problem. Lives have been lost because of someone's failure to take the correct actions based on the data delivered to them from a computer system.

7. Software that is difficult to use - Many people have experienced first-hand the frustration of using software that is cumbersome, difficult to navigate, and requires several steps to perform simple tasks. This problem relates to a lack of understanding of how humans interact with computers and is also the result of a history of modifications that are not planned and coordinated to account for ease of use. For example, the addition of numerous workarounds over a period of time in legacy systems can have the overall effect of convoluting the original system design.

8. Obsolete software - Software that no longer works due to new hardware or support software changes - This refers to software that is based on functions found in older versions of databases and operating systems. An example of this can be found in old COBOL code that will not compile on new compilers due to the use of verbs that are no longer supported in the compiler. Many vendors try to make new releases of support software upwardly compatible, but there are usually cases where one minor area of non-support from the base system can cause a major revision of the system. The only other option is not to upgrade the support software. This decision can be justified for the short-term, but a point is usually reached where the software must either be replaced or modified.

9. Inconsistent processing - Software that only works correctly in one environment – This refers to software that has been designed for only one environment and cannot be easily transported and used in another environment. Of course, some software is designed to work in only one environment. However, if an organization adopts new technology that requires software be portable to new environments, then the software will need to be modified or replaced if it can't meet the new technical requirements. An example of this is software that works in an MS-DOS environment, but will not work in a Microsoft Windows environment.

10. Difficult to maintain and understand - This refers to the ability of a programmer or developer to maintain the software. To maintain software, the person performing the maintenance must first analyze and understand the software. Much of the software in existence today was initially written in an unstructured manner and then patched on an as-needed basic over a long period of time. This type of software structure results in what is known as "spaghetti code," which is complex and unstructured. To add to the problem, when changes are made to this kind of software, there is a higher risk of creating new defects unintentionally.

11. Unreliable results or performance - This means that the software does not deliver consistently correct results or cannot be depended to work correctly each time it is used.

12. Inadequate support of business needs or objectives - This refers to software that is inflexible to meeting business needs. For example, a system may be difficult to modify to meet and organization's needs or may lack features to allow the users to customize business rules.

13. No longer supported by the vendor - This occurs when a vendor ceases to support a particular software product. This can occur due to the vendor's decision to no longer support a product, due to the vendor going out of business, or the vendor selling the product to another vendor.

14. Incorrect or inadequate interfaces with other systems - This means that the software does not correctly accept input (data, control, parameters, etc.) from other systems or sends incorrect output (data, control, parameters, print, etc.) to other systems. An example of this is when a system has an electronic data interfaces (EDI) with external systems, but does not correctly receive or format the information.

15. Incorrect matching and merging of data - This refers to situations where data is obtained from one source and matched or merged with data from another source. Examples include sorting multiple files into a single file or table or matching data from a master file to an ID number entered as a lookup entry.

16. Data searches that yield incorrect results - This means that a search retrieves incorrect data as the result of a search. In the worst case situation, the data retrieved appears to be correct in format, but only by tracing back to source documents and other original data can it be determined that the data is incorrect for the search criteria. An example of this would be searching for the time worked by a particular employee in a payroll system. The employee's name at the top of the information may be displayed correctly, but the detailed time data may belong to another employee. The only ways to verify the information would be to compare the time worked back to time sheets or to tables that indicate the employee ID.

17. Incorrect processing of data relationships - This means that data relationships are not created or maintained correctly between one or more data elements. These data elements can reside on interactive interfaces, reports, or files. For example, a system may allow a user to incorrectly enter a telephone area code invalid for the state specified in an address field.

18. Incorrect file and data handling - This refers to the software incorrectly retrieving data from files or tables. This could include retrieving the wrong data from the right source or the right type of data from the wrong data source. An example of this would be retrieving data from an old version of a file or table, thinking the data is being retrieved from the most current version. Another example is the inability of the software to process empty or full files correctly.

A secondary problem could relate to the software's inability to pass data correctly through the system. An example of this would be the incorrect processing of transactions, where data is inadvertently dropped during processing.

19. Inadequate security controls - This means that unauthorized access to the system is not adequately controlled and detected. In addition, people may also be able to perform transactions in excess of the authorization levels appropriate for their job functions. For example, a person without managerial levels of security access might be able to approve their own overtime. Or, a person not in the payroll department might be able to view the employee payroll files.

20. Inability to handle production data capacities - This refers to the software's inability to process data at the level required by the organization. An example of this would be a system that is required to process financial transactions that exceed $10 million, but the system can only process amounts up to $9,999,999.99. Another example is the classic case of the Year 2000 computing problem, where dates in the Year 2000 and beyond are incorrectly recognized as being in the early 1900's.

If you developed test cases to address each of these problems, you would have a huge challenge to cover all of them completely. Like everything else in testing, looking at relative risk and the situation at hand will narrow your focus. One approach would be to design a risk questionnaire for each problem area. Another method would be to interview users to assess the risk impact. A third technique would be to study past defect reports and correlate them to each of the problem areas, giving the highest priority to the most troublesome areas.

As I indicated at the top of this article, I do not presume to have presented the ultimate list of problem sources for dirty systems. If you have others, I would appreciate hearing your feedback.