Escape Defects are issues that are identified by customers after the product has shipped, and as such have also been missed by the test teams. Defects identified after a product has shipped have the most significant cost impact to a business, both in development costs (find, fix & update) and impact to brand.
Escape Defect analysis looks to understand how these issues escaped in the first place, and put in place corrective steps to ensure they do not occur again.
The first step in Escape Analysis involves separating the escapes into useful and meaningful categories, such as:
- Process Step (where the escape should have been caught) – Development or Test
- Product Component (in which component of the code the defect occurred)
- Defect Impact (the impact experienced by the customer) – Crash, usability, performance, etc
- Version Introduced
- Release Discovered
Once issues have been categorised into their root causes, trends can be used to analyse if corrective changes to the process are reducing Escape Defects.
The following measurements can be used to analyse the effectiveness of process changes w.r.t. Escape Defect analysis
- Total Field Valid Unique Defects (TFVUD) per thousand lines of code (kloc)
- Total Field Valid Unique Defects (TFVUD) are all of the defects found by customers that were not cancelled, duplicates, user errors, or suggestions.
- The number of valid problems found by customers relative to the size of the code should decrease as new versions of the software are released
- The relationship of Post Development Defects (PDD) found in the test cycle and after development is complete, but not including those found after release to the field versus TFVUD
- PDD / (PDD + TFVUD)
- The purpose of this measurement is to drive the discovery of defects back earlier into the process, and to incur the least amount of cost
- This value should go up over time
The key purpose of Escape Defect analysis is to improve product quality and customer experience of the product. By applying continual process improvement to reduce the Escape Defect rate, the overall cost of developing and maintaining the software can also be reduced over time.