Experience has taught users to avoid the latest versions of software applications until the inevitable maintenance releases, and patches have been released.

Even Apple is not immune to buggy launches — its initial iOS 8 release was riddled with major bugs, and customers were vocal about their dissatisfaction. Those issues still exist today with their iOS 11 releases.

While everyone is aware of the software quality gap that exists between initial release V.a, and stable release V.y, unfortunately not much progress is being made toward solving the problem.

This post discusses five actionable ideas to help development groups close the quality gap.

1. Use code coverage analysis

Code coverage analysis reports on the portions of the application source code that have been executed by a set of test cases. Analysing code coverage is the best way to measure the completeness of your test activities. Without measuring code coverage you are “testing in the dark.”

In a study done by HP, it was found that testing without measuring code coverage exercises only 55% of code (Robert O’Grady, HP).

It is important to remember that while achieving “100% code-coverage” does not prove that an application is perfect, it is a critical component of engineering high quality software. In fact, all of the standards associated with safety critical software development mandate code coverage as part of the development process. Aerospace uses DO-178B/C, automotive has ISO 26262, IEC 61508 for industrial controls, FDA and IEC 62304 standards for medical devices and the CENELEC standard applies for rail applications.

2. Improve test coverage with unit tests

Once you start measuring coverage, you’ll likely find that existing tests provide significantly less than 100% coverage, this coverage gap results from testers focusing on nominal use cases and not on error cases or boundary conditions.

The obvious way to close the coverage gap is to add additional functional tests, but it is likely that 20 -30% of the application code is really difficult to test with functional tests in a production environment, because it is difficult to inject the faults required to trigger the error handling.

Figure 1 – Augment coverage gaps from system testing with unit tests to achieve 100% code coverage

It’s no surprise that most critical bugs that occur in the field are the result of an odd combination of stimulus to the application that was never anticipated. Enter the fabled Heisenbug, a bug that disappears or alters its behaviour when one attempts to probe or isolate it. For C programmers, these are thought to be the result of uninitialised auto variables, and are a source of frustration because simply observing the code appears to be altering it. (Hristov, Ivan. September 16, 2012. Chasing Heisenbugs from an AKKA actor integration test with awaitility.)

This is where using low-level unit testing is critical. Unit tests allow fault injection i.e. the testing of error handling in ways that are impossible in a production environment.

3. Make tests easy to run, and results easy to understand

Figure 2 – Different testing types

In theory, it sounds like a simple plan: make your tests easy to run and the test results easy to understand. In practice, however, this can be a challenge. Historically, different flavors of tests are built and maintained by different engineers, often using different tools:

• Developer tests are used to prove correctness of the low-level building blocks of an application
• Integration tests are built to prove the correct functioning of complete sub-systems
• System tests are built to prove correctness from an end-user point of view

When tests are partitioned this way, each flavor of tests is owned and maintained by a different group of engineers rather than being shared across all members of the development team. In fact, in most organisations, it is probably impossible for a QA engineer to run a developer test or a developer to run a system test.

In order to improve quality these barriers must be broken down. It must be possible for any member of the development team to run any test at any time on any version of the application.

The key to enabling this work-flow is a common test collaboration platform, which captures all tests, along with their pre-conditions and expected results. Engineers should be able to run a single test, or all tests with the “click of a button”. In addition, it is essential that engineers are able to quickly debug failing tests.

4. Implement automated, parallel, and change based testing

Figure 3 – Only regression test the test cases impacted by the code change

Once testing completeness is improved by code coverage analysis, and tests are deployed across the entire organisation, the next step is to ensure that tests run quickly. One of the reasons tests are partitioned between multiple groups is that a complete system test might take hours or days to run. Obviously, if you ask a developer who has changed one line of code to run 10 hours of testing, you’ll get some push-back. So how can we decrease test time, while still ensuring testing completeness?

The key is to build a testing infrastructure which is scalable, using parallel and change-based testing. Individual tests must be atomic, small, and fast. Two often test suites become tightly coupled over time with new tests simply being inserted into existing tests. This makes tests fragile and test maintenance time consuming. A simple thought to keep in mind when designing tests is, each test should define its own pre-conditions not rely on the output of other tests.

Beyond the benefits of test maintenance, re-architecting your tests to be atomic enables:

• Change-based testing, running only those tests affected by each software change
• Parallel test execution, running hundreds of individual tests simultaneously

While every organisation has developed a software build system that allows for unattended incremental application building, most have not implemented incremental testing. Too often, testing is performed periodically rather than constantly and incrementally with complete automation. Change-based testing (CBT) analyses each set of changes to the code base, and intelligently selects the sub-set of all tests affected by those changes. This results in complete testing in a fraction of the time of a full test run. In addition, change-based testing provides an accessible means for implementing a rigorous continuous integration (CI) development process; during the check-in phase of CI, CBT provides an efficient means to verify the build and detect problems early.

To improve speed even further, consider parallel testing. By integrating your test platform with a continuous integration server, and virtualised test machines, you can reduce total test times from hours to minutes, or minutes to seconds.

5. Re-factor code bases to improve maintainability

Code refactoring is the process of restructuring application components without changing its external behaviour (API). Without re-factoring, application code becomes overly complicated, and hard to maintain over time. As new features and bug fixes are bolted onto existing functionality, the original elegant design is often a casualty.

Code re-factoring improves code readability and reduces complexity, hence maintenance cost. Code refactoring, executed well, offers the additional promise of resolving hidden, dormant, or undiscovered computer bugs or vulnerabilities in the system by simplifying the underlying logic and eliminating unnecessary levels of complexity.

One of the greatest impediments to re-factoring is the lack of tests which formalise the existing behaviour. In a prior post on Technical Debt, we discuss an approach for addressing this.

Figure 4 – Code refactoring approach

Every application has fragile and buggy sections which developers are hesitant to change for fear of breaking existing functionality. The only way to confidently refactor these fragile modules is to ensure that you are building tests to formalise the expected behaviour.


Over the last thirty years, there have been a steady flow of tools, design patterns, and development paradigm shifts. Many of these have promised improved quality without increased time or effort. It should be clear to everyone in the software industry by now, that there is not, and will never be, a silver bullet that provides improved quality at no “cost”. The only sensible way to improve software quality is to improve the effectiveness of software testing.