The quality culture of the QA and engineering teams can be the key driver to overall quality and QA productivity for a small company. The quality of the code leaving engineering at the start of testing and the quality of the defect repairs drive the culture of the teams and the company. Unfortunately, many engineers do not give either of these issues enough thought.
A product's defect count as it enters the testing phase defines the quality of the shipped product and the schedule. The quality of the product is proportional to the quality of the product entering testing, because testing finds percentages of problems, not a fixed number of defects regardless of how many exist. Defect counts at the start of testing affect the schedule by adding to the overall cycle time for the testing team. Defects require identification and repair time, and large numbers of defects require more test passes to ensure quality code.
You can observe the effect of defect count at the start of testing on the total test schedule. Table 17-5 shows an example that compares starting defect counts of 300, 40, and 10 entering testing against the number of test passes. In all cases, assume that engineers will repair 85 percent of the defects properly, but they will repair 15 percent improperly, and these defects will require more work. Notice how starting with 300 defects requires four test passes to achieve zero defects. Starting with 40 defects needs three test passes. Starting with 10 defects requires two test passes. This illustrates how the initial defect count can have a big impact on schedule.
Table 17-5. Defect Reduction vs. Testing Passes with 15 Percent Error Rate
Testing stage | 300 start # | 40 start # | 10 start # |
|---|---|---|---|
Pass 1 end count | 45 | 6 | 2 |
Pass 2 end count | 7 | 1 | 0 |
Pass 3 end count | 1 | 0 | 0 |
Pass 4 end count | 0 | 0 | 0 |
Also important is the quality of the repairs. Repair quality can be measured in the test pass reduction rate—the percent of identified defects that are not properly fixed and new failures that are caused directly by the repair. It is fair to characterize an organization by its correct repair rate or, conversely, by the repair failure rate. For example, a 25 percent failure rate indicates a sloppy organization in which engineers do not check their work or communicate with QA. A reasonably well-run organization should have a failure rate of 10 percent or lower. A 5 percent (1 failure in 20) organization is performing very well indeed: Development engineers are working closely with QA engineers before implementing a repair to make sure they understand the problem. Engineers are also checking their work to avoid introducing new problems.
Let's take a look at another example. Assume engineering builds 100 defects into a project that QA finds in the first test pass. Each pass takes a week because a full regression test is undertaken on each pass to catch side effects of defect repairs. Consider the simple scenario shown in Table 17-6 with three different repair rates, 25 percent, 10 percent, and 5 percent, starting at 100 defects.
The outcomes of the three approaches are considerably different. The 25 percent team takes four test passes before the product is ready to ship. The 10 percent team takes three passes. The 5 percent team takes two passes through the code for success. If each pass takes 5 days, then the 5 percent team takes 10 days less than the 25 percent team for the same initial defect count. In practice, the time difference is even greater, because this example does not include engineering time to make the repairs. If you assume 20 percent of a day spent per defect found and 5 days per test pass, you get the data illustrated in Table 17-7.
Table 17-7. Total Time vs. Different Repair Failure Rate
25% repair failure rate | 10% repair failure rate | 5% repair failure rate | ||||
|---|---|---|---|---|---|---|
Stages | Defect Count | Time (days) | Defect Count | Time (days) | Defect Count | Time (days) |
Start count | 100 | 100 | 100 | |||
Pass 1 | 25 | 5.0 | 10 | 5.0 | 5 | 5.0 |
Pass 1 repairs | 5.0 | 2.0 | 1.0 | |||
Pass 2 | 6 | 5.0 | 1 | 5.0 | 0 | 5.0 |
Pass 2 repairs | 1.3 | 0.2 | ||||
Pass 3 | 2 | 5.0 | 0 | 5.0 | ||
Pass 3 repairs | 0.3 | |||||
Pass 4 | 0 | 5.0 | ||||
Total days | 26.6 | 17.2 | 11.0 | |||
Wow! The team with a 5 percent repair failure rate closes a release 15 days faster than the team with a 25 percent rate. The time savings are much larger as the size of the project increases. You can clearly see that driving a positive engineering attitude toward defects entering the testing phase does cut schedule time and reduces post-release defects.