Contrary to what Larry and Byron think, requirements do have to be tested. At least, they must be tested if the project is to have a fighting chance of success. In Part V, we'll develop specific ways in which requirements can be tested.
The whole purpose of the requirements process is to reduce ambiguity in the development process, so the most basic test of any requirement is to measure its ambiguity. We've discussed requirements ambiguity, but the term "ambiguity" is itself ambiguous. There are many ways to reduce the ambiguity in "ambiguity," but the best would be to specify a precise way of measuring it. After all, there's little ambiguity in this requirement:
A. Draw a straight line on a blank sheet of 8.5" x 11" white paper, 13.150 ± .025 centimeters in length, .500 ± .025 centimeters in width, using a freshly sharpened Dixon Ticonderoga 1388 number 2 pencil. The line should be parallel to the top of the page, 2.000 ± .025 centimeters from the top, and touching the unbound edge.
On the other hand, we've already seen how much ambiguity there can be in a requirement that says:
B. Design a transportation device.
If "ambiguity" is a property of a requirement, then we'd like to be able to measure it in such a way that statement A has a small amount and statement B has a large amount.
The ambiguity measure we will develop is based on something we already observed in the "Design a transportation device" example. We gave statement B to a thousand individuals, instructing them to work independently on a solution. As we saw, this problem statement lacks many essential ingredients, and we could have anticipated that each participant would resolve each one uniquely. Even if all solvers were trained to use the same design process, we'd be shocked to find that even two had produced exactly the same design.
Now suppose we gave statement A to a thousand individuals, working independently on a solution. The problem statement lacks a few essential ingredients, and since people are different, we would expect some individuals would create idiosyncratic solutions. Generally speaking, though, we'd expect there would be only a few variations, and most solutions would be indistinguishable.
This mental experiment suggests we can measure ambiguity as the diversity of interpretation. We have actually performed this experiment with large groups. For problem A, we measured the diversity by comparing the lines drawn. Everyone drew a single line, in pencil, and there were only minor variations in line length and placement on the page.
For the transportation problem (B), we gave each person one minute to develop a conceptual solution. At the end of that time, we asked each to give a single number as an estimate of the manufacturing cost of the proposed transportation device. Obviously, if they all had designed the same solution, their manufacturing costs would have varied somewhat, but not by much. In fact, for roughly a thousand people, the cost estimates varied from $10 to $1.5 billion. This ratio of 1,500,000,000/1 indicates an extreme difference of opinion regarding the meaning of the problem, but isn't that precisely what we mean by ambiguity?
Such a ratio of largest to smallest estimate in a poll of informed individuals can be used as an ambiguity metric, a measurable entity for which we can obtain a precise value. Of course, a precise metric may not be extremely accurate, but it's far better than no measure at all. Indeed, it's a rather practical measure, one we have often used in real design situations.
A manufacturer of precision electrical components was studying the feasibility of developing an automated system for handling catalog information requests. The company's president had cost estimates ranging from "moderately inexpensive" to "moderately expensive," and didn't know whether to authorize the project. We asked each of 14 qualified individuals to write an independent estimate of the project cost. The estimated amounts ranged from $15,000 to $3 million. When the company president saw this 200/1 ratio, he halted the feasibility study to wait for a less equivocal requirement.
"Manufacturing cost" is only one possible basis for an ambiguity poll. Others that come quickly to mind are
1. total design and development cost
2. worker-years required for design and development
3. number of unique parts in the solution
4. minimum calendar time required to deliver the first product
These measures would apply to any project, but others would be more specific. For instance, in the transportation device problem, we might ask for estimates of
5. efficiency: average energy consumption per hour of use
6. range: how far it could transport
7. capacity: maximum weight it could carry at one time
One useful model of design says design is the process of removing ambiguity. In terms of this model, design proceeds through a series of steps: creating an approximate design, testing for ambiguity, removing the ambiguity found, and retesting the new approximation. Eventually, the tests say the latest approximation is close enough, and design stops.
In this view, everything done from recognition of a problem to final disposition can be considered design, including the unconscious, implicit design assumptions as well as the conscious, explicit design decisions. As an oversimplification, we can think of three major kinds of ambiguity we must measure and remove, each associated with a major part of the total design activity.
1. Problem-statement ambiguity, as we know, is ambiguity in the problem statement, or requirements (see Figure 19-1).
Figure 19-1. Problem-statement ambiguity is ambiguity in the process producing requirements for the design. All three of these designs will meet the requirement to "design a transportation device."
2. Design-process ambiguity is a measure of the variation in the process that will produce a design, in the sense of a picture of the solution (see Figure 19-2).
Figure 19-2. Design-process ambiguity is a measure of the variation in processes producing a design (that is, a picture of the solution). Three different processes to "design a car" could yield these three different cars.
3. Final-product ambiguity is a measure of the variation in the process that will create the physical solution, that is, the product itself (see Figure 19-3).
Figure 19-3. Final-product ambiguity is a measure of the variation in the process creating the physical solution. For example, no two cars coming off an assembly line are ever identical, even though the specifications may be identical.
As a result of measuring these three types of ambiguity with our poll, you will know the amount of ambiguity remaining, which indicates the amount of design work remaining. You will also know something about which stage needs work to reduce the ambiguity. In the beginning, of course, this measure is largely a measure of the work necessary to hone the requirements—that is, the requirements ambiguity.
In our example, if we ask for estimates of "manufacturing cost of one device," the variations we get may stem from different understandings of this phrase. Some might interpret "manufacturing cost of one device" to imply a unit cost of volume manufacturing. In this case, design, development, and tooling costs may be insignificant because they are amortized over many devices. But other people might infer the production of only a single device, in which case, design, development, and tooling would represent a substantial part of the product cost.
This particular kind of ambiguity has little to do with the actual product of design—the resulting motorcycle, bus, cruise ship, space shuttle, or skateboard. For instance, suppose two different estimators had a motorcycle in mind as the product emerging from the design process. If both interpreted "manufacturing cost of one device" as the unit cost in mass production, their actual production estimates might range from $200 to $4,000, an ambiguity of only 20/1. If both interpreted "one device" to mean the cost of development of a prototype, their estimates might range from $200,000 to $5 million, a 25/1 ambiguity. These ratios are far from the 10,000,000/1 that might arise from one person contemplating a space shuttle and another visualizing roller skates.
On the other hand, some of the ambiguity could stem from design-process ambiguity. Two people might both be thinking of cars, but one might be thinking of a car for a group of professional auto designers and the other for a soap-box derby competitor. How can we tell the difference? The simplest way is to ask the people polled:
What part of the problem statement led you to those inferences?
This question will locate the ambiguous parts, so we know what to clarify and also have a pretty good idea of how and when to do it.
We can derive additional information from the ambiguity poll by noticing not just the total spread of estimates, but also how they bunch into clusters. In our survey of the transportation problem, three ranges seemed to attract large numbers of estimates: $40-$60, $300-$700, and $8,000-$15,000. Such clustering may indicate three common sets of simplifying assumptions.
When interviewed, the estimators revealed reasons for the extremes and clusters of assumptions shown in Figure 19-4. The vast majority of estimators assumed a private transportation system and never considered public transport such as planes, cable cars, or elevators. Most also assumed the device would have a large market, so that design and development costs would have minimal impact on their estimated unit cost. Neither of these inferences were explicit in the problem statement, yet most respondents felt comfortable with their own judgment.
Figure 19-4. The clusters of estimates may reveal assumptions about the potential solution.
In this poll, the estimators were drawn from a population of professional systems designers with three to thirty years experience in the design and development of computer hardware and software. The great majority spent their professional lives living and working in the United States, which might account for the assumption of private transportation, as well as for the skateboard, bicycle, and automobile clusters. It might also account for none of them imagining a camel.
These observations suggest the most effective ambiguity poll is one which polls a great diversity of respondents. The more differences among respondents, the more the poll will turn up different resolutions of ambiguity. Therefore, when measuring ambiguity, your first effort should be to find as diverse an audience as possible, even stopping people in the street and asking them the measurement questions.
1. Our colleague Ken de Lavigne asks, "How about the expertise of the estimators?" With unskilled estimators, some of the variation will simply be due to errors in estimating, which is one reason to be sure to track down the source of the greatest and smallest estimates. Skilled estimators will avoid many mistakes, but in gaining the advantage of their skill, perhaps you lose variation—the more experienced they are, the more their experiences will tend to be the same. It's best to have some skilled estimators, but definitely not to have only skilled estimators.
2. On the other hand, a lack of difference in the poll, even when the individuals are quite different, does not necessarily mean lack of ambiguity. For instance, it's possible to receive three independent estimates of $10,000 for the manufacturing cost of a transportation device, even when three different devices are being estimated. A high-tech, one-of-a-kind, anti-spill, collision-avoiding, floating-bearing skateboard might cost as much as an automobile, and certainly an Olympic racing bicycle falls in the same price category as a medium-sized car. This convergence of estimates explains why it's always a good idea to use several questions in the poll.
3. One final word about polling as a method of exposing ambiguity: Polling will be effective only when the respondents' estimates are independent. If, for example, the company president's estimate is given first, and in public, then others are asked to present their estimates in public, you're likely to see a false convergence around the president's estimate. Any pressure, either overt or implied, will tend to reduce disparity in the estimates, and the less disparity, the less ambiguity revealed by the poll.
4. Another colleague Eric Minch comments, "The paragraph on independence of estimates raises a large issue: the existence of biases and 'errors' in sampling and measurement is of interest to designers, too. Even if we could get perfect information on individual preferences, we'd still want to know what sorts of social dynamics are operative in consensus. This means we have to somehow tease out the influences at both levels from a single set of measurements. The social sciences have evolved a set of techniques for the design of experiments, from 'discreet' to 'clandestine,' which may be useful here." Eric's comment reminded us to tell our readers we apply almost all our measurement techniques in situations where we can watch people performing tasks. And, as the great social scientist Yogi Berra said, "You can observe a whole lot just by watching." Our own preference is to start simple before moving on to more sophisticated techniques. If simple's not good enough, however, you might want to look up some social science literature. Start with anthropology.
Use an ambiguity poll to estimate the ambiguity in a requirement. The poll measures how much work there is to do, and perhaps directs attention to where the work should be done.
Whenever a piece of requirements work is said to be "finished," subject it to an ambiguity poll, to see if it really is finished.
Conduct the poll as follows:
1. Gather a group of people to answer questions about the document whose ambiguity is to be measured.
2. Be sure that there is no pressure to conform, or no influence of any sort of one participant on another.
3. Propose a set of questions, each of which can be answered with a number, such as: How fast? How big? How expensive? What capacity?
4. Estimate the ambiguity by comparing the highest and lowest answers.
5. Interview the high and low estimators to help locate the sources of the ambiguity.
The group used for estimating ambiguity should be as diverse as possible, at the very least including a sample from each population to be affected by the eventual product.