Requirements information can be wrong in two principal ways: inadequate or inaccurate. Although most of our emphasis in this book is on adequacy—generating a sufficient quantity of significant ideas—that's only to balance our other work on accuracy—ensuring each retained idea is of sufficient quality.
In this chapter, we'll give an overview of technical review meetings, the principal tools customers can use throughout the requirements process to test whether requirements contain all and only reliable information. [This chapter is but a short summary of what you need to know about technical reviews in order to run a successful requirements process. For a full guide, consult Daniel P. Freedman and Gerald M. Weinberg, Handbook of Walkthroughs, Inspections, and Technical Reviews, 3rd ed. (see Bibliography)]
All of what we have said about meetings in general in the first volume applies equally to technical review meetings, yet these meetings have a discipline all their own. The reason for this discipline is clear if you have ever attended a "review" like the one following.
(Jack, Zara, Martha, Sid, Ned, and Retha arrive at Room 1470B at 8:50 a.m. There is a carafe of coffee, a pot of hot water, and a huge pile of blueberry muffins. Everyone is in a good mood, and they attack the muffins. At 9:00, Jack steps to the front of the room, picks up one of several marker pens, and writes the agenda on the board:
Walkthrough agenda:
What's wrong with zara's do not disturb sign requirements?)
Jack: Okay, we're walking through what Zara did wrong on the sign requirements, so let's stay focused.
Zara: Don't I get to hear what I did right?
Jack: Sure, maybe if we have time at the end of the review. I mean, we all know it's a terrific piece of work, but you made a lot of mistakes.
Zara: Thanks.
Martha: Well, personally, I wouldn't have made a sign at all, because signs get lost. I think the requirement should have had the sign built into the door.
Sid: Heavy.
Jack: What are you talking about?
Sid: Lost.
Martha: I think he means that a heavy sign won't get lost, or at least people won't carry it away with them by accident.
Retha: Handicapped people couldn't use it. Zara's requirement doesn't say anything about handicapped people, and that's essential.
Ned: Not really. Handicapped people are going to be noticed by the hotel staff, and they're not going to disturb them. Besides, even if the Do Not Disturb sign was out, and you hadn't seen this epileptic for two days, wouldn't you want to go into the room anyway and check it out?
Retha: Well, if the sign was built in, it could have a special time limit, kept by a digital countdown timer, after which it wouldn't be valid. Zara, your requirement doesn't say anything about time.
Zara: Well, I was trying to keep the expense down.
Sid: Parking cards.
Jack: What does that mean?
Martha: I think he means that the sign could have one of those little cardboard clocks, where you could set the hands.
Retha: That's still pretty expensive. And they'll get stolen.
(Retha stands up, goes to the board, and starts to write.)
Jack: What are you doing, Retha? I thought we agreed that I would be the one to write down all of Zara's faults.
Ned: Well, you're not writing anything.
Jack: Okay, okay. I'll write.
(Jack takes the pen from Retha and starts writing. Retha sits down and starts reading through Zara's requirements document.)
Sid: Language.
Jack: Can you hold that until I'm finished writing?
Ned: What are you writing, anyway?
Jack: I'm copying my list of Zara's faults. Then you can add yours to the list.
Ned: Well, how about giving someone else a chance? I've got a list, too. In fact, I've got a long list. I swear, Zara, your document is about the most ridiculous excuse for a requirement that I've seen in thirty years. Nothing personal, of course.
Zara: It sounds kind of personal.
Ned: Don't be stupid. You shouldn't get so attached to your work. I don't.
Martha: Lay off, Ned.
Ned: You don't need to defend Zara. She's a big girl.
(Zara spills her tea in Ned's lap, then stomps out of the room.)
Jack: Well, I guess the review is complete. I'm sure Zara will get over it soon. It's always a bit tough to have your work subject to a critical review, but I think it was well worth it. We've got this great list of faults to pass on to management. Retha, will you copy them down and have them typed?
Retha: Bleep!
The technical review is a testing tool, a tool for indicating progress of the requirements work. Technical reviews come in many variations, under many names, but all play the same role in managing the requirements process, as indicated in Figure 20-1.
Figure 20-1. Technical reviews serve at least two functions in a project: feedback of issues to the producers to help improve the product, and feedback to management on the actual technical status of the project.
To manage the requirements process, managers and even the producers themselves need reliable feedback on progress. That's the reason for having a review of each requirements document by people who were not involved in producing it. This practice is known as the formal technical review. In an informal technical review, the producers of the document review their work critically within their own group.
"Formal" and "informal," as we use the terms here, have nothing to do with how the meeting is conducted, but serve only to identify where the output of the meeting goes: results from the informal reviews go to the producers only. Information from formal reviews goes to both producers and others—customers, project managers, or anyone else who needs reliable information on the progress of the requirements work. Inside an actual meeting, the two types of technical review may be identical in format. Indeed, an "informal" review could be conducted with a great deal more formality than a "formal" review of the same requirements document.
The other difference between the two reviews is the objectivity of the reviewers involved. While informal reviews are an excellent way to discover problems in a document, they are not always the best source of reliable information, because participants are less likely to discover their own mistakes and assumptions. In their brief meeting of the Do Not Disturb Project, for example, Jack and his coworkers managed to commit more than a dozen common reviewing errors, all the time seemingly unaware of what they were doing.
Figure 20-2 illustrates the important
distinction between a technical review and a project review,
sometimes called a management review. Technical reviews include
only experts on the subject matter and deal solely with technical
issues. In reviewing a requirements document, these subject-matter
experts are the customers or their representatives—the ones for
whom the product is being designed. Their job is to provide
information about the quality of the document. This information on
quality goes to the document's producers, for them to make
corrections.
Figure 20-2. Technical reviews provide information to project reviews, which also consider information on nontechnical matters, such as resources consumed and available.
The outcome of a technical requirements review would say nothing about such management issues as resources. The outcome of a project review, however, might be an adjustment of requirements to match resources, or of resources to match requirements.
A summary of the quality information from the technical review also goes to the project reviews. There, managers consider quality information along with information about costs, schedules, and resources in order to make informed judgments of what actions to take. We're now familiar with the consequences of failing to control the quality of requirements—missed schedules, cost overruns, unmet requirements, inadequate performance, error-prone products, and never-ending rework of the product, to name only a few. Only a proper combination of technical reviews and project reviews can prevent such nasty outcomes.
Whatever takes place inside a requirements review, its major project control function is to provide to interested parties such as customers, users, and management a reliable answer to the fundamental question:
Does this requirement do the job it's supposed to do?
That answer is conveyed in the review reports.
Once any piece of the requirements document has been reviewed and found acceptable, it becomes part of the ongoing requirements process. Therefore, within a very small risk factor, the document must be
1. complete
2. correct
3. dependable as a base for related work
4. measurable for purposes of tracking progress
Without technical reviews, there are no consistently reliable methods for measuring the progress of the requirements work. Sometimes the producers themselves give a reliable report, but sometimes is not good enough. No matter how good their intentions, producers are simply not in a position to give consistently reliable reports on their own products.
For small, simple projects, with well-intentioned, competent producers, there is a finite chance of success without reliable measures of progress. As projects grow larger and more complex, however, working without reviews is like exploring the jungle without a compass. The chance of at least one self-report being overly optimistic is no longer a chance but a certainty. And if the requirements are mistakenly thought to be correct, the project may never emerge from the jungle alive.
The reports issued by a formal review are like a compass for management. They serve as a formal commitment by competent and unbiased people. According to their best expert judgment, a piece of the requirements work is complete, accurate, and dependable. By themselves, the review reports do not guarantee a project will not end in failure or crisis, any more than a compass guarantees you won't get lost in the jungle.
One of our clients, after a very poorly done requirements review, remarked, "Around here, we call such a review a 'Tates.'" When we looked puzzled, he explained how Colonel Stoopnagle on the old radio show, "It Pays to Be Ignorant," once invented a compass that points in random directions, which he named a "Tates." "And that's because," he said, "he who has a Tates is lost."
Trying to manage a requirements process with a "Tates" is a sure way to get you lost, but well-done review reports are not in themselves sufficient to guarantee process success. To keep the process on track, the project's management must use the information in the review reports to make informed and wise management decisions about what direction to take.
As a review meeting progresses, it raises and records issues with the requirements document, which become an issues list. This list is essential because it tells the producers why their work was not totally acceptable, ideally in appropriate detail—and with sufficient tact—to enable them to remedy the situation. Figure 20-3 is an example of an issues list from a requirements review.
Figure 20-3. An example of a requirements review issues list. No special form is needed, but special care is required to write accurate and effective issues.
The issues list serves primarily as a communication tool from one technical group to another. As it is not intended for nontechnical readers, it need not be "translated" for their eyes. It need not be fancy, as long as it is clear.
A common subversion of the review process is attempting to make the issues list into a solutions list. The job of the review committee is only to raise issues. It is the job of the producing unit to resolve them. A review committee is generally no better at resolving issues than a producing unit is at raising them. Even when the producers are reviewing their own work, attempting to produce solutions—idea generation—at the same time they're raising issues—idea reduction—is a one-way ticket to an unfocused meeting.
The one report always generated by a formal review is the technical review summary report. This report, intended for management and perhaps customers, carries the committee's assessment of the work, which is derived from a weighted opinion of the seriousness of the items on the issues list. Thus it is the fundamental link between the review process and the project management system. For effective project management, review summary reports must identify three items (see Figure 20-4):
1. What was reviewed?
2. Who did the reviewing?
3. What was their conclusion?
Figure 20-4. A sample requirements review summary report form. The form should fit on one page, with supplementary pages added if necessary to describe exceptional situations, so the overall status of a piece of work can be seen at a glance.
An informal review obviously does not need a review summary report, though in practice many informal reviews go through the discipline of creating a summary appraisal of the work unit as acceptable or not acceptable, for what reasons, and to what degree. This final appraisal is essential. Nobody can just count items on the issues list and know how significant those issues are.
There are many different kinds of reviews, with varying formats and purposes. In this section, we'll describe some of the most common variations.
A requirements review can be conducted without any particular meeting discipline decided in advance. You simply adjust the course of the meeting to the demands of the product under review—like adding toppings to vanilla ice cream to create a do-it-yourself sundae. This type of "vanilla" review can be rather effective because of its adaptability, but for this reason, it requires a more skilled facilitator than some of the more structured varieties.
Over time, the project culture will tend to evolve special disciplines emphasizing certain aspects of reviewing at the expense of others. Even more important, it will also develop more skilled review leaders. Unfortunately, on a new project, these developments may arrive too late to help the requirements process. For requirements work, it's a good idea to have a pre-planned review discipline and/or to retain experienced facilitators to lead the earliest reviews.
Many of the best-known review disciplines are attempts to cover a great quantity of material in the review. For example, the "inspection" approach tries to gain such efficiency by focusing on a narrow, sharply defined, set of questions. In some cases, an inspection consists of running through a checklist of potential faults, one after another, over the entire document
Faults not appearing on the checklist constitute major dangers of such inspection approaches. That's why effective inspection systems generally evolve methods for augmenting checklists as experience grows.
Another way to cover more material is by having someone who is very familiar with the product "walk through" it. When walking through the product, much detail can be skipped—which can be good if you're just trying to verify an overall approach or bad if your object is to find errors of detail.
In some cases, the walkthrough is close to a lecture about the product, which suggests another reason for varying the review approach. In some cases, rapid education of large numbers of people may suggest some variation of the walkthrough. But, as always, it's a good idea to make clear from the outset if the main purpose of the meeting is education, not idea reduction.
A walkthrough process is driven by the structure of the product being reviewed. In inspections, the list of points to be inspected determines the sequence. In vanilla reviews, the order is determined by the flow of the meeting as it unfolds. In contrast to these types, round-robin reviews emphasize a cyclical sharing by the various participants, with each taking an equal and similar portion of the entire reviewing task.
Round-robin reviews are especially useful in a situation often found in requirements work, when the participants are at roughly the same level of knowledge—a level that may be as high as can be obtained, but not as high as is needed. This might be the case when a large number of different users need to review the requirements, each from a different perspective. The round-robin format ensures nobody will shrink from participation through lack of confidence, while at the same time ensuring a more detailed look at the requirements, part by part.
If you observe actual reviews, you will never find one following all the "rules" of its particular type. That's because each type of review, like any other tool, has its own advantages and disadvantages, so that any real review contains some sort of mixture of all types. Let's take the walkthrough as an example to illustrate why real reviews always vary from the ideal.
Because the presenter has prepared in advance, a walkthrough can process a great deal of material at high speed. Moreover, since walkthrough reviewers are far more passive than those in other varieties, large numbers of people can attend the meeting and become familiar with the walked-through material. This larger audience can also bring many diverse viewpoints to bear on the presented material. If the audience is alert, and represents a broad cross section of skills and viewpoints, the walkthrough can give strong statistical assurance no major oversight lies concealed in the product.
The walkthrough does not make demands on the participants for advance preparation. In requirements reviewing, a large numbers of participants may be needed to include the views of all different users of the product. These users may come from diverse organizations not under the same operational control. In such situations, it may prove impossible to get everyone prepared for a review, so the walkthrough may be the only reasonable way to ensure all those present have actually even looked at the requirements.
So much for the advantages of a walkthrough. As with all the review varieties, the problems of the walkthrough spring rather directly from these unique advantages. Advance preparation by all participants is not required, so everyone may have a different depth of understanding. Those close to the work may be bored and not pay attention. Those who are seeing the requirements for the first time may not be able to keep up with the pace of presentation. In either case, the ability to raise penetrating issues is impaired.
In order to retain their attention, the review leader may, at intervals, ask participants in turn for comments, thus introducing a round-robin element to the meeting. In order to keep up with complex material, individuals may bring their own private checklists, thus adding an element of inspections to the meeting. Rather than try to suppress these "deviations," use them as information about how the review tool needs to be sharpened.
1. Everything we have said about meetings as tools naturally applies to requirements reviews as well. For instance, the requirements review should be prepared to produce a report of related issues, to help keep the meeting on track.
2. The reports from technical reviews, when analyzed together, can provide an important historical overview of the project so far. They can reveal patterns among issues over time, which can guide management in changing which tools are used, how tools are used, and what kind of training is given to project participants. To make historical analysis possible, create a file of all summary reports and issues lists, and try to ensure they are done in comparable formats. Don't, however, let the formatting needs of historical analysis put a dead hand on the live process of doing the reviews.
3. Nobody likes to be criticized, and it is far too easy to believe a technical review is criticizing you as a person, rather than your product. A thoughtful set of agreed-upon rules helps get through this delicate time, as does a talented corps of review leaders and a set of experienced reviewers. If people are reluctant to have their products reviewed, or even attend reviews, that's an indication it's time to look at the rules you use, the leaders you select, and the reviewers you invite.
Requirements reviews are needed because no one can produce error-free requirements on a consistent basis, and because the producers are the least likely people to see the errors in their own products.
Use requirements reviews whenever you need to determine whether a requirements document does indeed do the job it was intended to do.
There are many review disciplines from which to choose.
1. Try them all, and adapt them to your particular needs.
2. Allow time for learning, and have patience with your awkwardness.
3. Develop a feedback system so that what you learn in early reviews is used to improve future reviews. The easiest way to do this is to develop a trained corps of review leaders.
Choose review leaders for their interpersonal skills and their special training in the various review disciplines.
Select review participants on the basis of their ability to detect issues in the product, as well as their ability to handle themselves appropriately in what can be a most delicate situation. Customers and users are the primary candidates for participation, but you may need some skill and tact to decide which customers and users are to be invited. This is particularly true of certain combinations of users, and you may wish to conduct separate reviews of certain requirements documents just to keep the meetings from becoming battlefields for political factions.