Almost any good book on systems analysis will teach the reader how to use information from existing products as a requirements starting point. [See, for example, Stephen M. McMenamin and John F. Palmer, Essential Systems Analysis (Englewood Cliffs, N.J.: Prentice-Hall, 1984).] Therefore, instead of duplicating work done well elsewhere, we want to describe the use of existing products as an ending point. In this chapter, we'll discuss how to use the existing product to test the adequacy of the requirements job.
Thousands of years ago, Aristotle said, "It is not once, nor twice, but times without number that the same idea appears in the world." Now, as then, very few "new" products are entirely new. And, given the nature of people, almost any product you design will eventually be compared with some existing thing. Moreover, it will be compared in terms of the metrics developed for the existing products, not on its own merits.
Alexander Graham Bell may have invented the first telephone, but he had to suffer endless unfavorable comparisons with the telegraph. Today it may seem obvious that the telephone is a superior device in almost every way, but it wasn't obvious then. The standard of comparison for telegraph systems was "correct words received per minute," over a single line by skilled operators. Indeed, skilled telegraph operators could send and receive correct words at least as fast as they could be heard over a primitive telephone and written down.
Fortunately for Bell, he was aware that his invention would be held up to unfair comparison with the telegraph, and much of his energy was spent finding other measures for comparison. His colleague, Watson, was a tenor of some talent, so Bell had him sing over the phone to impress would-be investors. Over the telegraph, singing couldn't be done at all. Moreover since Watson sang familiar songs, his listeners didn't really notice they couldn't hear all the words very well.
Figure 23-1. New products are compared to old in terms of the metrics developed for the existing products, not on their own merits.
Whether you start or end the requirements process by looking at existing products, then, the new product will have to survive a comparison with those products. Since the existing products will be used as norms, regardless of whether or not they are appropriate norms, you need to use them as tests of requirements.
Just because you have interviewed certain users during requirements work doesn't mean you can't use them again as a test when a draft requirement is finished. This is especially true when you speak to representatives of users, because you merely have to choose a different set of representatives for the test. This approach also tests how good the representation was. But even if you interview the same people before and after, there are many useful tests you can perform, as we describe below.
In practice, the first thing people notice about a new product is whether it's missing anything they have come to expect in the products it is perceived as replacing. A few days after our colleague Stiles Roberts sent us a critique of an early draft of this book, he called to say, "I don't remember a requirement in the Elevator Information Device to indicate whether the elevator was about to go up or down." When reading the manuscript, he hadn't noticed the omission (did you?), but after riding in an ordinary elevator, the lack of an indication suddenly became obvious.
The same psychology forces up the rent every time we move to a new apartment. The new apartment must have everything the old one had, or we will sorely miss it. Or so we think initially. As it's unlikely the new apartment will have all and only what the previous apartment had, we will also pay for the new features. These new features, in turn, become requirements in our next apartment.
Thus, the first requirements test using existing products is to ask about missing functions. Not all the missing functions will be oversights. There may be functions we don't want the new product to supply. But we will at least want to be prepared to explain why those functions are missing, and how the user is to do without them. If we aren't prepared, the users will be sure to complain, and may reject the new product without ever appreciating its superiority.
Of course, we may actually find some oversights, in which case we will want to add them to the requirements documents. If there are many oversights, we will want to ask ourselves, "What deficiency in our requirements process led us to overlook so much? And what other problems may have arisen from that deficiency?"
Some years ago, a publisher commissioned a new information system to keep track of royalty payments to authors. When the requirements team believed their work was complete, they called upon Jerry to lead a technical review of the documents. He invited the six clerks who handled the present system to review the document, specifically comparing it to what they now did. The first thing they noticed was not the wonderful new features, but the lack of any procedure for correcting errors.
The systems analysts who had prepared the requirements believed the new system wouldn't make errors, and thus did not need any special error-correcting procedures. The clerks pointed out that only a few of the errors arose from their work, but instead originated in faulty reporting from the bookstores, as well as from misinterpretation of contract provisions.
Jerry then asked, "Why did the analysts leave out correction procedures?" They had examined the interface of this system with only one bookstore—the one the publisher ran themselves—and hadn't talked seriously to any authors. The requirements process was reopened, and an error-correction function was added. Perhaps even more important, as a result of the analysts' talking to a few authors, the entire royalty report was redesigned, much to everyone's satisfaction.
In this example, the test against the existing product revealed a flaw in the user-inclusion strategy. But even when the user inclusion has been more thoroughly done, it's easy to miss many functions. For one thing, interviewees can't always think of everything when put on the spot. For another, in the early stages of a project, the interviewers don't fully appreciate the significance of every answer they hear. It's always safe to assume something will be missed, and thus to schedule a test at the end.
Users are ordinarily very aware of certain annoying shortcomings of a product, and this information can be obtained through fairly direct questions, such as,
1. What doesn't work in the present product?
2. What groups of people have what troubles with the present product?
On the other hand, people who use a product can adapt to almost anything, and may no longer notice what's missing. Questions such as the following will tend to tease out this "forgotten" information:
3. How much time do people spend on various parts of the present product?
4. How much money do people spend on various parts of the present product?
Of course, there's an even easier way to find out what's missing in the old product: Look at the requirements for the old product and compare them to the product actually produced. When you make this comparison, you will learn something about the process by which requirements are translated into products. There is a reason for each missing requirement, and a reason for each function that's in the product though not in the requirements. If you can track down these reasons, you may learn a great deal about what you need in the present requirements.
For example, near the end of the requirements work of an electronic mail system, we discovered a copy of the requirements for the existing system. One "required" item conspicuously missing was a broadcast function enabling a user to transmit a message to every one of the system's users. The new draft requirements called for a similar function because some users had requested it. We asked one of the old-timers why it hadn't been implemented in the old system.
"Oh, but it was implemented," he told us. "After the system had been operating for a few weeks, someone broadcast a technical question about the system itself."
"That sounds like the kind of usage we were trying to foster," we said.
"Well, about twenty people had answers to the query, so each of them broadcast an answer. Then there were another twenty or so who didn't agree with each of those answers, and then there were those who didn't agree with their disagreements. By the time the mess died down, we figured there were more than a hundred thousand messages transmitted because of that one broadcast."
"So you removed the broadcast function?"
"Well, let's say we made it harder. You can still broadcast, but you'd have to make an explicit address list of every single person, which discourages swamping the system."
This little interview saved us from making the same costly mistake a second time, but we wouldn't have been able to ask the right questions if we hadn't run across a copy of the old requirements. It would be nice if companies routinely saved the requirements documentation for every product still in use, but sometimes you have to be a bit more creative about getting this kind of information. Here are some sources we've used successfully at least once in the past:
1. "Pack rats" are likely to have an unofficial copy of the old requirements. Ask people "Who saves everything?" to help locate the pack rats. Jerry once found a twenty-three-year-old requirements document this way.
2. Old news releases or articles, sometimes in-house organs, often have lists of features, at least. They can usually be found in complete sets in company libraries. If there is a company librarian, you have a powerful ally in your search.
3. Old advertisements and marketing literature for products are another source. The early releases are usually based on requirements, rather than on the actual product, and so reveal discrepancies between the two.
4. You can interview old-timers, but they're usually not too reliable unless they're prompted. At least try asking, "Can you remember anything promised but never delivered? How about something in the new system, but later taken out?"
Looking at existing systems is like opening a can of worms. In the worst case, you will tend to add features to match the existing product, rather than the functions you really want. This tendency is especially pronounced when product marketing people get into the requirements act with a competitive analysis—a list of all possible features for all possible competing products. By implication, they want all of these features to become requirements in the new product.
If your sales force can say your product has every feature of the competitor's product, then you can make the sale. This idea is sound enough. It should be duck soup to sell a Rolls Royce clone with the price and gas mileage of a Volkswagen Beetle (Figure 23-2).
Figure 23-2. It should be easy to sell a Rolls Royce with a Volkswagen's price and gas mileage, but can you build it at a profitable cost?
But can you build such a car and make a profit? Every time you add a feature, it tends to add constraints to the other attributes. Ten thousand blades will provide great competition for the Swiss Army knife—if you can keep the weight under ten pounds and the cost under a hundred dollars.
So how do you keep your product from becoming a Swiss Army product? Going through all the heuristics can help, but the most effective step is to create functions at the right level of description. Instead of trying to compete with everyone on a feature-by-feature comparison, try to identify the functions needed to compete, then develop the attributes of those functions in the usual way.
For instance, suppose you are building a new car, and the competitive analysis says the other cars in this class all have gold monograms on the steering wheel. When the marketing people try to make "gold monogram on steering wheel" into a requirement, ask, "What function does it serve?"
Most commonly, you'll get the reply, "It doesn't serve any function. It's just there to match the competitors."
"In that case," you say, "the function is 'match the competitors.' And just what is it about the competitor's we're trying to match?"
"Their pizzazz! Their sales appeal!"
"Great. So we can say that the function we want is 'match the competitor's sales appeal.' "
"Sure, but how are we going to do that?"
"Possibly with a gold monogram on the steering wheel, but possibly not. Perhaps the designers will give us a platinum monogram on the dashboard. Or perhaps they'll dream up a combination of price, performance, and pizzazz to do the trick. That's up to the designers to decide. Our job is to tell them our requirements."
1. Ask people to reminisce about the early days of the previous product, if anyone was present and can remember so far back. To the extent the culture hasn't changed, these memories will give you hints about features needed to get the product accepted for use in each environment.
2. A related approach is to try to discover what entire product is missing. Whenever we're working with an organization for the first time, we ask about other projects whose products never saw the light of day. We also ask about finished products withdrawn from use. Tracking these down always yields a barrelful of information.
3. Upgrading present products tends to produce "Swiss Army knife" products. To serve the existing population of users, you must retain all the old features as well as add new features to keep them happy. But to win new users, you must keep the product simple and cheap, attributes which tend to be destroyed by a Swiss Army of features. At some point, your best choice may be to split the product into two related products—a simple, unburdened one to attract novices, and a sophisticated one for jaded old-timers. Knowing when to split a requirements process into two is a great art and a possibility you should always keep in mind when the function list becomes unwieldy.
Use existing products as tests of requirements because they are yet another source of information about what is or is not desirable in the new product.
Do this test when you've completed a draft of requirements, though of course partial tests can be performed at any time during the requirements process. Indeed, when the old product is in daily use, it may be hard to avoid them, but don't assume these on-the-fly tests are the same as a careful survey.
Do the following:
1. Compare the products to develop a list of possible missing functions in the new requirement.
2. Interview users of the old product to develop a list of functions missing in the present system.
3. Compare the old product with its original requirements to prepare a list of potential problems in developing the new product. Especially look for requirements never implemented, or implemented but then scrapped.
4. Avoid the temptation to create a Swiss Army knife out of every product. Don't let features creep in without subjecting them to the full requirements process.
Try to find users who weren't involved in the original interviewing, in order to test your original sampling of users. If necessary, however, you can still gain much of value from the same users you interviewed initially.
Also, find people who don't use the existing product, and find out why not.