Operant conditioning shapes behavior as a sculptor shapes a lump of clay. Although at some point the sculptor seems to have produced an entirely novel object, we can always follow the process back to the original undifferentiated lump, and we can make the successive stages by which we return to this condition as small as we wish. At no point does anything emerge which is very different from what preceded it. The final product seems to have a special unity or integrity of design, but we cannot find a point at which this suddenly appears. In the same sense, an operant is not something which appears full grown in the behavior of the organism. It is the result of a continuous shaping process.
The pigeon experiment demonstrates this clearly. “Raising the head” is not a discrete unit of behavior. It does not come, so to speak, in a separate package. We reinforce only slightly exceptional values of the behavior observed while the pigeon is standing or moving about. We succeed in shifting the whole range of heights at which the head is held, but there is nothing which can be accurately described as a new “response.” A response such as turning the latch in a problem box appears to be a more discrete unit, but only because the continuity with other behavior is more difficult to observe. In the pigeon, the response of pecking at a spot on the wall of the experimental box seems to differ from stretching the neck because no other behavior of the pigeon resembles it. If in reinforcing such a response we simply wait for it to occur—and we may have to wait many hours or days or weeks—the whole unit appears to emerge in its final form and to be strengthened as such. There may be no appreciable behavior which we could describe as “almost pecking the spot.”
The continuous connection between such an operant and the general behavior of the bird can nevertheless easily be demonstrated. It is the basis of a practical procedure for setting up a complex response. To get the pigeon to peck the spot as quickly as possible we proceed as follows: We first give the bird food when it turns slightly in the direction of the spot from any part of the cage. This increases the frequency of such behavior. We then withhold reinforcement until a slight movement is made toward the spot. This again alters the general distribution of behavior without producing a new unit. We continue by reinforcing positions successively closer to the spot, then by reinforcing only when the head is moved slightly forward, and finally only when the beak actually makes contact with the spot. We may reach this final response in a remarkably short time. A hungry bird, well adapted to the situation and to the food tray, can usually be brought to respond in this way in two or three minutes.
The original probability of the response in its final form is very low; in some cases it may even be zero. In this way we can build complicated operants which would never appear in the repertoire of the organism otherwise. By reinforcing a series of successive approximations, we bring a rare response to a very high probability in a short time. This is an effective procedure because it recognizes and utilizes the continuous nature of a complex act. The total act of turning toward the spot from any point the box, walking toward it, raising the head, and striking the spot may seem to be a functionally coherent unit of behavior; but it is constructed by a continual process of differential reinforcement from undifferentiated behavior, just as the sculptor shapes his figure from a lump of clay. When we wait for a single complete instance, we reinforce a similar sequence but far less effectively because the earlier steps are not optimally strengthened.
This account is inaccurate in one respect. We may detect a discontinuity between bringing the head close to the spot and pecking. The pecking movement usually emerges as an obviously preformed unit. There are two possible explanations. A mature pigeon will already have developed a well-defined pecking response which may emerge upon the present occasion. The history of this response might show a similar continuity if we could follow it. It is possible, however, that there is a genetic discontinuity, and that in a bird such as the pigeon the pecking response has a special strength and a special coherence as a form of species behavior. Vomiting and sneezing are human responses which probably have a similar genetic unity. Continuity with other behavior must be sought in the evolutionary process. But these genetic units are rare, at least in the vertebrates. The behavior with which we are usually concerned, from either a theoretical or practical point of view, is continuously modified from a basic material which is largely undifferentiated.
Through the reinforcement of slightly exceptional instances of his behavior, a child learns to raise himself, to stand, to walk, to grasp objects, and to move them about. Later on, through the same process, he learns to talk, to sing, to dance, to play games—in short, to exhibit the enormous repertoire characteristic of the normal adult. When we survey behavior in these later stages, we find it convenient to distinguish between various operants which differ from each other in topography and produce different consequences. In this way behavior is broken into parts to facilitate analysis. These parts are the units which we count and whose frequencies play an important role in arriving at laws of behavior. They are the “acts” into which, in the vocabulary of the layman, behavior is divided. But if we are to account for many of its quantitative properties, the ultimately continuous nature of behavior must not be forgotten.
Neglect of this characteristic has been responsible for several difficult problems in behavior theory. An example is the effect sometimes spoken of as “response generalization,” “transfer,” or “response induction.” In reinforcing one operant we often produce a noticeable increase in the strength of another. Training in one area of skilled behavior may improve performance in another. Success in one field of activity may increase the tendency to be active in other fields. By arranging optimal reinforcing contingencies in the clinic or institution, the psychotherapist strengthens behavior in the world at large. But how is this possible? What is the “transfer” which appears to strengthen behavior without reinforcing it directly? This is a good example of a pseudo problem. We divide behavior into hard and fast units and are then surprised to find that the organism disregards the boundaries we have set. It is difficult to conceive of two responses which do not have something in common. Sometimes the same muscular system is used. The effect of a reinforcement may reflect this fact rather than our arbitrary practice of calling the responses separate units. Again, when we reinforce the final response in a sequence containing many precurrent members, we may strengthen all units which contain the same precurrent members. Our skill in manipulating tools and instruments transfers from one field of reinforcement to another.
The traditional explanation of transfer asserts that the second response is strengthened only insofar as the responses “possess identical elements.” This is an effort to maintain the notion of a unit of response. A more useful way of putting it is to say that the elements are strengthened wherever they occur. This leads us to identify the element rather than the response as the unit of behavior. It is a sort of behavioral atom, which may never appear by itself upon any single occasion but is the essential ingredient or component of all observed instances. The reinforcement of a response increases the probability of all responses containing the same elements. Verbal behavior supplies especially good examples of the need to consider these atoms. An enormous number of verbal responses are executed by the same musculature. They are responses, therefore, which are presumably composed of a fairly small number of identical elements. This is not usually recognized in the customary practice of regarding verbal behavior as composed of separate units—for example, the “words” of the grammarian. A rigorous analysis shows that the word is by no means the functional unit. Larger complexes of words—idioms, phrases, or memorized passages—may vary together under the control of a single variable. On the other hand, we may observe the separate functional control of “atoms” at least as small as the separate speech sounds. We have to recognize these small units in order to account for such distorted verbal responses as spoonerisms and certain verbal slips, as well as the stylistic devices of alliteration, assonance, rhyme, and rhythm.
We lack adequate tools to deal with the continuity of behavior or with the interaction among operants attributable to common atomic units. The operant represents a valid level of analysis, however, because the properties which define a response are observable data. A given set of properties may be given a functional unity. Although methods must eventually be developed which will not emphasize units at this level, they are not necessary to our understanding of the principal dynamic properties of behavior.
Although operant reinforcement is always a matter of selecting certain magnitudes of response as against others, we may distinguish between producing a relatively complete new unit and making slight changes in the direction of greater effectiveness in an existing unit. In the first case, we are interested in how behavior is acquired; in the second, in how it is refined. It is the difference between “knowing how to do something” and “doing it well.” The latter is the field of skill.
The contingency which improves skill is the differential reinforcement of responses possessing special properties. It may be supplied automatically by the mechanical exigencies of the environment. In learning to throw a ball well, for example, certain responses must release the ball from the fingers at the moment of its greatest forward speed. These responses are differentially reinforced by the fact that, when so released, the ball covers a considerable distance. Other instances in which the release comes before or after the proper moment are not so reinforced. We are likely to forget how complex an act this is and how much differential reinforcement is required in the young child to produce a properly timed sequence. In games, crafts, and certain artistic performances extremely fine differences in the execution of behavior make important differences in the consequences. (The consequences at issue are generally the conditioned reinforcers summarized in Chapter V. Primary reinforcers are seldom involved. The negative reinforcers to be considered in Chapter XI also are important. For example, the consequences which are effective in conditioning postural responses in locomotion or the maintenance of an upright position are largely the avoidance of falls, bumps, and awkward or painful postures.)
The reinforcement which develops skill must be immediate. Otherwise, the precision of the differential effect is lost. In many practical areas skilled behavior is encouraged by arranging a quick report of accomplishment. In rifle practice, for example, extremely small-scale properties of response are differentially reinforced by a hit or a miss. Properties of this magnitude can be selected only if the differential reinforcement is immediate. But even when a hit can be seen by the rifleman, the report is delayed by the time which the bullet takes to reach the target. Possibly this gap is bridged by conditioned reinforcement from the “feel” of the shot. The rifleman eventually “knows” before the target is hit whether the shot was good or bad. His own behavior generates a stimulating feed-back, certain forms of which are followed by hits, others by misses. The more immediate problem is to shoot in such a way as to generate the “feel” followed by a hit. In more vigorous enterprises the feed-back is clearer. Good form in bowling, for example, is reinforced by feed-back from the bowler’s body. This does not mean that the rifleman will continue to shoot well, or the bowler to bowl well, even though he receives no report of the effect upon the target or pins. The report is needed to maintain the conditioned reinforcing power of the feed-back.
If the differential contingencies change, the topography of behavior changes with them. Even the very common responses which enable us to walk upright continue to be modified by the environment. When we walk on the deck of a ship at sea, a special set of contingencies prevails in maintaining our orientation in the gravitational field. The new differential reinforcement sets up “sea legs.” At the end of the voyage the old contingencies work a reverse change. Contingencies of reinforcement which are arranged by society are especially likely to shift. Verbal behavior supplies many good examples. In the nursery, crude vocal responses are successful; the indulgent parent may even reinforce “baby talk” into adolescent or adult years. But eventually, verbal behavior is successful only when it generates suitable behavior in the average listener; therefore, the form of the behavior comes to correspond more and more closely to the standards of a given community. When we move from one community to another, the topography of our behavior may change.
Some differential reinforcements make a response more or less intense or forceful without appreciably altering its topography. Certain natural contingencies in the environment lead us to push or lift harder to move objects, to pull harder to break objects apart, to jump harder to reach a given height, and so on. In calling to someone at a distance or in talking to a deaf person, our verbal behavior is reinforced only when it reaches a certain level of loudness. Tests of strength and other competitive games supply examples of these differential contingencies. When a heavy ball is thrown beyond a certain mark, when a horizontal bar is cleared in vaulting or in jumping, when a ball is batted over the fence (and when, as a result, a record is broken or a match or game won), differential reinforcement is at work. It may to some extent change the topography of the behavior and produce “good form,” but it has an important effect upon the mere force with which the behavior is executed.
We use differential reinforcement to shape and intensify the behavior of others in what may be spoken of, as we shall see in Chapter XX, as deliberate control. The effect may also be wholly unintentional. The mother who complains that her three-year-old child whines and cries for attention in an annoying way may not realize that her own reinforcing practices are responsible. If she is busy with other matters, she is likely not to respond to a call or request made in a quiet tone of voice. When the child raises his voice, she replies. This is differential reinforcement. The average intensity of the child’s vocal behavior rises. When the mother has adapted to the new level, again only the louder instances are reinforced. Further differentiation in the direction of loud responses follows. The child’s voice may also vary in intonation. What we call “whining” may be thought of as speaking with a small admixture of crying. Such speech is more likely to secure an effect and is therefore differentially strengthened. In fact, what we call annoying behavior in general is just that behavior which is especially effective in arousing another person to action. Differential reinforcement supplied by a preoccupied or negligent parent is very close to the procedure we should adopt if we were given the task of conditioning a child to be annoying.
One reason the term “learning” is not equivalent to “operant conditioning” is that traditionally it has been confined to the process of learning how to do something. In trial-and-error learning, for example, the organism learns how to get out of a box or how to find its way through a maze. It is easy to see why the acquisition of behavior should be emphasized. Early devices for the study of learning did not reveal the basic process directly. The effect of operant reinforcement is most conspicuous when there is a gross change in behavior. Such a chance occurs when an organism learns how to make a response which it did not or could not make before. A more sensitive measure, however, enables us to deal with cases in which the acquisition of behavior is of minor importance.
Operant conditioning continues to be effective even when there is no further change which can be spoken of as acquisition or even as improvement in skill. Behavior continues to have consequences and these continue to be important. If consequences are not forthcoming, extinction occurs. When we come to consider the behavior of the organism in all the complexity of its everyday life, we need to be constantly alert to the prevailing reinforcements which maintain its behavior. We may, indeed, have little interest in how that behavior was first acquired. Our concern is only with its present probability of occurrence, which can be understood only through an examination of current contingencies of reinforcement. This is an aspect of reinforcement which is scarcely ever dealt with in classical treatments of learning.
In general, behavior which acts upon the immediate physical environment is consistently reinforced. We orient ourselves toward objects and approach, reach for, and seize them with a stable repertoire of responses which have uniform consequences arising from the optical and mechanical properties of nature. It is possible, of course, to disturb the uniformity. In a “house of mirrors” in an amusement park, or in a room designed to supply misleading cues to the vertical, well-established responses may fail to have their usual effects. But the fact that such conditions are so unusual as to have commercial value testifies to the stability of the everyday world.
A large part of behavior, however, is reinforced only intermittently. A given consequence may depend upon a series of events which are not easily predicted. We do not always win at cards or dice, because the contingencies are so remotely determined that we call them “chance.” We do not always find good ice or snow when we go skating or skiing. Contingencies which require the participation of people are especially likely to be uncertain. We do not always get a good meal in a particular restaurant because cooks are not always predictable. We do not always get an answer when we telephone a friend because the friend is not always at home. We do not always get a pen by reaching into our pocket because we have not always put it there. The reinforcements characteristic of industry and education are almost always intermittent because it is not feasible to control behavior by reinforcing every response.
As might be expected, behavior which is reinforced only intermittently often shows an intermediate frequency of occurrence, but laboratory studies of various schedules have revealed some surprising complexities. Usually such behavior is remarkably stable and shows great resistance to extinction. An experiment has already been mentioned in which more than 10,000 responses appeared in the extinction curve of a pigeon which had been reinforced on a special schedule. Nothing of the sort is ever obtained after continuous reinforcement. Since this is a technique for “getting more responses out of an organism” in return for a given number of reinforcements, it is widely used. Wages are paid in special ways and betting and gambling devices are designed to “pay off” on special schedules because of the relatively large return on the reinforcement in such a case. Approval, affection, and other personal favors are frequently intermittent, not only because the person supplying the reinforcement may behave in different ways at different times, but precisely because he may have found that such a schedule yields a more stable, persistent, and profitable return.
It is important to distinguish between schedules which are arranged by a system outside the organism and those which are controlled by the behavior itself. An example of the first is a schedule of reinforcement which is determined by a clock—as when we reinforce a pigeon every five minutes, allowing all intervening responses to go unreinforced. An example of the second is a schedule in which a response is reinforced after a certain number of responses have been emitted—as when we reinforce every fiftieth response the pigeon makes. The cases are similar in the sense that we reinforce intermittently in both, but subtle differences in the contingencies lead to very different results, often of great practical significance.
Interval reinforcement. If we reinforce behavior at regular intervals, an organism such as a rat or pigeon will adjust with a nearly constant rate of responding, determined by the frequency of reinforcement. If we reinforce it every minute, the animal responds rapidly; if every five minutes, much more slowly. A similar effect upon probability of response is characteristic of human behavior. How often we call a given number on the telephone will depend, other things being equal, upon how often we get an answer. If two agencies supply the same service, we are more likely to call the one which answers more often. We are less likely to see friends or acquaintances with whom we only occasionally have a good time, and we are less likely to write to a correspondent who seldom answers. The experimental results are precise enough to suggest that in general the organism gives back a certain number of responses for each response reinforced. We shall see, however, that the results of schedules of reinforcement are not always reducible to a simple equating of input with output.
Since behavior which appears under interval reinforcement is especially stable, it is useful in studying other variables and conditions. The size or amount of each reinforcement affects the rate—more responses appearing in return for a larger reinforcement. Different kinds of reinforcers also yield different rates, and these may be used to rank reinforcers in the order of their effectiveness. The rate varies with the immediacy of the reinforcement: a slight delay between response and the receipt of the reinforcer means a lower over-all rate. Other variables which have been studied under interval reinforcement will be discussed in later chapters. They include the degree of deprivation and the presence or absence of certain emotional circumstances.
Optimal schedules of reinforcement are often of great practical importance. They are often discussed in connection with other variables which affect the rate. Reinforcing a man with fifty dollars at one time may not be so effective as reinforcing him with five dollars at ten different times during the same period. This is especially the case with primitive people where conditioned reinforcers have not been established to bridge the temporal span between a response and its ultimate consequence. There are also many subtle interactions between schedules of reinforcement and levels of motivation, immediacy of reinforcement, and so on.
If behavior continues to be reinforced at fixed intervals, another process intervenes. Since responses are never reinforced just after reinforcement, a change, to be described in Chapter VII, eventually takes place in which the rate of responding is low for a short time after each reinforcement. The rate rises again when an interval of time has elapsed which the organism presumably cannot distinguish from the interval at which it is reinforced. These changes in rate are not characteristic of the effect of wages in industry, which would otherwise appear to be an example of a fixed-interval schedule. The discrepancy is explained by the fact that other reinforcing systems are used to maintain a given level of work, as we shall see in Chapter XXV. Docking a man for time absent guarantees his presence each day by establishing a time-card entry as a conditioned reinforcer. The aversive reinforcement (Chapter XI) supplied by a supervisor or boss is, however, the principal supplement to a fixed-interval wage.
A low probability of response just after reinforcement is eliminated with what is called variable-interval reinforcement. Instead of reinforcing a response every five minutes, for example, we reinforce every five minutes on the average, where the intervening interval may be as short as a few seconds or as long as, say, ten minutes. Reinforcement occasionally occurs just after the organism has been reinforced, and the organism therefore continues to respond at that time. Its performance under such a schedule is remarkably stable and uniform. Pigeons reinforced with food with a variable interval averaging five minutes between reinforcements have been observed to respond for as long as fifteen hours at a rate of from two to three responses per second without pausing longer than fifteen or twenty seconds during the whole period. It is usually very difficult to extinguish a response after such a schedule. Many sorts of social or personal reinforcement are supplied on what is essentially a variable-interval basis, and extraordinarily persistent behavior is sometimes set up.
Ratio reinforcement. An entirely different result is obtained when the schedule of reinforcement depends upon the behavior of the organism itself—when, for example, we reinforce every fiftieth response. This is reinforcement at a “fixed ratio”—the ratio of reinforced to unreinforced responses. It is a common schedule in education, where the student is reinforced for completing a project or a paper or some other specific amount of work. It is essentially the basis of professional pay and of selling on commission. In industry it is known as piecework pay. It is a system of reinforcement which naturally recommends itself to employers because the cost of the labor required to produce a given result can be calculated in advance.
Fixed-ratio reinforcement generates a very high rate of response provided the ratio is not too high. This should follow from the input-output relation alone. Any slight increase in rate increases the frequency of reinforcement with the result that the rate should rise still further. If no other factor intervened, the rate should reach the highest possible value. A limiting factor, which makes itself felt in industry, is simple fatigue. The high rate of responding and the long hours of work generated by this schedule can be dangerous to health. This is the main reason why piecework pay is usually strenuously opposed by organized labor.
Another objection to this type of schedule is based upon the possibility that as the rate rises, the reinforcing agency will move to a larger ratio. In the laboratory, after first reinforcing every tenth response and then every fiftieth, we may find it possible to reinforce only every hundredth, although we could not have used this ratio in the beginning. In industry, the employee whose productivity has increased as the result of a piecework schedule may receive so large a weekly wage that the employer feels justified in increasing the number of units of work required for a given unit of pay.
Under ratios of reinforcement which can be sustained, the behavior eventually shows a very low probability just after reinforcement, as it does in the case of fixed-interval reinforcement. The effect is marked under high fixed ratios because the organism always has “a long way to go” before the next reinforcement. Wherever a piecework schedule is used—in industry, education, salesmanship, or the professions—low morale or low interest is most often observed just after a unit of work has been completed. When responding begins, the situation is improved by each response and the more the organism responds, the better the chances of reinforcement become. The result is a smooth gradient of acceleration as the organism responds more and more rapidly. The condition eventually prevailing under high fixed-ratio reinforcement is not an efficient over-all mode of responding. It makes relatively poor use of the available time, and the higher rates of responding may be especially fatiguing.
The laboratory study of ratio reinforcement has shown that for a given organism and a given measure of reinforcement there is a limiting ratio beyond which behavior cannot be sustained. The result of exceeding this ratio is an extreme degree of extinction of the sort which we call abulia (Chapter V). Long periods of inactivity begin to appear between separate ratio runs. This is not physical fatigue, as we may easily show by shifting to another schedule. It is often called “mental” fatigue, but this designation adds nothing to the observed fact that beyond a certain high ratio of reinforcement the organism simply has no behavior available. In both the laboratory study of ratio reinforcement and its practical application in everyday life, the first signs of strain imposed by too high a ratio are seen in these breaks. Before a pigeon stops altogether—in complete “abulia”—it will often not respond for a long time after reinforcement. In the same way, the student who has finished a term paper, perhaps in a burst of speed at the end of the gradient, finds it difficult to start work on a new assignment.
Exhaustion can occur under ratio reinforcement because there is no self-regulating mechanism. In interval reinforcement, on the other hand, any tendency toward extinction is opposed by the fact that when the rate declines, the next reinforcement is received in return for fewer responses. The variable-interval schedule is also self-protecting: an organism will stabilize its behavior at a given rate under any length of interval.
We get rid of the pauses after reinforcement on a fixed-ratio schedule by adopting essentially the same practice as in variable-interval reinforcement: we simply vary the ratios over a considerable range around some mean value. Successive responses may be reinforced or many hundreds of unreinforced responses may intervene. The probability of reinforcement at any moment remains essentially constant and the organism adjusts by holding to a constant rate. This “variable-ratio reinforcement” is much more powerful than a fixed-ratio schedule with the same mean number of responses. A pigeon may respond as rapidly as five times per second and maintain this rate for many hours.
The efficacy of such schedules in generating high rates has long been known to the proprietors of gambling establishments. Slot machines, roulette wheels, dice cages, horse races, and so on pay off on a schedule of variable-ratio reinforcement. Each device has its own auxiliary reinforcements, but the schedule is the important characteristic. Winning depends upon placing a bet and in the long run upon the number of bets placed, but no particular payoff can be predicted. The ratio is varied by any one of several “random” systems. The pathological gambler exemplifies the result. Like the pigeon with its five responses per second for many hours, he is the victim of an unpredictable contingency of reinforcement. The long-term net gain or loss is almost irrelevant in accounting for the effectiveness of this schedule.
A combined schedule. It is fairly easy to combine ratio and interval reinforcement in a laboratory experiment so that reinforcement is determined both by the passage of time and by the number of unreinforced responses emitted. In such a case, if the organism is responding rapidly, it responds many times before being reinforced, but if it is responding slowly, only a few responses occur before the next reinforcement. Such a schedule resembles either interval or ratio reinforcement, depending upon the values chosen in the combination, but there is some evidence that there is a middle ground in which neither schedule predominates and that the resulting behavior is unstable. Although this combined schedule may seem quite arbitrary, it is exemplified by many social situations where, as we shall see in Chapter XIX, the reinforcing agent may be affected by the level of the behavior reinforced.
We can reinforce an organism only when responses are occurring at a specified rate. If we reinforce only when, say, the four preceding responses have occurred within two seconds, we generate a very high rate. This is maintained even when we reinforce only at varying intervals with a fairly long mean interval. The rates exceed those which prevail under a variable-ratio schedule for the same net rate of reinforcement. Reinforcing a low rate of responding at variable intervals has the opposite effect of generating a sustained low rate. These studies have yielded many facts, too detailed to be discussed here, which explain why a given schedule of reinforcement has the effect it has. The effects of a schedule are due to the contingencies which prevail at the moment of reinforcement under it. Such schedules are, in other words, simply rather inaccurate ways of reinforcing rates of responding. They are often the most convenient way of doing this, and this may explain their widespread use in the practical control of behavior. But with proper instrumentation it should be possible to improve upon established practices in all these fields. Thus gambling devices could be “improved”—from the point of view of the proprietor—by introducing devices which would pay off on a variable-interval basis, but only when the rate of play is exceptionally high. The device would need to be more complex than the slot machine or roulette wheel but would undoubtedly be more effective in inducing play. Schedules of pay in industry, salesmanship, and the professions, and the use of bonuses, incentive wages, and so on, could also be improved from the point of view of generating maximal productivity.
Whether these improvements should be permitted is a matter to be discussed later. A schedule of reinforcement not only increases productivity, it also increases the interest, morale, and happiness of the worker. Any decision concerning a choice of schedules is complicated by this fact. In any event, we can act intelligently in this area only if we are in possession of clear-cut information regarding the nature and effect of the devices responsible for the maintenance of behavior in strength. We have much to gain from a close study of the results of experimental analyses.