The sensory information from receptors in the skin and muscles is transmitted to the brain, where it is processed in various cortical areas to give rise to our haptic experiences. This chapter examines the haptic sense with a focus on how it is specialized to perceive the material properties of objects, and so is the sense of choice when judgments about such features have to be made. Here we compare the attributes of the haptic sense—its ability to process information spatially and temporally—to those of vision and audition. This comparison demonstrates that it is an intermediary sense in terms of its capacities. An important feature of haptic perception is the way that information about the external world is acquired. We review the important role of different hand movements in extracting specific information about objects, showing these movements, known as exploratory procedures, to be both optimal and sufficient for the property of interest.
Haptic perception senses the physical properties of objects encountered in the environment, thus its focus is external rather than on internal tactile sensations. Since the hand is the primary structure used in tactual exploration of the external world, the study of haptic perception primarily involves studying the hand.1 The distinction between haptic and tactile sensing, or between active and passive touch, is based on the active component of haptic sensing: the hand is voluntarily moved across a surface or manipulates an object to obtain specific information. For some properties, like perceiving the roughness of a surface or detecting minute surface irregularities, as in the finish on a car, it matters little whether the hand moves across a stationary surface (haptic sensing) or the surface is moved across the stationary fingers (tactile sensing). What is critical to the performance of such perceptual tasks is the relative motion between the fingers and the surface.2 For other properties, such as the weight of an object, perception is facilitated by hand movements; for example, we are much more sensitive to weight when an object is lifted and jiggled up and down than when it rests passively on our outstretched hand.3 The ability to make judgments about properties like weight or surface texture depends critically on two senses, touch and kinesthesia.
People can generally identify common objects haptically within a couple of seconds of contact. When we pick up an eraser, its material properties, such as surface texture or compliance (hardness/softness), are easy to perceive and play an important role in the recognition process. Think of the ease with which we can determine with our eyes closed whether it is a golf ball or an egg that has been placed in our hands. An object’s geometric characteristics, such as its shape or volume, must generally be determined by following the contours of its shape, and so the perception of these attributes requires integration of haptic information over time. The haptic system is therefore much quicker at processing an object’s material as compared to its geometric properties.
The interaction between the fingers and the surface of an object, when broken down into its basic elements, is revealed to be surprisingly complex. The finger pad becomes flatter as pressure is applied, and there is lateral movement of the skin as it is compressed. The surface topography of the fingertip with its ridges affects this interaction, and then frictional forces between the skin and surface being explored affect the movements generated as the finger scans over the object’s surface. Finally, as the finger moves, small and complex vibrations occur that are transmitted as traveling waves across the skin surface and sensed by all types of mechanoreceptors in the skin. The elements of this complex process are illustrated in figure 5. Collectively, these interactions enable us to perceive a wealth of information about an object; they form the basis of haptic perceptual experience.
Figure 5 Schematic representation of the interaction between the fingertip and an object. There is considerable compression of the finger pad, even at low contact forces, and as the finger moves across the object frictional forces are generated, which result in minute vibrations that travel along the skin surface.
Vision and audition dominate our interactions with the world, so we tend to think of them as essential to our existence. However, as illustrated by the case of the individual with overwhelming sensory loss described in chapter 2, in the absence of our tactile and kinesthetic senses we are profoundly limited in our daily interactions. The tactile sense is considered to have a much lower bandwidth than either vision or audition, in that the amount of information it can process through time is lower than that of the other two senses. It has also been suggested that because the tactile modality is the first sensory system to develop—an embryo develops sensitivity to tactile stimulation by the eighth week of gestation—it may be more “primitive” than vision and audition. A number of metrics have been used to capture these differences among the sensory systems, and they reveal that in many ways touch is an intermediary sense.
Sensory systems are often compared in terms of their spatial and temporal resolution. The former refers to the spatial separation between stimuli that can be detected, for example, how close two probes can be on the skin for someone to perceive them as being distinct, a quantity known as the two-point threshold. On the fingertips, we can resolve a separation of about 1 mm, which places touch between vision and audition, visual spatial acuity being the best of the three senses. Spatial acuity can also be measured in terms of the feature size that can be detected. With no movement across the skin, small bumps around 0.2 mm can be detected, but when there is movement between the finger and the surface a feature as small as 13 nm (13 billionths of a meter) can be perceived on the fingertips.4 The latter number indicates that human tactile perception extends to the nanoscale. Temporal resolution refers to the time difference required for two pulses delivered to the skin to be perceived as successive and not simultaneous. For touch, a temporal difference of 5 ms can be resolved, which is better than vision (25 ms) but worse than audition (0.01 ms). Touch is therefore an intermediary sensory system in that its spatial resolving power is poorer than that of vision but superior to that of audition, and its temporal resolving capacity is better than vision but inferior to audition.
Another metric sometimes used to compare sensory systems is their information processing capacity as defined in terms of the number of bits of information per second that can be processed. Since the fingertip is one of the most tactually acute areas of the body, its processing capacity has been compared to those of the ear and the eye. The eye is the most efficient; estimates are in the range of 106 bits per second (bps), next is the ear at 104 bps, and finally the finger at 102 bps.5 Obviously these comparisons involve very different types of stimuli and tasks for each of the modalities, but the ranking does reflect the slower processing that characterizes the tactile/haptic sense.
Measures of spatial acuity on the skin include the two-point threshold described above; the smallest perceptible difference in the orientation of gratings (i.e., vertical versus horizontal) applied to the skin; and the minimum distance between two points of stimulation on the skin that is perceived to be different, known as point localization. For gratings, the smallest groove or ridge width at which the orientation is perceived defines the threshold; in the fingertips this is around 1 mm. Other measures of tactile acuity include the minimum level of pressure that can be applied to the skin and perceived, and the smallest feature that can be detected on an otherwise smooth surface. Pressure sensitivity is usually measured by indenting the skin with nylon monofilaments (similar to fishing line) of varying diameter until the filament buckles, and asking people whether they perceive the stimulus or not. The smaller the diameter, the lower the buckling force. Tactile pressure sensitivity is highest on the face, particularly around the mouth, followed by the torso and arms and hands, and lowest on the legs and feet. Gender influences pressure thresholds; women, on average, have lower thresholds (i.e., higher sensitivity) than men. However, it is possible that this sex difference in pressure sensitivity reflects finger size differences between men and women. The crucial variable that accounts for sex differences in tactile spatial acuity, as measured using the grating orientation task, is the size of the fingers. With decreasing finger size, tactile spatial perception improves, presumably due to the higher density of Merkel cells that cluster around the bases of sweat pores in the epidermis. The density of these mechanoreceptors, and of their associated SA1 afferent units, is directly related to tactile spatial acuity. This means that women, who on average have smaller hands than men, have finer spatial acuity, and that a woman and a man with similar sized fingers will, on average, have similar acuity.6
Sensitivity can also be considered in terms of how people perceive changes in force or pressure that are well above the threshold level. For example, if I press on your fingertip with a force of 4 newtons, which is very perceptible, how much does that force have to change before you perceive a difference? A more concrete example would be if you were holding an apple in each hand and wanted to know how different the weights had to be for you to perceive them as being different. For force and weight, the change is around 6%, which means that an apple weighing 213 gm (7.5 oz) or 241 gm (8.5 oz) is perceived as being different in weight from an apple weighing 227 gm (8 oz). This difference, known as the differential threshold, is sometimes also referred to as the just noticeable difference (JND) or the Weber fraction. It has been calculated for a broad range of sensory attributes, and because it is dimensionless it can be used to compare the sensitivity of different senses.1
Fluctuation of pressure on the skin through time is referred to as tactile vibration. The sensitivity of the skin to vibratory movements has been measured as a function of the frequency of vibration. We can detect vibrations delivered perpendicular to the skin from about 0.5 Hz up to about 700 Hz, but are not equally sensitive to displacement of the skin across this range of frequencies. We are most sensitive to vibrations between 200 and 300 Hz and progressively less sensitive at lower and higher frequencies. The relation between vibration frequency and displacement of the skin is therefore a U-shaped threshold function. The four types of cutaneous mechanoreceptors found in glabrous skin (described in chapter 1) are sensitive to different frequencies of vibration within this range. At threshold levels of vibration, it is the type of mechanoreceptor most sensitive to a particular range of vibrotactile frequencies that determines the perceptual threshold: our capacity to detect very small vibrations between 200 and 300 Hz is due to the FA II receptors that respond to vibrations above 50 Hz, whereas FA I receptors respond to vibrations between 25 and 40 Hz.7 For stronger vibrations well above these threshold levels, more than one type of mechanoreceptor typically responds to movements of the skin.
Haptic exploration enables us to perceive both the geometric and material properties of objects. The former refers to features such as size, shape, orientation, and curvature, whereas the latter includes attributes such as surface texture, compliance, and thermal characteristics. The size and shape of objects that fit within the hand can be perceived on the basis of skin indentation and the pose of the fingers, because the hand can enclose the object. For larger objects, which must be sequentially explored by the hand to determine shape, kinesthetic cues also contribute to perception. This sequential nature of haptic exploration, in which a percept or image of an object’s shape and size must be built up over time, imposes a load on memory processes that does not occur when shape is perceived visually and the object is seen in its entirety. This temporal aspect of sensory processing is one of the limitations of haptic perception of geometric properties. In addition, the process of perceiving geometric features haptically is subject to a number of biasing influences that systemically affect the veridicality of some percepts. For example, the haptically perceived length of a horizontal or vertical line is very similar to its physical length, but when its orientation becomes oblique its length is perceived less accurately. This effect also occurs when length is estimated visually. Several other visual illusions, such as the well-known Müller-Lyer illusion in which two lines of equal length are perceived to differ in length depending on whether the ends of the lines are enclosed by arrowheads or by fins, have also been shown to occur haptically. These illusions are described in more detail in chapter 4. The similarity in how the haptic and visual sensory systems process the geometric features of these stimuli indicates that their underlying perceptual processes are similar.
Of all the material properties of objects, the one that has been subject to most research is surface texture, which can be further classified in terms of roughness, stickiness, slipperiness, and friction. As these terms suggest, texture is a multidimensional construct. Understanding how we perceive these various aspects of a surface is important in a number of areas, from the feel of consumer products that we hold, such as phones, handles, paper, and fabrics, to creating such textures artificially on flat screens. Of all the products purchased online, the one category that people report a significant need to touch prior to making an Internet-based purchase is clothing. The challenge of conveying the distinctive feel of different fabrics—for example, velvet, leather, or silk—to Internet purchasers is daunting, given the subtleties of the tactile sense that would have to be recreated for a display to be effective. The “feel” of fabrics is related to a number of features including their thickness, compressibility, and tensile properties. In addition, when we rub a fabric between our fingers, we notice how the surface moves as we create shear forces.
When people are asked to classify different textures in terms of their perceived tactile dissimilarity, three dimensions often emerge around which the various textures cluster. The first two dimensions are roughness/smoothness and hardness/softness, which are independent and robust. The third dimension is sticky/slippery, which is not as salient as the other two dimensions and not as reliably perceived across all individuals.8 Stickiness refers to the sensations associated with making contact with adhesive tape or the resin on a pine tree, whereas slipperiness is experienced when the hand grasps a bar of soap that is wet. For this perceptual dimension, tangential forces are critical to perception.
Much of the research on surface texture has focused on roughness and understanding how judgments of roughness are made, including the influence of various parameters, such as the size and spacing of the elements that cover the surface, on these judgments. For relatively coarse textures like sandpaper, the perceived roughness relates to the gap between the elements comprising the surface, known as the spatial period, with larger spacing being perceived as rougher than textures with a smaller spacing. The perception of roughness for finer textures with much smaller spatial periods is based on the high-frequency vibrations generated by movement of the fingers over the surface.
Compliance refers to how deformable a surface is when force is applied, so it is defined as the ratio between displacement, or the movement of the surface, and the forces applied. The inverse of compliance is stiffness. These terms define physically what we often refer to perceptually along a softness to hardness dimension. When we shop for food, we may be interested in evaluating the compliance of a piece of fruit or a soft cheese because that provides us with information about its ripeness or freshness. With deformable items like a mattress or cushion, we are capable of discriminating varying degrees of compliance or softness. The change in compliance that is perceived, that is the differential threshold, is around 22%, which is considerably larger than the 6% described for weight and force. A distinction is often made between objects that have compliant or continuously deformable surfaces, like a rubber ball, versus rigid surfaces such as a foot pedal in a car, which also moves with applied force. With soft or deformable objects, information from cutaneous mechanoreceptors is critical to evaluating compliance, whereas for rigid objects both tactile and kinesthetic (i.e., haptic) cues are essential. Understanding how people perceive the compliance of rigid objects is important to the design of all types of interfaces that move when forces are applied such as keyboards, control panels, and steering wheels. For most of these interfaces we want to know when we have generated sufficient force to register a response—this is called the breakaway force—and so tactile feedback is often incorporated in the design of the display to signal when the breakaway force is exceeded. For keyboards, the force at which tactile feedback is presented ranges from 0.04 to 0.25 newtons.
A perceptual entity related to compliance is viscosity, which physically is the ratio of force to velocity. We are aware of viscous forces when we try to run in the ocean or when we mix substances like cornstarch and water together to create a viscous mixture. One area where the perception of viscosity has historically been important is in bread making and cheese and butter manufacture. These activities require judgment of the rheological properties (flow and deformation) of the materials as they are being manipulated. Although many of the processes associated with production of these foods have been automated, some culinary procedures, like making sauces and kneading dough for bread, still rely on discrimination of changes in viscosity. With the advent of teleoperated robots another application for understanding the perception of viscosity has emerged, since these robots are often used in viscous environments inside the human body or in the sea. In these applications it is important to understand how consistently human operators are able to discriminate changes in viscosity as the robots are moving. Differential thresholds for viscosity have been estimated to be around 19%,9 which is similar to the value reported for compliance (22%). This suggests that when force and movement cues have to be integrated, there is a loss in perceptual resolution since the differential thresholds for both force and limb movement are around 6% to 8%.
One of the interesting aspects of studying haptic sensing is determining what information we can acquire about an object as we manipulate it, and what types of hand movements are optimal for perceiving specific properties. For example, if people are asked to estimate the weight of a small melon, they will typically lift it up in the palm of the hand and move the hand up and down to perceive its weight. In contrast, if they are asked to judge how ripe the melon is, they will poke the surface of the fruit with one or two fingers to determine whether it is hard or soft. Such movements are referred to as exploratory procedures (EP) and are defined as a stereotyped pattern of manual exploration that is used when people are asked to judge a particular property.10 These movements have been shown to be optimal, in that they provide the most precise information for extracting the property of interest. To perceive temperature, for example, the EP of static contact is used in which a large area of the hand rests on an object without moving, maximizing the contact area between the skin and the object for heat flow between the two. This EP is optimal for perceiving temperature when compared to all other actions. The EPs that have been documented in laboratory studies are the movements that most people spontaneously use when asked to make a judgment about a specific property. The enclosure EP, in which the hand encloses the entire object to determine its global shape or volume, has been shown to be the most efficient way of judging the shape of an object. In contrast, to assess compliance (or hardness) the pressure EP is used to poke at the object and so determine its compressibility, whereas the lateral motion EP, in which a finger is moved back and forth across a surface, is used to perceive texture. Some of these EPs are illustrated in figure 6.
Figure 6 Exploratory procedures (EPs) and associated properties (in parentheses) that each EP is optimal at providing. From Jones and Lederman (2006)1 and adapted from Lederman and Klatzky (1987)10 with permission of Oxford University Press.
The EP that is executed to perceive a particular property, such as lateral motion to perceive texture, is in turn optimized to the specific situation in which it is being used. For example, people will vary the contact forces more when exploring smooth as compared to rough surfaces, and scan the surfaces more rapidly if they are asked to discriminate between them rather than just identify them. Similarly, when determining the compliance of an object, greater forces are used for rigid objects than compliant ones. In each of these situations, although the EP can be considered a stereotypical movement, it is itself modulated based on the task being performed.
There are costs and benefits associated with the various EPs in terms of exploration time and the additional information about the object that may be acquired along with the specific property being identified. The static contact EP provides information about the thermal properties of an object and at the same time yields incidental cues regarding its surface texture, shape, and volume. The most efficient approach to finding out about an unknown object is to grasp and lift it, as this provides coarse information about both its material and geometric properties. Three EPs are involved in this simple action: static contact, unsupported holding, and enclosure. Subsequently other EPs will be used to determine the presence of a particular property.
When we attempt to find an object that is hidden from view we use different strategies to simplify the perceptual experience. The ability to selectively attend to salient features is an example of complexity reduction. For example, if we are searching for our keys at the bottom of a bag filled with many items, we will make our search more efficient by attending only to those items that feel cold and hard. In such situations we are focusing our attention on a specific feature or target that we hope will make our exploration time shorter. In studies of the visual system, features have been identified that are more distinctive than other properties and are said to stand out from others. This phenomenon is known as the pop-out effect.11 An example from vision is the ease with which we can find a red circle among blue circles; it is seen immediately and we do not need to search through all the circles to find it. It is much harder, however, to find a slanted line among an array of vertical lines; in this case we need to search carefully through the visual scene to identify the line. That kind of search is a serial process in which items are explored one by one, whereas with features that pop out, the search occurs in parallel, and multiple items can be searched at once. The difference between these two strategies is reflected in the time taken to find the target—much longer response times are associated with the serial strategy, and also occur when the number of nontarget or distractor items increases.
When we identify haptic features that pop out we learn something about the processes used to recognize objects and discriminate between them. Haptic features that have been shown to pop out include roughness, coldness, and hardness. These features are not necessarily symmetric, that is, although a rough surface pops out in an array of smooth surfaces, a smooth surface does not pop out among rough surfaces. Other haptic features, such as the orientation of edges (vertical or horizontal), do not pop out, so individual elements in an array have to be serially processed to determine whether the selected feature is present. The high salience of material properties to haptic perception is reflected in the features that pop out during haptic search. These are also the features that people attend to when they are asked to judge the similarity of different objects using touch alone, because they are perceived quickly and efficiently. It is only when the same objects are visually perceived that geometric properties, like shape and size, become important in determining similarity.