The Routledge Companion to Semiotics

4
LOGIC AND COGNITION

PEER BUNDGAARD AND FREDERIK STJERNFELT

The present essay aims to explicate a key area of contemporary semiotics by laying bare the importance assigned to schematic representations in recent theories of meaning: so-called morphodynamic semiotics and cognitive linguistics. It suggests and gives evidence for the existence of a prelinguistic level of meaning organization on which categorization and conceptualization is founded, and it shows how this level supports different sorts of inferences. Where, previously, much sign theory had relied on a conception of meaning as generated by a system independently of cognitive factors such as perception, action, etc., and of logical factors such as reasoning, truth, etc., the ideas underpinning the theory of schemata suggest a new basis for meaning. This basis can be developed in interaction with the reactualization of Peirce’s semiotics and the crucial role it assigns to diagrams in cognition and theoretical reasoning.

PREAMBLE

The combination of logic and cognition in semiotics goes without saying, since in one tradition semiotics is the study of how we humans, when thinking and reasoning, make use of, communicate, perceive and interpret signs and make inferences on the basis of these interpretations. In this, Peircean, view semiotics is the inquiry which lays bare how logical reasoning takes place in sign use, both in single inferences and in the scientific quest for knowledge in general.

However, in another, ‘continental’, tradition cognition and logic pertain to two realms which, from a scientific perspective, are heterogeneous: cognition belonging to psychology and its investigations into the empirical mental and psychophysical processes supporting human meaning making, as opposed to logic, now rephrased ‘semiotics’ proper, as the science which aims at establishing the formal tenets and internal scaffolding of meaning, freed from all psychology, context, concrete meaning intentions, actual manners of reasoning, etc. Roughly speaking, this latter position characterizes European semiotics from its roots in Husserlian phenomenology and structural linguistics – take, for instance, Algirdas Julien Greimas and the so-called Paris School of Semiotics (Greimas 1987). Here, the meaning system is self-contained, with its own intrinsic principles of organization, meaning generating rules, correlations between the deep structures of meaning and the significant surface phenomena, etc.; thus, semiotic investigations can be and should be conducted in isolation, as it were, independently of other sciences and their findings (here, mainly psychology, philosophy of mind and philosophy of language). In short, semiotics, just like structural linguistics and the subsequent formal, generative and transformational branches of linguistics, is concerned with the formal properties of its object (meaning and how it is generated), not with cognition: the properties of meaning making and the pragmatic constraints on human meaning making.

MEANING AND SCHEMAS IN MORPHODYNAMIC SEMIOTICS

Now, if the splitting of cognition and the logic of meaning characterizes continental semiotics in its origin and heyday (as crystallized in Greimas and Courtés 1979), from the mid-1980s a process of convergence was initiated. The ground for this convergence had been prepared by the French mathematicians René Thom’s (1972, 1983) and Jean Petitot’s (1992, 1995, 2004) groundbreaking contributions to linguistics and semiotics. Thom’s hypothesis, and Petitot’s further elaboration and refinement of it, is that there exist fundamental constraints on the construction of meaning which are not modular constraints, i.e. constraints stemming from the self-contained formal ‘meaning system’ (say, as it was defined in classical Greimasian semiotics independently of other systems such as perception and action). Rather, the constraints identified by Thom stem from mathematical topology and facilitate the mapping onto structural properties of the environment. They thus provide the structure of action and perception systems, in which meaning making – or semiosis – takes place.

Key to this approach is the concept of schema. Roughly speaking, Thom defines schemas as abstract representations of elementary spatio-temporal interactions between entities. In fact, he brings the argument a step further, in that in his view schemas are not only skeletal representations, they are the structural scaffolding of things themselves which may be extracted through perception. Now, what does ‘elementary interactions between spatio-temporal elements’ mean? Well, it means that they are recurrent across different domains of experience, and, of course, that they display the same structural characteristics across these domain differences: a simple example of this would be one entity’s ‘inclusion’ in some other entity, an abstract representation which then would make out the core meaning of all sorts of different verbal predications – say, to capture, swallow, eat, integrate, enter, etc. Another example could be the passage of an entity from one position to another, a transmission schema, which again constitutes the structural foundation of a long series of verbal predications, which aside from referring to the abstract schema specify it in different respects – say, send (object A from position X to position Y), give, receive, etc. (cf. Thom 1983 and the articles within it from the beginning of the 1970s for the first introduction to this hypothesis).

What is new – at least from a modern semiotic point of view – about this theory? Well, first, it displaces the origin or focus of semiosis: meaning is not defined as the outcome of certain abstract symbol manipulating processes, nor of some symbolic meaning generative device, but is something captured already in perception: in perception of events and their core interactional structure. Or, as Thom himself puts it,

As a consequence, can we not agree that the factors of phenomenological invariance, which create in the observer the feeling of meaning, come from real properties of the objects of the external world and the objective presence of formal entities linked to those objects, which we call ‘bearers of meaning’?

(Thom 1983: 169)

Next, interaction with the environment is based on a limited set of primitive, so-called archetypical schemas: as represented in the human being, these are first and foremost the cognitive counterpart to the invariant structures of the environment or the recurrent patterns of experience. In this respect they are claimed to play a linking function between man and environment: our full-blown representations are founded on these skeleton representations and hook up with the world by virtue of them. What is more, the fundamental schemas are in fact not only considered to be sheer templates available for event categorization, they are also claimed to be syntactic constituent structures, i.e. they serve a semantic binding function: when we use a given verb, say give, hand, send, etc., the verb will not only refer to a specific sort of action, but also evoke a schema consisting of a SENDER-postion (abstractedly understood), an OBJECT-position, a TRAJECTORY-position and a RECEIVER-position; these positions function as semantic roles when represented in the mind, and the schema therefore functions as a syntactic gestalt, a configurational structure. When it is activated, it will be activated with all its semantic roles and therefore call for completion in terms of a particular kind of action and the particular elements instantiating the different semantic roles.

Seen retrospectively, the implications for semiotics of René Thom’s topological linguistics and Jean Petitot’s initial developments of it are primarily epistemological: that is to say they champion and seem to justify the rejection of the traditional approach to meaning as a modular phenomenon, generated by a modular system, independently of perception, action, etc. Thom and Petitot address the crucial issue concerning the relation between perceived types of invariance in the environment and mental representations, and redefine meaning as a not exclusively – in fact, far from exclusively – linguistic phenomenon: rather, linguistic meaning is a rearticulation of prelinguistic meaning structures. In short, it brings the world back into semantics, as the English philosopher Barry Smith (1993) once put it. However, it does not, for that matter, provide semioticians and linguists with many, nor with sufficiently elaborated, tools for description. To this end, its schemas are far too coarse grained and the exclusive focus on the interactional schemas is far too restricted.

IMAGE SCHEMAS, SCRIPTS AND FRAMES IN COGNITIVE LINGUISTICS

Now, more or less parallel with, but independently of, the above, a similarly schema-driven approach to human meaning making developed, mainly in the United States. It is today known as cognitive linguistics and is represented by linguists, philosophers, psychologists and cognitive scientists such as Leonard Talmy (2000), Ronald Langacker (1987–1991), George Lakoff (1987), Mark Johnson (1987), Gilles Fauconnier and Mark Turner (2002), Jean Mandler (1992, 2005), Eve Sweetser (1990), etc. Key tenets of this programme are a couple of the assumptions which also constitute the cornerstone of the Thom–Petitot approach:

1 Language and cognition at large are not modal systems, they are not self-contained; they have developed along with and under the influence of other systems such as the perception and the action system.

2 Linguistic structure is grounded on a prelinguistic schematic structure, so-called schemas or image schemas (Johnson 1987; Mandler 1992, 2005); these schemas or abstract mental representations are acquired through perception and are as such cognitive counterparts to recurrent patterns of experience (more or less as in Thom) or they are acquired through bodily interaction with the environment: such schemas (as we shall see in somewhat more detail below) play a crucial role both in the organization of perceptual information (categorization and manner of experiencing a ‘referent scene’) and at a linguistic level as a core semantic structure.

Now, in contradistinction to previous approaches, cognitive linguistics supplies detailed, subtle and systematic descriptions of the relation between perception and language, that is between the way in which a given referent scene (that is, the objective situation referred to) has been experienced and the way in which this mode of experience is specified in language. (It is obviously crucial to give a satisfying account of this correlation if one champions the claim to the effect that meaning is not exclusively a linguistic phenomenon.) The notion of image schemata has a rich philosophical background including Kant’s definition of schemata as figures making possible the meeting between perception and intuition on the one hand and understanding and concepts on the other hand – as well as the notion in Gestalt theory where gestalts are structured wholes of perception, action and environment organization. In the following we shall introduce three aspects of image schemas. First, their origin in perception and action. Second, their cognitive function as regards categorization and conceptualization. Third, their logical import, that is to say the fact that they, in various ways, also constitute the basis for basic forms of reasoning – that is, they support inferences.

THE PSYCHOLOGY OF IMAGE SCHEMAS: WHEN AND HOW ARE THEY ACQUIRED?

As to the origin of image schemas, Jean Mandler (1992, 2005) provides exhaustive evidence for the fact that infants’ acquisition of concepts in general, for example the concept of ‘animacy’, is based on a limited number of primitive image schemas. Already from the age of three months they perform distinctions based on pattern recognition: as regards animacy, they are capable of distinguishing between mechanical and biological movement; and they, of course, perform such distinctions across different types of experiences (for example, they recognize incorrect biological movement whoever enacts it). Between four and six months, the core image schema of biological versus mechanical movement is supplemented with yet another essential moment: namely, the distinction between self motion and caused motion (the former being the distinctive feature of biological movement). The cognitive reality of this schema for core causality is attested by the fact that infants do consider designs (of a Henri Michotte sort – Michotte 1963; see also Scholl and Tremoulet 2000) involving plain ‘billiard ball causality’, where one sphere is seen to move toward and hit another which then itself moves, as perfectly normal. Yet they dishabituate from, or react against, the same set-up whenever there is either a temporal or spatial gap between the movements of the two spheres. The simple hypothesis is, of course, that infants categorize an event in terms of its correspondence or match with an image schema, so that whenever a fit obtains between the structural make-up of an experienced event – captured by the cognitive mechanism Mandler calls ‘Perceptual Meaning Analysis’ – and the acquired image schema (e.g. the schema for ‘launching’), then experience is categorically speaking felicitous; the inverse, of course, being the case for the misfit.

A decisive stage in the acquisition of the concept of animacy consists in the formation of a schema for goal-oriented behaviour and thus the recognition of purposeful movement. The experiment in Figure 4.1 (Gergely et al. 1995; here we have used Scholl and Tremoulet’s diagram) seems to provide evidence for the fact that nine–twelve-month-old infants do possess a ‘shortest-path-to-goal’ schema and thus assess the purposefulness of an action with respect to its conformity with such a schema.

In short, infants already engage in the process of capturing the structural design properties of events and objects. They do so by the cognitive mechanism of Mandler’s Perceptual Meaning Analysis. What are, then, the fundamental properties of this cognitive processing device (which must, by the way, be presupposed by any theory which defends a claim to the effect that abstract, structural information can be grasped through perception)?

Figure 4.1 Nine- and twelve-month-old infants were familiarized with a movie showing either scenario (a) or (b). In (a) an oblong figure approaches an obstacle and jumps over the obstacle to join the circle; (b) shows the same action but with the obstacle moved to one side. After familiarization the infants were tested by being shown (c) and (d). When confronted with cases (c) and (d) where the obstacle had been removed, infants that had been familiarized with (a) dishabituate from (d), i.e. show astonishment (look longer at (d) than at (c)), even though the movement is identical with the one they were habituated to. They recognize the movement in (c) as normal, i.e. the most direct way of reaching a goal. Infants familiarized with (b) did not manifest this behaviour. This seems to suggest that infants reason and assess on the basis of an abstract. (From Scholl and Tremoulet 2000.)

1 It is a spontaneous, attentive process which operates on perceptual information.

2 It is clearly preverbal, since performed by infants (even very young infants).

3 It recognizes and extracts invariant structure from the referent scenes.

4 It generalizes across particularities of different perceptual inputs.

5 It leads to the constitution of abstract representations which contain the skeleton structure common to different referent scenes.

6 This perceptual skeleton structure is what we understand by image schema.

As we have already seen, infants grasp and thus categorize events or relations in the environment by means of image schemas; image schemas are furthermore used to shape our experiences, i.e. they are used in the conceptualization of referent scenes; and they are finally mental devices supporting reasoning and inferences. So far we have only accounted for the fact that image schemas support concept formation and categorization. We have not commented on how such schemas are actively used in experience as tools for conceptualization, nor have we shown how humans use them to think or reason (although the Gergely experiment shows how infants assess the congruity of an action with respect to its conformity with a rational or adequate ‘shortest-path-to-goal’ schema). We shall now attempt to come to grips with these essential properties of schemas.

THE PHENOMENOLOGY OF IMAGE SCHEMAS: HOW DO THEY SHAPE OUR EXPERIENCES?

A common supposition in cognitive linguistics and cognitive semiotics is thus that humans acquire an inventory of fundamental images through perception or bodily interaction with the environment. The origin of image schemas, however, is multiple, and a plurality of other possible sources must be mentioned: inborn brain structures (giving us, e.g., the image schema of a face), a priori structures (giving us, e.g., simple arithmetic, geometry, topology, etc.) and language and culture (selecting and combining image schemas while strengthening some and suppressing others). Once they are deployed, they are of course available for use; they may, as we have seen, then facilitate smooth and efficient categorization of spatio-temporal interactions between elements in terms of causality, purpose, wholes, etc. Schemas can not only be used in the categorical comprehension of the world, but also, as it were, in the intentional apprehension of it: in fact, whenever we perceive or experience a scene, a situation (what in phenomenological terms Husserl (1973 [1939]) called a Sachlage), we organize this situation, by distributing our attention to alight on certain and not other of its elements, profiling this and not that aspect of it, paying attention to this part and not that part of it, etc. The result of the way in which we have organized the referent scene is the object of experience proper (Husserl called it the state of affairs, the Sachverhalt). Image schemas play a crucial role in the organization of the referent scene and thus the constitution of the experienced object: it is by and large possible to show how the semiotic scaffolding of an object of experience is relative to the image schemas applied to the referent scene (and specified by certain linguistic expressions); image schemas which are, in turn, applied according to the experiencer’s or the speaker’s intentional focus. This is particularly clear in cases of alternate schematizations; that is to say, cases where the same situation or the same object is referred to in two or more different ways. Consider:

1a	He went over the field
1b	He went through the field

2a	The boat is on the sea
2b	The boat is in the sea

3a	I jog together with him
3b	I jog along with him

As Talmy (2000, 2005) has remarked, in 1a, by virtue of the schematic meaning of ‘over’, the field is conceptualized as a sheer surface, with all other properties abstracted away, whereas in 1b it is rather conceptualized as a container-like entity which partially covers the agent (i.e. the vegetation of the field is more profiled). In 2a, the sea is conceptualized as a surface, in 2b as a container, with focus on the fact that the boat is partially immersed in the water, which suggests a proximal viewpoint in 2b and a distal viewpoint in 2a (from where material properties of the water, waves, viscosity, etc. are not perceivable). And, finally, in 3a ‘together’ suggests that ‘I’ and ‘he’ are ‘co-equal’ (Talmy 2005: 220) participants in the activity, whereas in 3b ‘along’ qualifies one of the participants (‘him’) as the main participant, and ‘I’ as ancillary with respect to him.

Examples like the above are legion. They all point to the fact that meaning organization takes place already in perception, that the structure of the perceived scene can be determined in image schematic terms (or in its simplest version in terms of figure-ground segregation as in ‘Peter has hit Paul’ versus ‘Paul has been hit by Peter’), and that this structure can be suitably expressed in language. In fact, Talmy claims that a whole subsystem of language – composed by the so-called closed word classes – is specialized in specifying the structure of the cognitive representation evoked by a given sentence or complex expression.

We are therefore now in a position where we can refine in some small measure the determination of the cognitive import of image schemas. The nice match between language and perception is an image schematic fit: the structures we make use of in our experience of the world are also the structures that make out the semantic core level of language. Language does refer to things in the world, and real properties in the world, but it does so through the schemas (and concepts) by means of which these things and properties have been intended.

SCHEMAS AND LOGIC: HOW SCHEMAS SUPPORT INFERENCES AND REASONING

As regards the schematic foundation of reasoning, arguably image schemas (as well as other more fleshy types of schemata, as we shall see) support inferences of different sorts, some very simple, others more subtle. As suggested above, it seems reasonable to consider the kind of cognitive reactions infants manifest when confronted with incongruous patterns of interactions (see Figure 4.1) as examples of inferences supported by image schemas. The referent scene is indeed assessed in terms of a schema for purposeful action (cf. Csibra et al. 2003 for more experimental evidence).

Now, it may be fruitful, however, to distinguish this sort of reasoning which concerns the fit or the misfit between an acquired schematic representation and the spatio-temporal structure of a referent scene, from reasoning and inferences directly based on the schema itself. Epitomes of such inferences are those based on the container-schema. Consider the schema of containment (which is one of the most fundamental, primitive and pervasively used schemas, probably due to early experience of our bodies as containers). A container-schema consists of an interior, a boundary and a relation to an exterior. Things are either inside or outside the container. From this a basic logic of transitivity can be derived: A is a container-schema, and X is in A. B is a container-schema, and if A is in B, then X is in B:

[A [X]], [B [A[X]]] => [B[X]]

This can be considered (as Lakoff 1988 claims) as the intuitive basis for modus ponens: if all As are Bs, and X is an A, the X is a B.

Everyday cognition offers, however, many other types of inferences, whose content is, say, more fluid and the extension more vaguely defined than in the above example. These are cases where the speaker’s meaning intention should be inferred from the way in which he has activated a given schema. Consider classical examples of implicature:

4a	He is a Republican, but he is smart
4b	They are married but they don’t live together

In both cases ‘and’ could, of course, be used with exactly the same truth value as ‘but’, but it would not imply the same thing: namely the fact that the speaker to some vaguely defined extent considers there to be some sort of incongruity between what is to the left and what is to the right of ‘but’. In a nutshell, ‘but’ has a plain schematic meaning: whatever is to the left and right of it are considered to clash in some respect. This schema supports and guides the inferences we are likely to make about the speaker’s attitude to what he is talking about.

Still, schemas need not be abstract as in the case of ‘but’, or purely topological (i.e. structures between positions in space) like in the case of the container-schema. They may very well be skeleton-like, but nevertheless contingent on cultural phenomena. This is evidently the case for Schank and Abelson’s ‘scripts’ (Schank and Abelson 1977), i.e. the abstract representation of a complex event in terms of its constitutive elements, its sub-events, and the order of the latter. Most of us possess a restaurant-script canonically composed of subevents like: 1) Entering the restaurant, finding a seat; 2) waiter brings menu card; 3) ordering; 4) eating; 5) paying. Now importantly, all the sub-events are systematically correlated to each other and to the whole they are part of so that whenever only one of them is mentioned the whole script is activated and the full restaurant scene is evoked. This obviously supports inferences based on minimal linguistic cues:

5a	They got in, ate, left a tip, and went home.
5b	They sat down, read the menu card, ate, and went home.
5c	They sat down, asked the waiter for advice, ate, and went home.

In none of the above sentences is the restaurant explicitly mentioned; in all of them, however, the representation of it is clearly evoked.

The activation of scripts by one or a few of its constituent parts shares an essential property with schemas of the ‘but’ sort: they also specify the speaker’s intentional relation to what s/he is speaking about. Indeed, since any constituent element triggers the full representation, the one which is chosen specifies, willy-nilly, how the speaker distributed his/her attention on the scene: we can infer the speaker’s intentional attitude to the referent object from the way in which s/he activates the scripts. Consider a cousin to Schank and Abelson’s script, namely Charles Fillmore’s ‘frame’ (Fillmore 2006 [1982]): just as with ‘script’, it is defined as a complex semantic whole consisting of systematically correlated concepts so that you cannot understand one of them without understanding the whole structure it is a part of (the difference is just that frames do not imply any intrinsic temporal order of sub-events). Consider the classical ‘commercial frame’, consisting of a ‘buyer’, a ‘seller’, ‘goods’ and ‘money’. Many different verbs can activate such a frame; all of them, however, will profile certain relations, thus endowing them with particular salience relative to the speaker’s intentional focus. Thus, ‘buy’ profiles the buyer-role and the goods-role; ‘sell’ profiles the seller-role and the goods-role; ‘purchase’ profiles the money-role and the goods-role, etc.

Even though an example like the above is simple and does not seem to yield very interesting distinctions, two things should be observed. First, it is not because it is simple that it is trivial: it is a non-trivial fact – systematically explainable in terms of how schemas, scripts and frames are activated – that perspective, point of view and intentional attitude can be linguistically specified. Second, it is indeed easy to give simple, but nevertheless both remarkable and powerful, examples of how even worn-out frames, such as the ‘commercial frame’, can be instantiated so as to produce strong semantic effects. Consider the following dialogue from Cormac McCarthy’s No Country for Old Men. Two policemen talk about the drugs they have found on a crime scene:

He handed the transponder unit to the sheriff.

What am I supposed to do with this?

It’s Maverick County property. Crime scene evidence.

The sheriff shook his head. Dope, he said.

Dope.

They sell that shit to schoolkids.

It’s worse than that.

How’s that?

Schoolkids buy it.

(Cormac McCarthy, No Country for Old Men, p. 194)

The point being, of course, that adults’ selling of illegal substances or otherwise trying to pervert children for the purpose of personal gain is wicked but, alas, well known and, statistically speaking, expectable: it is part of the prototypical adult-frame. Whereas the fact that children actively desire and purchase the very thing which will pervert their nature is not expectable: it is not part of the prototypical child-frame.

A partial conclusion: up till now we have accounted for the role played by image schemas and other types of schemas such as scripts and frame in cognition and language. We have, very roughly, tried to account for their ontology, i.e. what they are: abstract structures which may inhere in representations which often are counterparts to recurrently experienced spatio-temporal structures in the environment. We have developed the psychology of schemas in the sense that we have accounted for the way in which such representations emerge in the (infant’s) mind, through Perceptual Meaning Analysis or through bodily interaction with the world. Next, we accounted for what could be called the phenomenology of schemas, in the sense that we have shown how they structure our experiences, shape our conceptualizations and specify our intentional focus. Finally we have developed elements of a logic of schematic reasoning with particular focus on how schemas support inferences in everyday cognition and linguistic meaning construction.