Jump to ContentJump to Main Navigation
The Cognitive-Emotional BrainFrom Interactions to Integration$

Luiz Pessoa

Print publication date: 2013

Print ISBN-13: 9780262019569

Published to MIT Press Scholarship Online: May 2014

DOI: 10.7551/mitpress/9780262019569.001.0001

Show Summary Details
Page of

PRINTED FROM MIT PRESS SCHOLARSHIP ONLINE (www.mitpress.universitypressscholarship.com). (c) Copyright The MIT Press, 2022. All Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in MITSO for personal use. Subscriber: null; date: 16 May 2022

Affective Visual Perception

Affective Visual Perception

(p.41) 3 Affective Visual Perception
The Cognitive-Emotional Brain

Luiz Pessoa

The MIT Press

Abstract and Keywords

The evidence reviewed in this chapter suggests that the idea of a subcortical pathway specialized for the processing of emotional visual stimuli as maintained by the “standard hypothesis” is much in need of revision. The chapter describes the multiple waves model, which has several implications for the characterization of amygdala function in the processing of emotional visual information. The amygdala plays significant functions in a wide array of networks. Though the precise contribution of the amygdala in these networks is still unknown, it does not map specifically onto emotion but, instead, corresponds to broader and more abstract dimensions of information processing, including salience, ambiguity, unpredictability, and other aspects of biological value. The chapter also describes the anatomy and physiology of the pulvinar nucleus of the thalamus, and specifies how this structure is important for emotional processing.

Keywords:   low road, subcortical pathway, amygdala, pulvinar, thalamus

In discussing the amygdala’s involvement in multiple functions such as attention, value representation, and decision making, chapter 2 rejected a narrower characterization of this structure as limited to fear processing or the handling of negative information. The present chapter will evaluate a specific aspect of purported amygdala function, one suggested to involve a subcortical-only pathway that rapidly conveys affective information. The evidence related to the subcortical pathway is discussed in detail, given the functional implications of the associated framework.

How does an animal assign biological value to a stimulus? Which stimuli are good and which are bad? Which should be approached and which should be avoided? The early work of Heinrich Klüver and Paul Bucy (1939) implicated temporal lobe structures in the evaluation of affective significance. Indeed, temporal lesions led to what they termed “psychic blindness”: “The ability to recognize and detect the meaning of objects on the basis of visual criteria alone is either lost or seriously disturbed…. Certain properties of the objects, their being ‘dangerous,’ ‘inedible,’ or ‘indifferent,’ have suddenly become ineffective in determining visually guided reactions” (Klüver and Bucy 1939, 609, 612).

Work by Lawrence Weiskrantz (1956) involving more circumscribed lesions implicated the amygdala in the behavioral changes originally reported by Klüver and Bucy (1939).1 Subsequent findings not only pointed to the amygdala in the evaluation of affective significance but also led to a framework that runs as follows: emotional stimuli are processed initially by a dedicated, modular system that operates rapidly, automatically, and largely independently of conscious awareness (Tamietto and de Gelder 2010). This framework, termed the “standard hypothesis” (Pessoa and Adolphs 2010), has two central and interrelated components (Öhman 2005; Tamietto and de Gelder 2010): (1) the purported role of the amygdala in the rapid, automatic, and nonconscious processing of emotional stimuli; and (2) a specific subcortical route of information (p.42)

Affective Visual Perception

Figure 3.1 (plate 2) Visual pathways. (A) Traditional flowchart of visual processing typically emphasizes the lateral geniculate nucleus–V1–V2–V4–TEO–TE pathway, although the scheme is not strictly hierarchical. According to the “standard hypothesis,” a subcortical pathway involving the superior colliculus and the pulvinar nucleus of the thalamus provides fast and automatic access to the amygdala. (B) Alternative flowchart of visual signals via multiple pathways, including “shortcuts.” The “multiple waves” of activation initiate and refine cell responses. For simplicity, feedback pathways, which are extensive, have been omitted. The existence of such feedback pathways dictates, however, that a complex ebb and flow of activation sculpts the neuronal profile both of activation throughout visual cortex and of amygdala responses. Some of the connections between the pulvinar and visual cortex and between the pulvinar and “associational” areas are also indicated. The curved line in the pulvinar schematically separates the medial pulvinar (to the right of the line) from the rest of the structure. FEF, frontal eye field; LGN, lateral geniculate nucleus; MT, medial temporal area (also known as “area V5”); OFC, orbitofrontal cortex; SC, superior colliculus; TE, TEO, inferior temporal areas TE, TEO; VLPFC, ventral-lateral prefrontal cortex; V1, V2, V4, visual areas 1, 2, 4.

Reproduced with permission from Pessoa and Adolphs 2010.

processing—the “low road” (LeDoux 1996)—that bypasses the presumably slower, resource-dependent cortex and that culminates in the amygdala by way of the superior colliculus and the pulvinar nucleus of the thalamus (figure 3.1A; plate 2). The fact that this pathway bypasses cortex is thought to endow the processing of emotion-laden visual stimuli with the properties set forth in the framework’s first component.

Broadly speaking, the concept of a modular subcortical pathway is strongly aligned with one of the most entrenched themes in cognitive neuroscience and cognitive psychology: a dual scheme distinguishing automatic from controlled (p.43) processing (Shiffrin and Schneider 1977). It is also aligned with the idea that emotion and cognition constitute separate mental and neurobiological domains. The standard hypothesis has shaped both basic and clinical research. For example, defects in the modular brain system it proposes are suggested to underlie phobias, mood disorders, and posttraumatic stress syndrome. This chapter provides a critical reexamination of the hypothesis with an emphasis on revising the roles of the brain areas involved.

Standard Hypothesis

The data and theory that underpin the standard hypothesis are not typically articulated in detail, and the central tenets of the hypothesis are often expressed vaguely. The main argument is that, insofar as affectively laden information has survival value, it has driven adaptations in information processing that are reflected in a structurally and functionally modular system (Öhman and Mineka 2001). The standard hypothesis is based not only on empirical data but also on theoretical considerations. In a nutshell, it claims that a system specialized for rapid detection of predators makes sense from an evolutionary perspective.

The purported modularity of the system entails automaticity (Dolan and Vuilleumier 2003): owing to the potency of affective information, this information is processed independently of attention and awareness (see chapter 4 for further discussion). For example, threat-expressing faces are reported to be processed preattentively in visual search paradigms (Öhman, Lundqvist, and Esteves 2001), and fearful facial expressions break into consciousness more quickly than happy ones during continuous flash suppression (a technique to render visual stimuli nonconscious; Yang, Zald, and Blake 2007). Moreover, hemodynamic responses in the amygdala are proposed to occur in response to backward-masked fearful faces (putatively rendering the faces invisible; Morris, Öhman, and Dolan 1998; Whalen et al. 1998) and even in response to unmasked fearful faces in patients with blindsight (Morris et al. 2001; Pegna et al. 2005; see “Blindsight” under “Is Affective Visual Processing Independent of Attention and Awareness?” below).

The standard hypothesis assumes as well that the anatomical components of the system enable emotion processing to occur entirely subcortically (Morris, Öhman, and Dolan 1999). This assumption has its roots in rodent studies demonstrating the existence of a subcortical pathway through the auditory thalamus to the amygdala that is sufficient for some forms of auditory Pavlovian fear conditioning (LeDoux 1996). A similar subcortical route is assumed to exist for visual information in primates, including humans. The (p.44) notion of such a subcortical pathway is appealing because it is assumed to be faster than a cortical one, and processing of affective stimuli is thought to be adaptive in part because it is fast. For instance, judgments of threat can be made from facial stimuli that are displayed as briefly as 39 ms (backward masked; Bar and Neta 2006). Because the route is assumed to be subcortical, processing of visual information along this pathway is assumed to be coarse. Thus coarse (i.e., low–spatial frequency) information from affective stimuli is believed to engage subcortical visual processing, consistent with findings that the amygdala is activated more strongly by emotional faces presented with low than high spatial frequency (Vuilleumier et al. 2003).

This chapter will discuss several shortcomings of the standard hypothesis and will describe an alternative model. The focus will be on visual processing in the primate brain, particularly in the pulvinar nucleus of the thalamus, given its critical connective role in the purported subcortical pathway. The chapter will review the functional properties of affective vision central to the standard hypothesis and highlight the issues of speed and coarseness. It will then briefly discuss other important notions linked to the hypothesis, including the role of attention and the modularity of the brain (these will be covered in detail in chapters 4 and 8, respectively; see also Adolphs 2008; Duncan and Barrett 2007; Lewis 2005; Pessoa 2005, 2008). Finally, it will outline an alternative model—the “multiple waves model”—that assigns a larger function to cortical processing of affective visual information. I suggest that this model can satisfactorily explain the findings used to support the existence of the subcortical route proposed by the standard hypothesis.

Before evaluating the standard hypothesis in greater detail, two general considerations are in order. First, to dissect the flow of visual information in the primate brain is extremely difficult, chiefly because its visual processing is highly distributed, both temporally and spatially (Bullier 2001). Although the majority of retinal ganglion cells project to the lateral geniculate nucleus (LGN) in the thalamus, there are additional projections to the superior colliculus, to the pulvinar, and to several other subcortical nuclei (see figure 3.1B; plate 2). In total, at least ten pathways from the retina have been established (Cowey 2004); thus neurons in visual cortex can receive information via multiple channels. Moreover, the actual visual “drive” into a region (e.g., ascending retinal projections to the LGN) constitutes a small fraction of the total synaptic inputs to that region, most of which reflect extensive intrinsic processing and feedback projections (Douglas and Martin 2004). These facts greatly complicate the accounts of even standard (i.e., nonemotional) visual neuroscience (Masland and Martin 2007; Nassi and Callaway 2009).

(p.45) Second, it is often also difficult to determine precisely all the anatomical and physiological components of visual processing linked to a given brain structure because the information available is derived from different species (e.g., rats and mice, monkeys, humans) and from different methods (e.g., electrophysiology, functional MRI, and lesion studies).

General Functional Issues

Speed of Visual Processing

The electrophysiological responses evoked by visual stimuli can be modulated by emotional content, and this modulation has been reported to occur at short latencies, in some human studies, at around 100 ms after stimulus onset (Pizzagalli, Regard, and Lehmann 1999; Halgren et al. 2000). In addition, the “N170” component of the electroencephalographic (EEG) signal, or the “M170” component in magnetoencephalography (MEG) studies, which is linked to face processing, is modulated by facial expression in some cases (Eger et al. 2003; Japee et al. 2009). In contrast, numerous investigations have shown such effects only at longer latencies, ranging from 200 to 400 ms (e.g., Krolak-Salmon et al. 2004). And some studies, including intracranial ones, have observed emotional modulation only at latencies from 600 to 800 ms and longer (e.g., Brazdil et al. 2009). It is unclear why the timing of emotional modulation varies so widely (from less than 100 ms to more than 800 ms), but one possibility is that the effect is highly context dependent (see below)— which would run counter to some of the main tenets of the standard hypothesis.

Both EEG and MEG are excellent techniques to study the temporal evolution of brain signals, but because signals measured at the level of sensors do not uniquely constrain the neural sources—the “inverse problem”—localizing responses is problematic with either technique. Subcortical neural sources pose additional problems, given their deep origin and noncortical structure. Thus, although signals detected in EEG/MEG studies reveal fast emotional modulation and are used as evidence in favor of fast subcortical processing, because their origin is unclear, the signals might not arise in the proposed subcortical pathway.

Affective modulation of brain responses certainly can be fast. But how does its timing compare to the speed of visual processing in general? One way to assess this is to measure and contrast response latencies across brain areas (Bullier 2001). For example, do responses in the proposed subcortical pathway occur earlier than those in cortical sites? Single-neuron recordings in monkeys show that responses in cortex (even in frontal cortex) are detected with latencies that are within the range of those observed in subcortical areas. Figure 3.2 (p.46)

Affective Visual Perception

Figure 3.2 (plate 3) Response latencies to visual stimulation in macaque cortex. The earliest latencies are remarkably short, and even mean response latencies reveal very fast cortical processing. Areas that became active at the given latency after visual stimulation are shown in red, those activated earlier in yellow and those not yet activated in white. Areas for which no information was available are shown in dark gray. EC, entorhinal cortex; FEF, frontal eye field; FST, fundus of superior temporal cortex; 5, 7a, 7ip, 8a, areas 5, 7a, 7 intraparietal, 8a; IPa, superior temporal area IPa; M1, primary motor cortex; MST, medial superior temporal cortex; MT, medial temporal area (also known as “V5”); OFC, orbitofrontal cortex; PFC, prefrontal cortex; PGa, superior temporal area PGa; PreM, premotor cortex; SEF, supplementary eye field; TAa, anterior subregion of superior temporal area TA; TE1–TE3, inferior temporal areas TE1–TE3; Tem/TEa, medial and anterior subregions of inferior temporal area TE; TPO, superior temporal area TPO; TS, superior temporal sulcus; V1–V4, visual areas 1–4.

Reproduced with permission from Pessoa and Adolphs 2010 and adapted from Lamme and Roelfsema 2000.

(plate 3) illustrates that in macaque cerebral cortex, the earliest latencies are remarkably short, and even mean response latencies demonstrate extremely fast cortical processing (Lamme and Roelfsema 2000). Visual response latencies in the pulvinar are between 60 and 80 ms and overlap with latencies observed in early visual cortex areas V1 and V2 (Ouellette and Casanova 2006). In inferior temporal cortex (i.e., “late” visual cortex), latencies can be as short as 60–85 ms (Lamme and Roelfsema 2000) and, strikingly, in some frontal sites such as the frontal eye field, as short as 40–70 ms. These latencies again overlap with those in area V1 (Nowak and Bullier 1997; Schmolesky et al. 1998). Thus, although mean response latencies increase gradually from posterior to anterior visual cortex areas, there is considerable overlap of response times across the brain, as Jean Bullier (2001) has demonstrated (figure 3.3). In the context of the standard hypothesis, it therefore seems that pulvinar responses are not particularly fast. That said, visual response latencies in the superior colliculus are somewhat faster than those observed in the pulvinar, showing an early, transient response around 40–70 ms that may support rapid eye movements during visual orienting (Boehnke and Munoz 2008; note that these response times overlap with frontal eye field responses, which are involved also in eye movements and attention).

Before discussing responses in the amygdala, it is worth describing a recent study that reported responses to face stimuli in the pulvinar (Nguyen et al. 2013). (p.47)

Affective Visual Perception

Figure 3.3 Latencies of visual responses of neurons in different cortical areas. For each area, the central tick marks the median latency and the extreme ticks the 10 and 90 percentiles. Numbers in parentheses refer to bibliographic references given in Bullier 2003. Considerable overlap in response timing is observed. LIP, lateral intraparietal area; STS, superior temporal sulcus; TPo, temporal-parietaloccipital region. (See caption to figure 3.2 for key to other abbreviations.)

Annotated and reproduced with permission from Bullier 2001.

A subset of the cells studied exhibited short latencies; on average approximately 60 ms after stimulus onset. Faster responses (30–50 ms) were also observed, although these were found when schematic, “face”-like patterns were employed. But because such schematic faces included stimuli with a “mouth” above the “eyes,” the critical stimulus feature that engaged the neurons is unclear (if indeed they were responding to “faceness”).

What are the response latencies of neurons in the amygdala? In the monkey amygdala, latencies to visual stimuli range from 100 to 200 ms (Gothard et al. 2007; Inagaki and Fujita 2011; Kuraoka and Nakamura 2007; Leonard et al. 1985; Nakamura, Mikami, and Kubota 1992), although shorter response latencies to unspecific stimuli (e.g., fixation spots) have been reported (Gothard et al. 2007). Differences in evoked responses between threatening and neutral or appeasing facial expressions range from 120 to 250 ms (Gothard et al. 2007). Intracranial studies in humans have reported that amygdala responses to visual stimuli (Mormann et al. 2008; Oya et al. 2002) and modulation of amydala responses by affective content (Krolak-Salmon et al. 2004; see also (p.48) Oya et al. 2002) both start at around 200 ms, although, intriguingly, response latency varies widely (200–800 ms) across experiments.

In summary, subcortical visual processing is not qualitatively faster than cortical processing. Moreover, the crucial variable is not when the initial stimulus response occurs but when reliable differences between affective and nonaffective stimuli can be detected. According to some proposals (e.g., Tovee and Rolls 1995), most of the information encoded by visual neurons may be available in 100-ms segments of activity (i.e., spiking data within a 100 ms block); a fair amount of information is available in segments of 50 ms, and possibly even in segments of 20–30 ms (note that these segments are taken after the normal latency for neurons to start firing in response to a stimulus). Although these considerations demonstrate the remarkable speed of neuronal computation (at least under some conditions), they add tens of ms to the time that is required to, for example, discriminate between stimuli. In other words, if the response latency in a given region is, say, 120 ms, the time required for differentiating affective and neutral stimuli would be expected to be at least 140–170 ms, and probably longer. A final point is that responses in humans appear to be slower than in monkeys. For example, according to one study (Yoshor et al. 2007), the fastest recording sites in human subjects had response latencies of just under 60 ms and were probably located in area V1 (or possibly area V2), whereas the fastest responses observed in monkey area V1 have latencies shorter than 40 ms (Lamme and Roelfsema 2000). These differences should be taken into account when comparing the speed of processing in monkeys and humans.

A complementary perspective on the speed of visual processing comes from behavioral and electrophysiological studies of scene perception in humans. The evidence suggests that visual processing in general (i.e., including nonaffective information) can be surprisingly fast. Based on a number of tasks involving classification of natural scenes, a substantial amount of information can be gathered from even a single glance at the stimulus. Michelle Greene and Aude Oliva (2009) reported that subjects typically required 19–67 ms to attain 75 percent correct performance on several tasks, including determining a scene’s global property (e.g., “natural scene”) and basic level categorization (e.g., “forest”); performance reached asymptote at around 100 ms of image exposure. These findings build upon earlier work on scene perception (Potter and Levy 1969), as well as “ultrarapid” visual perception. One investigation showed that EEG responses linked to categorizing images displayed for 20 ms developed approximately 150 ms after stimulus presentation (Thorpe, Fize and Marlot 1996; see also Fabre-Thorpe 2011).

(p.49) Advances in our understanding of rapid visual perception also stem from computational modeling work inspired by the organization of the visual system, which has revealed that feedforward architectures can account for the performance of humans in rapid categorization tasks (Serre et al. 2007; Serre, Oliva, and Poggio 2007). Several properties of the computer models match neuronal responses in monkey visual cortex (Hung et al. 2005) and responses in human temporal cortex measured intracranially. In one case, single-trial neuronal data from human visual cortex could predict the category of the object presented with data from the first 100 ms after stimulus onset (Liu et al. 2009)—and prediction performance was robust to depth rotation and scale changes. Together, these findings suggest that the feedforward machinery of the visual system is capable of performing complex computations in very short periods of time (such as generating viewpoint-invariant representations).

Another relevant aspect of the work on rapid visual perception concerns attention. There is evidence that visual perception of nonaffective stimuli is not only fast, but also may require less attention than often thought. For instance, rapid visual categorization of novel natural scenes requires little focal attention (Li et al. 2002), indicating that perception outside the focus of attention may extend beyond simple and salient stimuli (see also Peelen, Fei-Fei, and Kastner 2009). These findings bear on the standard hypothesis because they show that many types of complex visual processing can take place even when resources are scarce (but see Evans and Treisman 2005).

Thus visual cortex processing is both efficient and fast (see also Roland 2010). From a behavioral standpoint, sophisticated object perception takes place with very brief presentations. From a physiological standpoint, although mean response latencies increase gradually from posterior to anterior visual cortex, considerable response-timing overlap is observed. As argued below, the data are consistent with a “multiple waves” model of visual cortex processing in which visual information propagates rapidly via several alternative routes. These considerations suggest that the speed of cortical processing has been considerably underestimated in the literature on emotional perception. Consequently, the argument that cortical mechanisms are too slow and in need of a separate subcortical system to account for the properties of affective perception loses much of its force.

Is Affective Visual Processing Independent of Attention and Awareness?

A central component of the standard hypothesis, automaticity entails that processing of emotion-laden items take place without need for attention and outside of awareness. To anticipate chapter 4, which will review the literature (p.50) on attention in greater depth, though the issue is complex, mounting evidence of the effect of attention on emotional processing poses a considerable challenge to the idea of “strong automaticity” (Pessoa 2005). Like regular, nonemotional perception, visual attention is an important factor in determining the behavioral impact and neural fate of an affective visual stimulus.


A growing literature also suggests that awareness is a significant factor during emotional perception (Adams et al. 2011; Gray et al. 2013; Hsu, Hetrick, and Pessoa 2008; Pessoa 2005; Straube et al. 2010; see also Kang, Blake, and Woodman 2011). Nevertheless, findings concerning emotional “blindsight” are frequently used to argue in favor of nonconscious subcortical processing. Blindsight is a phenomenon in which patients with lesions in primary visual cortex, though lacking conscious visual experience, have residual visual abilities (Poppel, Held, and Frost 1973; Weiskrantz 1986). More precisely, these patients exhibit above-chance performance on detection and discrimination tasks in the absence of phenomenal visual experience. Several case studies of patients with striate cortex lesions have reported residual visual abilities when presented with emotional facial expressions (de Gelder et al. 1999) or other emotional stimuli (Hamm et al. 2003). In some reports, even with the lack of visual awareness, amygdala activation has been observed (Morris et al. 2001; Pegna et al. 2005). Based on these findings, several authors have argued that emotional visual stimuli activate the amygdala via the subcortical pathway—hence the absence of phenomenal visual experience.

Although the findings of emotional blindsight are noteworthy, the inferences that can be drawn from them are limited. First, the studies do not demonstrate that the amygdala is more important for nonconscious than for conscious processing. For example, in one study involving brief presentations and masking (Pessoa et al. 2006), the pattern of amygdala responses arose when the subjects reported seeing something fearful or not, rather than when the fearful face was actually presented. Indeed, amygdala responses during “false alarm” trials (when subjects incorrectly reported seeing a fearful face not presented) were greater than those during “correct reject” trials (when subjects correctly reported not seeing a fearful face not presented). Second, the studies do not demonstrate that the amygdala shows largely normal activation in the absence of cortical input. Third, and critically, these reports do not take into account the many additional alternative routes that bypass early visual cortex (discussed at length below). Thus, in principle, signals can reach the amygdala in several ways that bypass area V1. In sum, to infer the existence of a subcortical pathway from cases of emotional blindsight is unwarranted.

(p.51) Objective and Subjective Measures of Awareness

Although, as stated above, unaware emotional perception has been challenged, reports supporting it continue to be published. A factor that may help explain the discrepancy of published results is the use of different criteria to determine whether or not a subject is aware of perceiving a stimulus. According to subjective criteria, unaware perception occurs when subjects report not having seen target stimuli or report being unable to perform the task better than chance (independent of their actual performance). Subjective criteria hold that only the subjects themselves have access to their inner states and that their introspection is a reliable source of information about conscious experiences (Merikle, Smilek, and Eastwood 2001). According to objective criteria, unaware conditions occur when a subject’s performance in a yes/no or forced-choice task is at chance (e.g., d-prime is zero), such as when subjects fail to detect alternative stimulus states (presence vs. absence of targets). Under such conditions, behavioral effects of unaware stimuli (e.g., faster reaction time for undetected fearful faces), as well as associated physiological or neuroimaging signals, constitute correlates of unaware perception.

In my studies, objective criteria were used in studies reporting that awareness is necessary for the processing of emotional faces (e.g., Pessoa, Japee, & Ungerleider, 2005; Pessoa, Japee, Sturman, & Ungerleider, 2006; Japee et al. 2009). Unfortunately, no consensus exists regarding the “best” approach to measure and characterize awareness. In fact, there is a mounting tension between objective and subjective threshold approaches (e.g., Merikle et al., 2001; Seth et al. 2008; Snodgrass, Bernat, and Shevrin 2004). Historically, an important concern with subjective procedures is that they can be quite sensitive to response bias (Eriksen 1960). For instance, subjects may be reluctant to indicate having seen a stimulus when the available evidence is very weak. Such concern came into sharp focus with the development of signal detection theory (Green and Swets 1966). At the same time, the importance of subjective measures of awareness is that they resonate with the intuitively appealing idea that awareness should be based on introspective reports of individuals’ inner states (James 1890). In fact, from the subjective awareness perspective, at times, the utilization of objective measures may logically preclude the existence of unconscious perception (Bowers 1984).

I have argued that it is important to investigate both objective and subjective measures of perception. Thus, Remigiusz Szczepanowski and I investigated the perception of minimally visible stimuli by utilizing both types of measures (Szczepanowski and Pessoa 2007) within a single task. Thus subjects performed a single detection task for which they were required to make two evaluations: an evaluation of whether or not a fearful face was present and an (p.52) evaluation of their response confidence. The first evaluation corresponds to a yes/no choice during a standard detection task. The second evaluation can be considered to be a discrimination between correct and incorrect responses (by providing “low” and “high” confidence ratings). Whereas the detection task explicitly probed subjects’ perception of fear, confidence ratings provided an indirect assessment of the accessibility of information about fear. Our experiment thus allowed us to test for potential dissociations between these two measures of fear perception, for instance, whether the successful detection of fearful faces could be accompanied by random discrimination of response correctness. All but one subject exhibited above-chance detection of fearful faces for all stimulus durations (including 17- and 25-ms targets). Sensitivity for briefly presented stimuli was demonstrated for subjective perception as well (most often for durations of 25 ms or higher). Therefore, our results showed that sensitivity for briefly presented fearful faces is not limited to detection tasks but that it can be demonstrated for subjective perception as well.

Importantly, our findings revealed a dissociation between the two measures of fear perception such that, for some subjects, the successful detection of fearful faces was accompanied by random discrimination of response correctness. Figure 3.4 illustrates the potential relationship between the critical durations linked with objective and subjective measures. The shorter duration represents the critical duration for reliably detecting a fearful-face target, and the longer duration represents the critical duration for reliably discriminating

Affective Visual Perception

Figure 3.4 Dissociation between objective and subjective perception measures. The dissociation zone refers to stimulus durations for which the subject would be able to reliably detect a fearful-face target but for which the subject would not reliably discriminate between correct and incorrect responses. The arrows around the duration values suggest that, when a dissociation occurs, the values will vary for different subjects. A possible interpretation of this pattern of results is that a subject would be subjectively unaware but objectively aware of the stimulus, though other interpretations are possible.

Adapted with permission from Szczepanowski and Pessoa 2007.

(p.53) sociation zone” in which the subject is above the objective threshold but below the subjective one (for further discussion, see Szczepanowski and Pessoa 2007; see also Sweeny et al. 2013 and Winkielman and Schooler 2011).

Amygdala as Necessary

At the foundation of the standard hypothesis is the notion that the amygdala is necessary for critical properties of affective processing, including automaticity and nonconscious mechanisms. In this respect, a well-known study by Adam Anderson and Elizabeth Phelps (2001) compared performance on the attentional blink task in patients with amygdala lesions and in controls. Their findings supported a necessary role for the amygdala: the counteracting of the attentional blink by words with emotional content was not observed in patients with lesions of the left amygdala. Recent studies reviewed in chapter 2, however, have questioned the necessary status of the amygdala for rapid detection, search, or nonconscious processing of affective visual stimuli. The results of these studies strengthen the idea that the amygdala is not a necessary component of several of the functions traditionally attributed to the “low road” and that other mechanisms must therefore be invoked to explain the results, at least when the amygdala is damaged.

Processing of Affective Visual Stimuli Involves Both Coarse and Fine Information

According to the standard hypothesis, the subcortical pathway is particularly effective at carrying coarse information, mainly because the superior colliculus and pulvinar are assumed to convey little detailed information. This general notion was inspired by findings in rodents that simple (coarse) auditory conditioning does not require cortex, whereas conditioning that demands more complex stimulus discriminations does (LeDoux 1996; see also below).

Results from human neuroimaging studies appear to be compatible with this notion. Coarse and fine visual information are referred to as “low– and high– spatial frequency” (low– and high–SF) information, respectively, when understood in terms of frequency analysis (also called “Fourier analysis”). For example, in one study (Vuilleumier et al. 2003), amygdala responses were stronger when subjects viewed low– versus high–SF fearful faces, and when they viewed low–SF fearful versus neutral faces. Furthermore, activations in brain areas consistent with the locations of the superior colliculus and pulvinar were greater in response to low–SF fearful faces than to low–SF neutral faces. Although findings of this kind have sometimes been interpreted to indicate that the structure is relatively blind to high–spatial frequency information, the amygdala receives major projections from the anterior inferior temporal cortex (p.54) (Amaral et al. 1992) that convey highly processed object information. In fact, the amygdala receives highly processed cortical input from all sensory modalities except olfaction (Amaral et al. 1992). For example, in monkeys, functional MRI has shown that electrical microstimulation of a patch of temporal cortex strongly responsive to faces activates the lateral nucleus of the amygdala (Moeller, Freiwald, and Tsao 2008). Notably, electrophysiological studies have revealed that the monkey amygdala contains neurons tuned to the identity of specific faces (Gothard et al. 2007; Rolls 2005)—a property that requires high–SF information (figure 3.5). The human amygdala also displays categoryspecific responses, including responses specific to faces (Kreiman, Koch, and Fried 2000a; Mormann et al. 2008). Intriguingly, the right amygdala responds selectively to animals as well (Mormann et al. 2011).

Regarding the responses of the human amygdala to face stimuli, one study (Rutishauser et al. 2011) found that more than 50 percent of all amygdala neurons responded to normal images of faces, whereas only 10 percent responded to digitally scrambled images. A substantial proportion of the neurons showed responses selective for whole faces versus parts of faces, indicating that amygdala neurons encode holistic information about faces, rather than just about their constituent features (neurons also responded to facial features, though less vigorously). Also of relevance, most of the neurons that generated their strongest response to whole faces did not distinguish between fearful and happy expressions and may therefore be signaling something closer to stimulus relevance.

In a monkey electrophysiology study, Mikio Inagaki and Ichiro Fujita (2011) explicitly investigated spatial-frequency tuning by amygdala neurons. Neurons in anterior inferior temporal cortex showed distinct tuning characteristics compared to those in the amygdala—they were less dependent on the physical size of the stimulus, hence more invariant to changes. Although this finding is compatible with the existence of distinct routes to these two structures, and possibly with the existence of a subcortical pathway to the amygdala (as suggested by the authors), overall, amygdala neurons exhibited a broad range of response characteristics that overlapped considerably with those in anterior visual cortex.

Behavioral studies also clarify the type of visual information needed during emotional perception. They have shown that the discrimination of facial expressions relies on both low– and high–spatial frequency information (e.g., Smith et al. 2005). The perception of fear is particularly reliant on high–spatial frequency information (Smith and Schyns 2009). Indeed, in their study of a patient with bilateral amygdala lesions, Ralph Adolphs and colleagues (2005) found that the patient’s impaired recognition of facial expressions of fear was (p.55)

Affective Visual Perception

Figure 3.5 Face-identity responses in the monkey amygdala. Each row of images contains three facial expressions displayed by the same monkey. Below each image is the peristimulus time histogram (in 20 ms bins) and single-trial spike rasters of a neuron that responded with a tenfold increase in firing rate to the faces of the two monkeys in rows A and B. Note that the responses were relatively unchanged across facial expression. The “lipsmack” expression is one of appeasement.

Reproduced with permission from Gothard et al. 2007.

(p.56) due to impaired processing of the eye region of faces, and especially to impaired processing of high–SF information about the eyes (see figure 2.8). These results demonstrate the importance of high–spatial frequency information in fear recognition and indicate that the amygdala is required for this type of visual processing.

Although some of the findings reviewed above are consistent with the notion that subcortical areas process coarse visual information, when taken all together, they clearly establish that the perception of emotional expressions involves both coarse and fine information and that the amygdala not only receives but also uses both kinds of information to facilitate recognition of facial expressions.

Physiological and Anatomical Issues

The pulvinar is the key “link element” in the purported subcortical route (figure 3.1A; plate 2). In reviewing physiological and anatomical data on the pulvinar pertinent to the standard hypothesis, this section will first discuss data that bear on whether this structure is better characterized as a relatively passive way station or as a dynamic element of brain circuitry, then review evidence regarding the existence of a subcortical pathway via the pulvinar, and finally touch on properties of auditory and visual processing, given that one of the motivations for proposing the subcortical pathway is based on auditory conditioning data.


Pulvinar Input

The pulvinar complex, as this set of related nuclei is sometimes called, is the largest nuclear mass in the primate thalamus and thought to have expanded in size as it evolved in primates (Chalfin et al. 2007; Grieve, Acuña, and Cudeiro 2000). In terms of connectivity explicitly relevant to visual processing, the pulvinar receives direct visual input from the retina, indirect visual input via the superficial layers of the superior colliculus, and massive input from striate and extrastriate visual cortex (figure 3.6; plate 4). All of these projections terminate in the inferior pulvinar (see “Pulvinar Anatomy” below for a more detailed description).

Before proceeding to the pulvinar itself, we need to clarify the organization of the superior colliculus (SC), a laminated midbrain structure that acts as one of the centers organizing eye-gaze movements (May 2006; Wurtz and Albano 1980). Primarily visual sensory in nature, the superficial layers of the SC receive direct retinal input and project to the deeper layers, which are both (p.57)

Affective Visual Perception

Figure 3.6 (plate 4) Schematic diagram of pulvinar connectivity. Most pulvinar nuclei and subnuclei (including those not shown here) are involved in thalamo-cortical loops that target different cortical territories. The inferior nucleus is reciprocally connected to striate and extrastriate cortex, the lateral nucleus is connected to association cortex in temporal and parietal lobes (as well as extrastriate cortex), and the medial nucleus is connected to higher-order association cortex in parietal, frontal, orbital (not shown), cingulate and insular regions (not shown), in addition to the amygdala. Thus the medial nucleus is not only connected with the amygdala but is also part of multiple thalamo-cortical loops (note, however, that the connection to the amygdala does not seem to be bidirectional). The superior colliculus is a layered structure whose superficial layers are visual and project to the inferior nucleus. Its intermediate and deeper layers are multimodal and involved in motor preparation, including for eye movements, and project to the medial nucleus (not shown). Inf, inferior; IT, inferior temporal cortex; Lat, lateral; Med, medial; MT, medial temporal area (also known as “V5”).

Reproduced with permission from Pessoa and Adolphs 2010, adapted from Stepniewska 2004.

multimodal and motor, receiving input from somatosensory and auditory sources as well as from the basal ganglia and cerebellum. Sensory, association, and motor areas of cerebral cortex provide another major source of collicular input, particularly in more encephalized species, where visual sensory cortex projects to superficial layers while the frontal eye field targets the deeper layers. The deeper layers themselves project to brainstem structures containing gaze-related burst neurons, as well as to the spinal cord and medullary reticular formation regions that produce head turning (see “Role of the Superior Colliculus” under “Multiple Waves Model” below).

(p.58) Intriguingly, visual response properties of pulvinar cells do not reflect those of cells in the superior colliculus, and the precise contribution that input from the SC makes to pulvinar responses remains uncertain (Stepniewska 2004). Collicular lesions have little effect on electrophysiological responses of pulvinar cells, whereas striate cortex lesions abolish responses in the inferior pulvinar (Bender 1983). Likewise, collicular and pulvinar lesions result in different behavioral impairments (Robinson and Cowie 1997; see also the blindsight study Tamietto et al. 2010 for evidence of a dissociation of superior colliculus and pulvinar roles in visual processing). These findings support the idea that the pulvinar may be better characterized as participating in cortical networks than as relaying visual information from the superior colliculus, as does the finding that, unlike “driving inputs” to the lateral geniculate nucleus, those to the pulvinar originate in the cortex, whereas subcortical inputs are typically modulatory (Guillery 1995; Sherman and Guillery 1996).2

Pulvinar Function

Studies in monkeys and humans with pulvinar lesions suggest that the pulvinar is involved in determining what is salient in a visual scene (Ungerleider and Christensen 1979; Zihl and von Cramon 1979). Consistent with this, the response of pulvinar neurons to visual stimuli is increased when attention is paid to the stimulus or when it has behavioral relevance. For instance, primate pulvinar neurons respond more vigorously to behaviorally relevant targets than to unattended items (Robinson and Cowie 1997). In one monkey study, as many as 92 percent of cells exhibited attenuated responses when stimuli were task irrelevant (passively viewed) versus task relevant (Benevento and Port 1995). Furthermore, the impact of attention on evoked responses in the monkey pulvinar is spatially specific, such that a pulvinar neuron only increases activity when the animal attends to a stimulus that falls within the cell’s receptive field (Petersen, Robinson, and Keys 1985). Finally, the pulvinar seems to be critical (as shown by pharmacological inactivation) when a distractor stimulus needs to be “filtered out” (Desimone et al. 1990). Thus it has been proposed that the pulvinar is involved in attention and distractor filtering (Desimone et al. 1990).

Both neuroimaging and lesion studies in humans corroborate the notion that the pulvinar participates in attention. Although early positron emission tomography (PET) studies are compatible with a role of the pulvinar in visual attention (LaBerge and Buchsbaum 1990; Corbetta et al. 1991)—including attentional filtering in the presence of distractors and selective attention to stimulus features, such as shape and color—the low spatial resolution of PET precludes anatomically clear conclusions. More decisive evidence stems from studies at higher resolution using functional MRI. For example, responses in (p.59) the pulvinar were only observed when the stimulus was attended, but not when it was unattended (Kastner et al. 2004). In addition, lesion studies have described deficits of attention in the visual field contralateral to the pulvinar lesion (Arend et al. 2008).

The pulvinar is important for visual awareness, too. Thus pulvinar lesion studies have uncovered feature-binding deficits (Karnath, Himmelbach, and Rorden 2002; Ward et al. 2002; Zihl and von Cramon 1979); the pulvinar on the right hemisphere was identified as a subcortical node associated with spatial neglect in humans (Karnath, Himmelbach, and Rorden 2002; Mesulam 1981).3 Monkey physiology and human functional MRI studies have also revealed contributions of the pulvinar to visual awareness. During a visual illusion that induced the intermittent perceptual suppression of a bright luminance patch (Wilke, Mueller, and Leopold 2009), monkey pulvinar neurons showed changes in spiking rate in response to trial-by-trial stimulus visibility, suggesting that they reflected visual awareness. Similarly, a functional MRI study found that the human pulvinar responded trial by trial not to the affective significance of visual stimuli (positive vs. negative conditioned stimulus) per se, but to affective stimuli that were consciously perceived (Padmala, Lim, and Pessoa 2010; figure 3.7A). In another human functional MRI study (Pessoa

Affective Visual Perception

Figure 3.7 Pulvinar and amygdala during processing of affective stimuli. (A) Logistic regression analysis of evoked responses in the left pulvinar as a function of affective significance for a sample subject during an attentional blink task (see figure 2.5A). The slope of the logistic fit indicates the strength of the predictive effect. For clarity, only binned data for the conditioned stimulus (CS+) condition are included. The inset shows mean logistic fit slopes across subjects. (B) The medial pulvinar is proposed to amplify evoked responses of behaviorally relevant stimuli via circuits involving cingulate cortex, orbitofrontal cortex (OFC), and the amygdala, all regions important for stimulus valuation.

Reproduced with permission from Pessoa and Adolphs 2010 and adapted from Padmala, Lim, and Pessoa 2010.

(p.60) and Ungerleider 2004a), pulvinar responses were observed during “false alarm” trials (i.e., where a stimulus change was reported but did not actually occur) but not during “miss” trials (i.e., where a stimulus change occurred but went unnoticed by the subject). Together, the above results do not support the notion that the pulvinar is principally involved during nonconscious processing, and are therefore inconsistent with the passive relay role of the standard hypothesis. As stated before, “driving inputs” (as opposed to “modulatory inputs”) to the pulvinar appear to originate in cortex (Guillery 1995; Sherman and Guillery 1996; see also below). Thus pulvinar responses may be closely aligned with awareness because of the contributions from cortex, which is thought to be important for conscious perception (e.g., Kouider and Dehaene 2007; Alkire, Hudetz, and Tononi 2008).

Pulvinar Anatomy

An implicit assumption of the standard hypothesis is that the pulvinar, whatever its functions are, is basically a single structure—or at least that the part of it that receives collicular inputs is the same as the one that projects to the amygdala. But is the pulvinar in fact organized as a simple relay station that conveys signals from the superior colliculus to the amygdala? A review of pulvinar anatomy shows it is not. (Readers less interested in the details of the neuroanatomy may want to skip ahead to this section’s summary paragraph.)

In primates, the pulvinar is a set of thalamic nuclei that accounts for a quarter of the total thalamic mass (Grieve, Acuña, and Cudeiro 2000). Originally partitioned into three subdivisions, namely, inferior, lateral, and medial (Walker 1938), it is now typically divided into four, with anterior pulvinar representing the fourth subdivision. Broadly speaking, the inferior pulvinar is reciprocally connected to striate and extrastriate cortex; the lateral pulvinar is connected to association cortex in temporal and parietal lobes (though parts also receive extrastriate cortical inputs); and the medial pulvinar is connected to parietal, frontal, orbital, cingulate, and insular cortex, in addition to the amygdala (Grieve, Acuña, and Cudeiro 2000; Shipp 2003; figure 3.6; plate 4).

Several anatomical features highlight the extensive bidirectional connectivity between the pulvinar and cortex. For example, all twenty to thirty known visual areas connect with the pulvinar, sometimes in a relatively topographic fashion (Shipp 2003; Stepniewska 2004), and, as stated, parietal, frontal, orbital, cingulate, and insular cortex are all connected with the pulvinar as well. At a gross level, it is as if the entire convoluted cortex were “shrinkwrapped” around the pulvinar (Shipp 2003). Based on connectivity data, it has been suggested that the pulvinar may contain two “domains” (Grieve, Acuña, and Cudeiro 2000; Shipp 2003; see figure 3.6; plate 4). Densely connected (p.61) with visual cortex, including the V1–V4 and MT (middle temporal) areas (Shipp 2003), the “ventral domain” therefore has a strong visual component— indeed, it could be called the “visual pulvinar”—and its projections to the dorsal visual stream may mediate some of the visual abilities in people with blindsight (Lyon, Nassi, and Callaway 2010; Berman and Wurtz 2010). The “dorsal domain” has connections with cross-modal association cortex, including temporal and parietal areas, such as area 7A and the lateral intraparietal area, that participate in attention (Shipp 2003). The dorsal domain receives highly processed visual input from anterior parts of ventral visual cortex (Shipp 2003). And because it is also connected with cingulate cortex, frontal cortex (including orbitofrontal cortex), insula, and amygdala (Shipp 2003; see below), it has remarkable potential to integrate information from very diverse brain regions (figure 3.6; plate 4). Whereas the connectivity of the pulvinar’s ventral domain is restricted to the occipito-temporal cortex, sites in the dorsal domain may be connected with relatively distal parts of the brain, such as parietal and frontal cortex (Shipp 2003). Indeed, many extensive fronto-parietal cortical connections are mirrored by overlapping fields in the dorsal domain (Romanski et al. 1997; Barbas, Henion, and Dermon 1991): where regions in frontal and parietal cortex are interconnected in the cortex, their projection sites in the pulvinar typically coincide (and the connections are bidirectional between pulvinar and cortex)—an organization that further exemplifies the integration ability of the pulvinar’s dorsal domain.

In all, the ventral and dorsal domains of the pulvinar have very distinctive connectivity patterns: the ventral domain is strongly visual, whereas the dorsal domain is associational. Of relevance to the standard hypothesis, visual signals from the retina are conveyed to the superficial layers of the superior colliculus and then to the inferior pulvinar, which belongs to the ventral domain. Connections from the pulvinar to the amygdala, however, originate in the medial pulvinar, which is part of the dorsal domain. All of which serves to challenge the notion that the pulvinar links the superior colliculus and the amygdala in a straightforward way.

Because an indirect pathway might still convey signals to the amygdala— although via a longer (hence slower) route, we need to consider connectivity within the pulvinar. There is no good evidence to support the existence of connections from the inferior to the medial pulvinar, which would be required to provide a contiguous pathway from the superior colliculus to the amygdala. Like other thalamic nuclei, the primate pulvinar does not appear to have longrange intrinsic connections, either excitatory or inhibitory (Imura and Rockland 2006), although inhibitory interneurons and local processing may exist within the medial pulvinar (see also Ma et al. 1998).

(p.62) As a final complication, connections also seem to exist from the intermediate/deep (nonretinal) layers of the superior colliculus to the medial pulvinar (Romanski et al. 1997). Could the intermediate or deep layers be sending direct input to the amygdala via the medial pulvinar? Such a scenario is problematic for the following reasons. Although, for the sake of simplicity, we have treated the medial pulvinar as a unit, it consists of central/lateral and medial subunits with substantially different connectional patterns (see figure 10 of Romanski et al. 1997). Whereas the medial subunit projects to the amygdala, it is the central/lateral subunit that receives significant input from the nonretinal superior colliculus. Thus, again, it seems unlikely that a simple colliculo-pulvinoamygdalar pathway exists.4 Even more critically, because signals from the intermediate and deep layers of the superior colliculus are multimodal and possibly linked to saccadic eye movements, they would not easily fit the role commonly assigned them in the subcortical pathway of the standard hypothesis.

Next to nothing is known about the connectivity and electrophysiology of the pulvinar in humans, although a diffusion tensor imaging study has reported results consistent with connectivity to ipsilateral superior colliculus as well as temporal visual cortex (Leh, Chakravarty, and Ptito 2008; see also Tamietto et al. 2012 and the conclusion to this book for further discussion).5 A study in epileptic patients examined evoked responses to electrical stimulation to map the functional connectivity of the medial pulvinar and suggested functional connections between medial pulvinar and visual cortex (including sites in occipital and temporal cortex) and between medial pulvinar and the amygdaloid region (Rosenberg et al. 2009).6 Given the limited resolution and other characteristics of both diffusion tensor imaging and electrical stimulation, we cannot infer the existence of anatomical connectivity with any certainty, although it is generally assumed that the human and monkey pulvinar have much the same connectivity.

To summarize, the pulvinar has significant visual and integrative properties. Studies have characterized several ways in which the pulvinar is modulated by attention and awareness; indeed, the pulvinar is likely to be an important “control site” for attentional mechanisms more broadly (Shipp 2004). Anatomically, the pulvinar is a heterogeneous structure that may be viewed as containing at least two “domains” with very distinct connectivity patterns. Furthermore, the connectivity of the pulvinar is such that it is regarded as a higher-order thalamic structure that may be involved in cortico-thalamocortical communication (see below), not a simple first-order relay (Sherman 2007). Collectively, these observed characteristics of the pulvinar run contrary to the standard hypothesis, which assumes that largely automatic processing is mediated by a rapid subcortical pathway relying on a passive pulvinar.

(p.63) Does the Subcortical Pathway Exist in Primates?

The early inspiration and impetus for the standard hypothesis came from work with rodents that first gave rise to the idea of both subcortical (“low road”) and cortical (“high road”) pathways for processing fear-relevant information (LeDoux 1996, 2000). Work on fear conditioning showed that there are direct projections to the amygdala from the auditory thalamus (i.e., medial geniculate thalamus) in the rat (Campeau and Davis 1995; Romanski and LeDoux 1992). In the rat, there is some evidence that the lateral-posterior nucleus of the thalamus—whose properties are related to those of the pulvinar in primates (see Chalfin et al. 2007 for evolutionary considerations)—conveys visual information to the amygdala (Shi and Davis 2001; for related findings, see also Linke et al. 1999). Based on a series of lesion manipulations, Changjun Shi and Michael Davis (2001) have argued, however, that this pathway is unlikely to be functional in intact animals. Instead, a pathway connecting the lateral-posterior thalamus to the amygdala via anterior temporal cortex appears to be critical for transmitting emotion-laden visual information (figure 3.8).

Affective Visual Perception

Figure 3.8 Visual pathways in the rat. Schematic diagram of thalamo-cortico-amygdala and thalamoamygdala visual pathways involved in fear-potentiated startle in the rat. The pathway indicated by the dashed line may not be critical in normal visual fear conditioning. The lateral posterior nucleus of the thalamus in rats is thought to be related to the pulvinar nucleus in primates. BLA, basolateral nucleus of the amygdala; Ce, central nucleus of the amygdala; CRN, cochlea root neurons; LGD, dorsal-lateral geniculate nucleus; LP, lateral-posterior nucleus of the thalamus; PR, perirhinal cortex; PnC, pontine reticular nucleus, caudal part; SC, superior colliculus; TE2, temporal area TE2; V1–V2, visual areas 1–2.

Reproduced with permission from Shi and Davis 2001.

(p.64) More generally, the work of Shi and Davis (2001) highlights the need to assess whether specific brain structures are operational during normal vision, namely, in intact animals. In other words, a lesion may identify a pathway that is capable of mediating a behavior (e.g., fear conditioning) but one that might not be functional in intact animals: when the normal route is damaged by lesions, other pathways not typically engaged may take over, supporting the behavior in question.7

Another study involving “rewiring” suggests that visual and auditory pathways are also organized differently in rodents (Newton et al. 2004). Mice acquired a conditioned fear response rapidly (in terms of the number of trials) to auditory cues but slowly to visual cues. The authors proposed that the difference in behavior was due to distinct connectivity patterns in the visual and auditory modalities—direct projections to the lateral amygdala from the auditory thalamus but indirect ones from the visual thalamus. To test this hypothesis, the authors induced the growth of retinal projections to the medial geniculate nucleus (figure 3.9), which participates in auditory processing in normal animals. Acquisition of visually cued conditioned fear was accelerated in the rewired mice, where visual stimuli induced activity in the “auditory” thalamus and the lateral amygdala, much as auditory stimuli did in control mice. Together, their data suggest that the rewired pathway conveyed visual information and mediated “rapid” activity-dependent plasticity in structures influencing learned behavior.

Affective Visual Perception

Figure 3.9 Simplified fear conditioning pathways in normal and rewired mice. (Left) Schematic of the principal visual (black) and auditory (gray) conditioned fear pathways in normal mice. (Right) Schematic of the rewired visual (black) cued conditioned pathway. The inferior colliculus (IC) was lesioned bilaterally (dashed box) in neonatal mice to induce retinal projections to the medial geniculate nucleus. LGN, lateral geniculate nucleus; MGN, medial geniculate nucleus; TE2, temporal area TE2.

Reproduced with permission from Newton et al. 2004.

(p.65) What about evidence of subcortical connectivity in monkeys? Here, as we have seen, anatomical studies have shown connections between the superficial superior colliculus and the inferior pulvinar, both of which can be considered “visual” structures (Grieve, Acuña, and Cudeiro 2000; Stepniewska 2004), and between the pulvinar and the amygdala (Jones and Burton 1976; Romanski et al. 1997). But, whereas the inferior pulvinar is extensively interconnected with visual cortex (consistent with visual functions), the pulvinar’s projection to the amygdala originates in the medial pulvinar (Jones and Burton 1976; Romanski et al. 1997; see also Aggleton, Burton, and Passingham 1980). And, again as we have seen, because there is no good evidence of connectivity between the inferior and the medial pulvinar, a subcortical pathway from superior colliculus to pulvinar to amygdala seems unlikely.

Working with the tree shrew, Martha Bickford, Jonathan Day-Brown, and colleagues have documented a potential subcortical pathway that is in line with the standard hypothesis (Day-Brown et al. 2010). Placed in its own order (Scandentia), separate from insectivores and primates, this squirrel-like mammal is nonetheless considered a “basal primate” (Butler and Hodos 2005). Bickford and colleagues reported projections from the superior colliculus to the “Pd” nucleus of the pulvinar (part of the medial pulvinar), which in turn connects with the lateral amygdala (Day-Brown et al. 2010). Although this study provides perhaps the most compelling evidence for a subcortical visual pathway, the diffuse nature of the observed superior colliculus–to-pulvinar connectivity makes it unlikely that such a pathway could effectively convey visual form information. In related work, Ranida Chomsung, Heywood Petry, and Martha Bickford (2008) speculated that the primary signal conveyed from the superior colliculus to the pulvinar in the tree shrew may encode motion (see also Luksch, Khanbabaie, and Wessel 2004 for discussion of how collicular responses mediate sensitivity to motion independently of stimulus details). It is also worth noting that the tree shrew has surprisingly robust visual capabilities in the complete absence of primary visual cortex (Diamond and Hall 1969; Snyder, Killackey, and Diamond 1969). Therefore, even though subcortical pathways involving the superior colliculus may support several aspects of tree shrew vision, the anatomical findings of these studies would hardly seem to apply to humans, who experience blindness with lesions in primary cortex.

To summarize, except for the report on the tree shrew, there is scant evidence for a direct subcortical pathway conveying visual information to the amygdala in primates. It is thus unclear how findings from auditory fear conditioning studies in rodents can be applied to visual processing of affective stimuli in primates (see also next section). At the same time, work with rodents suggests that basic forms of vision-based fear conditioning may be mediated by a route (p.66) linking the thalamus to anterior temporal cortex. This type of cortical bypass connection is proposed below to rapidly convey affective information across the brain.

Subcortical Auditory Processing in Rats Is Qualitatively Different from Subcortical Visual Processing in Primates

The standard hypothesis derived a considerable portion of its historical motivation from the organization of the auditory system in rodents. But the auditory and visual systems differ in important ways. The temporal precision of the auditory system is substantially greater than that of the visual system. In contrast to vision, audition is omnidirectional, such that information from all directions in space can be sampled (though at relatively lower spatial resolution). Furthermore, the functional anatomy of the auditory system is very different from that of the visual system. Properties such as sensitivity to sound frequency, duration, amplitude, pitch, and binaural disparity, which are observed in primary auditory cortex (area A1), are already observed at subcortical levels. In fact, primary auditory cortex should not be thought of simply as primary visual cortex transplanted into the auditory modality since it seems to perform high-level functions. Indeed, given that several subcortical stages exist below the level of the primary auditory cortex, the inferior colliculus, which is involved in audition, occupies a processing level similar to that of area V1 in vision, and area A1 is more analogous to visual areas in inferior temporal cortex than to V1 (cell responses in inferior temporal cortex are considerably more elaborate than those in V1; King and Nelken 2009).

These considerations suggest that a subcortical pathway for auditory input to the amygdala in rodents would be quite distinct from the purported subcortical visual pathway in primates. Connections from the auditory thalamus to the amygdala, while bypassing cortex, still convey relatively processed information to the amygdala, in contrast to the suggested primate visual counterpart. Visual responses in the superior colliculus are quite rudimentary. Indeed, Peter Schiller and Fritz Koerner (1971, 924) described them as “event” and “jerk” detectors, noting that “none of the units we studied were shape or orientation specific. Similar responses could be elicited using squares, triangles, bars, or any of numerous other configurations.”

Multiple Waves Model

The standard hypothesis has influenced both basic and applied research and, at first glance, has intuitive appeal. Emotional reactions can be fast and relatively impervious to top-down effects when task demands are not high. Nevertheless, a host of problems plague the hypothesis in its basic form, chief (p.67) among them (1) visual processing of complex scenes in general (i.e., nonemotional scenes) is already surprisingly fast and not appreciably slower than affective visual mechanisms; (2) affective processing is not strongly independent of attention and awareness; (3) coarse visual processing (e.g., low–spatial frequency faces) does not map cleanly onto subcortical processing; (4) new findings show that complete lesions of the amygdala in humans spare rapid and nonconscious aspects of affective processing; and (5) the existence of a subcortical visual pathway linking the superior colliculus, pulvinar, and amygdala receives scant support from anatomical data.

Faced with these problems, the standard hypothesis can no longer be said to hold, although several of its themes and original motivation remain pertinent. We turn now to the multiple waves model Ralph Adolphs and I have proposed as an alternative to it (Pessoa and Adolphs 2010).

Multiple Visual Pathways and Coarse Information Processing

One of the primary motivations for the standard hypothesis is the perceived need for rapid processing: fast—though coarse—visual mechanisms are just what an organism needs to survive in a dangerous environment. As reviewed next, visual routes other than a colliculo-pulvino-amygdalar one are proposed to carry out this role (for further discussion of additional pathways in vision, see Catani et al. 2003; Chen et al. 2007; Cowey 2004; and, in particular, Vuilleumier 2005, which anticipates some of the themes elaborated below).

Although visual processing along the ventral stream, which is crucial for object recognition, has historically been described as occurring in a relatively hierarchical fashion, significant “shortcut” connections link areas V1 to V4 (Nakamura et al. 1993), V2 to TEO (Nakamura et al. 1993), and V4 to TE (Felleman and Van Essen 1991), providing the means for faster information transmission to anterior temporal cortex (Lamme and Roelfsema 2000; figure 3.1B; plate 2). Direct connections between the lateral geniculate nucleus and extrastriate regions, including areas V2 (Bullier and Kennedy 1983; Yukie and Iwai 1981) and V4 (Yukie and Iwai 1981), have also been reported.

Functional evidence for bypass systems has been observed in several studies. For example, combining electrophysiology and functional MRI in lesioned monkeys, Michael Schmid and colleagues (2009) detected robust visual activation in areas V2 and V3 in animals with lesions of area V1, demonstrating that routes bypassing V1 are sufficiently potent to drive extrastriate visual responses. In a second such study (Schmid et al. 2010), they detected widespread extrastriate activation in the absence of area V1 and observed responses in several visual areas, including areas V2, V4, and MT/V5, as well as parietal cortex. Notably, animals with V1 lesions were able to perform a (p.68) visual detection task when stimulus contrast was high. Further reversible deactivation of the visual thalamus (lateral geniculate nucleus) showed that successful behavior in this task was dependent on this structure. These findings are particularly significant because they demonstrate the importance of the visual thalamus for some types of blindsight, which it can be argued the monkeys exhibited.

In humans, functional evidence for the role of cortical bypass systems was reported in a case study of a patient with developmental agnosia and prosopagnosia (intriguingly, with no discernible macroscopic cortical lesion; Gilaie-Dotan et al. 2009).8 In this patient, whereas area V1 was robustly activated by visual stimuli (as measured with functional MRI), intermediate areas V2–V4 were not activated, although robust downstream activation was observed in the parahippocampal gyrus and other anterior regions, with spared visual selectivity. As in normal individuals, the patient’s parahippocampal gyrus responded robustly to outdoor scenes, including pictures of houses and “places” (Epstein and Kanwisher 1998; see also Boyer, Harrison, and Ro 2005 for evidence of a V1-bypassing pathway in humans).

In the past few years, researchers have demonstrated several other visual pathways that either bypass the lateral geniculate nucleus or involve the LGN but bypass early visual cortex. For example, Robert Wurtz and colleagues (2011) have described multiple LGN–bypassing visual routes in primates, including one from the superior colliculus to the frontal eye field through the medial dorsal thalamus, and two from the superior colliculus to the middle temporal area (area MT, which is strongly sensitive to motion) in visual cortex via the pulvinar. These pathways take part in multiple aspects of “active vision,” including saccade- and attention-related mechanisms. Other primate studies have also revealed LGN–to–MT connections that can convey visual signals directly to MT (Sincich et al. 2004; see also Bridge et al. 2008 for related evidence in humans). Indeed, interest in the contributions of multiple routes in generating visual response properties is producing a more nuanced understanding of visual processing (see, for example, Ponce, Lomber, and Born 2008; Ponce et al. 2011; Passarelli et al. 2011). Intriguingly, multiple pathways also might be involved with more sophisticated behaviors that rely on visual processing, such as reading (Richardson et al. 2011).

Figure 3.10 (plate 5) provides one view of the connectivity between visual areas, emphasizing the projections from subcortical regions (including the lateral geniculate nucleus, superior colliculus, and pulvinar) to cortical regions (Capalbo, Postma, and Goebel 2008).9 Derived by taking structural data into account and determining the “best-fit” connectivity based on cell response timing, the figure’s diagram highlights the property that signals from subcortical (p.69)

Affective Visual Perception

Figure 3.10 (plate 5) Visual cortex connectivity model. Connectivity was determined by taking structural data into account and estimating the “best-fit” pathways based on cell response timing. Including subcortical regions and their connections (red arrows) provided the best model. FEF, frontal eye field; MST, middle superior temporal area; MT, middle temporal area; SCA, subcortical areas; V1–V4, visual areas 1–4.

Reproduced with permission from Capalbo, Postma, and Goebel 2008.

areas can be rapidly disseminated across occipital and temporal visual cortex (in addition to frontal cortex). Notably, a model that includes subcortical regions and their connections (red arrows) provides a better account of response timing than do models that contain only cortico-cortical shortcuts, whereas assuming a purely hierarchical structure of the visual system fails to provide a good fit to existing latency data (Capalbo Postma, and Goebel 2008).

Long-range shortcuts also exist, such as projections from visual areas TEO/TE and the superior temporal sulcus that link regions in anterior ventral visual cortex with ventral-lateral and orbital prefrontal cortex (Rempel-Clower and Barbas 2000; Saleem, Kondo, and Price 2008). In orbital PFC, the projections are indeed quite widespread (Saleem, Kondo, and Price 2008). Some bypass connections involve magnocellular projections, known to convey low–spatial frequency and motion information at relatively short latencies, to cortical regions in prefrontal (and middle temporal) cortex.10 For example, frontal eye field (and MT) cells exhibit latencies 10–40 ms shorter than cells in areas V2 and V4 (Schmolesky et al. 1998). Additional long-range pathways connect regions as early as area V2 with prefrontal cortex (Barbas 1995; Rempel-Clower and Barbas 2000). Jean Bullier (2001) has suggested that low–spatial frequency information may rapidly reach parietal and frontal cortex from early (p.70) visual cortex, thereby providing coarse information about the gist of a visual scene and supporting object recognition (see also Bar 2003; Fabre-Thorpe 2011; Kveraga, Boshyan, and Bar 2007). For instance, Bullier (2001, 106) proposes “an integrated model that uses asynchronous transfer of information in the geniculo-cortical connection and the rapid activation of the dorsal stream by the M[agnocellular] channel to generate a first-pass analysis of the visual scene.” It is conceivable that these “first-pass” distributed volleys of activation are less susceptible to manipulations of attention and awareness (Barrett and Bar 2009; Bullier 2001; see chapter 4 for further discussion).

In summary, there are multiple parallel routes for visual information processing that lead to substantial temporal dispersion of evoked responses (see figure 3.3) and that enable “high-level” regions to respond with surprisingly short latencies (Nowak and Bullier 1997). Each processing stage adds approximately 10 ms to the latency (Nowak and Bullier 1997). The cost of using such bypassing stages may be that, at first, only relatively coarse information is available about a visual item. This is consistent with a coarse-to-fine processing strategy in which the more global content of a stimulus is processed earlier than finer details (Sugase et al. 1999; Sripati and Olson 2009).

Based on the considerations above, I propose that the initial processing of visual information proceeds simultaneously along parallel subcortical and cortical channels, creating “multiple waves” of activation across visual cortex and beyond (see Rudrauf et al. 2008). In this manner, visual stimuli that have affective and motivational significance are capable of engaging multiple brain sites—including the amygdala, orbitofrontal cortex, anterior insula, and anterior cingulate cortex—that can gauge their impact and further redirect resources toward behaviorally relevant items. Thus rapid processing of affective information is possible even in the absence of a specialized subcortical pathway (figure 3.1B; plate 2) or a single specific structure such as the amygdala. It should be emphasized, however, that fast visual processing is still very powerful. For example, information about a visual item is available in short segments of spiking data (30–100 ms; Tovee and Rolls 1995), and stimulus category can be predicted from human intracortical recordings within 100 ms (Liu et al. 2009).

In light of this, I suggest that affective blindsight involves some of the alternate pathways described here. A study of a patient with complete cortical blindness used advanced source modeling to investigate the time course of information processing (Andino et al. 2009). All facial expressions, including neutral ones, evoked relatively short latency responses (70–120 ms) localized to the superior temporal sulcus; emotion-specific responses that were localized to anterior temporal cortex and possibly amygdala occurred considerably later (p.71) (120 and 200 ms, respectively). Whereas this study (Andino et al. 2009) suffers from the localization problems alluded to before, its findings are consistent with the proposal that affective significance is computed in parallel along several circuits (see also Rotshtein et al. 2010 for a related proposal). More conclusively, studies like the one by Schmid and colleagues (2010) described above document instances during which “nonstandard” routes of the type described here do support blindsight.

Although this section has emphasized the role of multiple pathways during rapid affective perception, they also operate during less challenging situations, such as those involving longer stimulus durations. Findings from a recent study by Danai Dima and colleagues (2011), who investigated the processing of facial affect when subjects viewed angry, fearful, or sad expressions, each presented for 2 seconds, are interesting in this regard. In their analysis, Dima and colleagues employed dynamic causal modeling (see Friston, Harrison, and Penny 2003), a technique that attempts to estimate the strength of directed interactions between neural systems based on functional MRI data. Although results obtained with this technique need to be considered with caution (see Ramsey et al. 2010), their findings indicated that valence signals were communicated simultaneously across parallel channels. Notably, amygdala signals were not sufficient to explain valence-related interactions between visual cortex and frontal cortex, consistent with the existence of separate sources of valence modulation. Given the change of focus from a single specialized subcortical route to a multiple pathways model, it is important to reconsider the roles of both the pulvinar and the amygdala during processing of emotional visual stimuli.

Role of the Pulvinar in Processing Emotional Visual Stimuli

First, it is worth considering the functions of the pulvinar in general. Despite several decades of work, reviews of the pulvinar often note that “surprisingly little is known about its functions” and frequently refer to this structure as “enigmatic” (Grieve, Acuña, and Cudeiro 2000; Stepniewska 2004). Yet a recurring theme is that the pulvinar is involved in attention and in determining behavioral relevance. Based on the data described here, the proposed multiple waves model suggests that the pulvinar helps to coordinate and regulate the flow of multimodal information via a series of thalamo-cortical loops that highlight signals related to behavioral significance (see figure 3.6; plate 4). Notably, the model takes into account that most of the input to the pulvinar comes from cortex.

In the context of emotional processing, the medial nucleus of the pulvinar is probably the most relevant one, given that it connects not only with the (p.72) amygdala, but also with a large array of other brain regions. The model therefore suggests that the medial nucleus is involved in more general functions that impact emotional processes, such as determining the behavioral relevance of a stimulus. For example, it is connected with parietal areas that are engaged in attention, with orbitofrontal and cingulate cortex, which are important for computing an object’s biological value, and with the insula, which has a role in emotional feelings. These pathways are all bidirectional, providing ample opportunities for the medial nucleus to modulate and regulate information flow. According to the multiple waves model, the importance of the pulvinar in emotion is due not to its status as a subcortical “labeled line” conveying emotional information to the amygdala but rather to its pattern of connectivity with subcortical and cortical sites that have a role in determining the biological significance of a stimulus.

Studies by Robert Ward and colleagues have investigated the effect of pulvinar lesions on processing affective visual information in humans. A complete unilateral loss of the pulvinar led to a severe deficit in a patient’s ability to recognize fearful facial expressions shown in the contralesional visual field (Ward et al. 2007). According to the multiple waves model proposed here, when weak or brief visual stimuli have biological significance, cortico-pulvinocortical circuits coordinate and amplify signals in a manner that enhances their behavioral impact. This model is also compatible with impairment in recognizing anger (and possibly happiness; Ward et al. 2007). Of note, the essential pulvinar damage was found in the medial pulvinar, the region that in monkeys projects to the amygdala. The proposed model is consistent, as well, with the finding that viewing complex unpleasant images impaired performance in a subsequent simple (neutral) visual task in controls, but not in a patient with pulvinar damage (Ward, Danziger, and Bamford 2005); according to the model, the unpleasant stimulus did not garner additional resources in the patient (which would have interfered with performance, as it did in the controls).

Pulvinar involvement in processing affective information does not seem to reflect emotional content per se, however. In a human functional MRI study, my colleagues and I (Padmala, Lim, and Pessoa 2010) found that, in trials that contrasted affectively significant (CS+) and neutral (CS−) conditions, there was a significant relationship between the magnitude of evoked responses in the pulvinar and the probability of correctly detecting a target on a trial-by-trial basis during the affective but not during the neutral condition (figure 3.7A). These results reveal an emotion-visibility interaction that may characterize the role of the pulvinar more generally. In other words, the pulvinar amplifies responses to stimuli of potential value to the animal (such as those signaling the possibility of shock in the experiment; figure 3.7B).

(p.73) Cortico-Thalamo-Cortical Communication

Despite progress, our current understanding of pulvinar function is largely incomplete. In an interesting proposal by S. Murray Sherman (2007), the pulvinar participates in regulating cortical communication, with direct cortical connections between two areas supplemented by an indirect pathway coursing through the pulvinar or other higher-order thalamic nuclei (figure 3.11). Data consistent with this proposal were reported in rat somatosensory cortex, where activity was found to be driven by a cortico-thalamo-cortical pathway (Theyel, Llano, and Sherman 2010). Additional results of a monkey physiology study lend further support to the proposal. By recording simultaneously in the pulvinar and cortical visual areas V4 and TEO (the latter in inferior temporal cortex), Yuri Saalmann and colleagues (2012) obtained evidence that maintaining attention in the absence of visual stimulation (delay period after a cue stimulus disappeared) depended on pulvino-cortical interactions. In contrast, direct cortico-cortical influences during this delay period were weak (though strong when the cue was shown). It is particularly intriguing that the relative contribution of the pulvinar on cortico-cortical interactions was largest during the delay interval. At this juncture in the trial, cortical signals would presumably benefit the most from the support of the pulvinar. To conclude, as stated by Brian Theyel, Daniel Llano, and S. Murray Sherman (2010, 87), “corticothalamocortical information transfer may represent an important addition to, or even replacement of, the current

Affective Visual Perception

Figure 3.11 Conventional and alternative views of thalamo-cortical circuits. In the conventional view, cortical communication is accomplished via pathways between cortical sites. In the alternative view, as proposed by Sherman and colleagues, higher-order thalamic nuclei play a prominent role in this communication, and direct cortico-cortical pathways may be less important. FO, first order; HO, higher order.

Reproduced with permission from Sherman 2007.

(p.74) dogma that corticocortical transfer of primary information exclusively involves direct corticocortical pathways.”

Role of the Amygdala in Processing Emotional Visual Stimuli

Bypass connectivity data in visual cortex suggest that responses in the amygdala should be, at times, quite fast. Although fast responses are possible, the amygdala receives signals from anterior portions of ventral visual cortex, providing it with inputs that have potentially undergone considerable elaboration. In other words, in many cases the amygdala operates on signals that have received extensive processing, leading to responses that are highly selective (e.g., Mormann et al. 2008). Given the multiple pathways that reach the amygdala, it is not surprising that response latencies should span a considerable range, even within a single paradigm. Furthermore, neurons in the amygdala should exhibit a broad range of response characteristics, as observed in the study by Inagaki and Fujita (2011).

What roles does the amygdala play in the processing of affective visual stimuli? The structure’s connectivity pattern provides some clues. The predominant source of visual input to the amygdala, specifically to the basolateral amygdala, comes from higher-order visual cortex in the anterior temporal lobe (Amaral et al. 1992). This suggests that the amygdala is a convergence zone for highly processed sensory information. In addition, there are loops between visual cortex and the lateral amygdala, and this feedback is thought to modulate visual responses (Vuilleumier et al. 2004; see chapters 2 and 7). Further integrative functions of the amygdala stem from its extensive connections with much of cortex. In addition to its well-recognized connections with medial and orbital territories of prefrontal cortex, the amygdala is also connected to lateral prefrontal cortex, albeit in a weaker manner (Ghashghaei, Hilgetag, and Barbas 2007).What is more, the architecture of prefrontal cortex is such that, on average, inputs from the amygdala reach approximately 90 percent of PFC after a single connection within frontal cortex (Averbeck and Seo 2008). Finally, the amygdala seems to be part of a “core brain circuit” (Modha and Singh 2010) that is topologically central in terms of global brain connectivity and whose functions probably include aggregation and distribution of information (issues dealt with at greater length in chapter 9).

In light of the foregoing considerations, the amygdala’s contribution to processing of affective visual information arises not from a subcortical source of visual input, but rather from the structure’s broad connectivity with cortex and with other subcortical structures. Given this connectivity, the impact of the amygdala on behavior can be mediated through many routes, for instance, via both visual and prefrontal cortex, a possibility consistent with findings of (p.75) our study combining the attentional blink task with fear conditioning (Lim, Padmala, and Pessoa 2009; discussed in chapter 1. For emotion-laden stimuli, trial-by-trial fluctuations in evoked responses in the amygdala predicted whether a target would be detected. Furthermore, the amygdala’s influence on behavior was mediated by both visual and prefrontal cortex (as suggested by statistical path analysis). I propose that during the handling of affectively significant items, the amygdala enhances sensory processing through both direct (amygdala-to–visual cortex) and indirect (amygdala-to–prefrontal cortex–to–visual cortex) pathways (see chapter 7 for further discussion).

Prosopagnosia and Capgras Syndrome

Some patients suffering from prosopagnosia have a deficit in face perception but exhibit skin conductance responses when they see familiar faces (Bauer 1984; Tranel and Damasio 1985). A prediction of the multiple waves model is that this effect is mediated via routes that bypass regions in temporal cortex whose lesions compromise face perception. Intriguingly, Capgras syndrome (Capgras and Reboul-Lachaux 1923; Ellis and Young 1990) poses a somewhat reverse condition to prosopagnosia. In some instances, patients have altered familiarity of persons close to them, such as their parents—these are deemed to be “impostors” who look exactly like the real persons. In one study (Hirstein and Ramachandran 1997), in contrast to normal persons, a patient’s skin conductance responses to photographs of familiar people, including his parents, were not larger in magnitude than his responses to photographs of unfamiliar people. Capgras syndrome has been suggested to involve damage to the inferior temporal cortex (Capgras patients also have face processing impairments; Young et al. 1993), like other cases of visual hypoemotionality, where patients exhibit deficits of visually evoked emotions with preserved emotional responses to nonvisual stimuli (Bauer 1982; Habib 1986; Sierra et al. 2002). The link between Capgras syndrome and visual cortical damage poses problems for the subcortical pathway of the standard hypothesis. Presumably, a subcortical route would be able to carry signals leading to enhanced skin conductance responses even in Capgras syndrome patients. I am not aware that such responses have ever been observed.

Role of the Superior Colliculus in Processing Emotional Visual Stimuli

Although the focus in this chapter has been on the pulvinar’s role as the key “link element” in the purported subcortical pathway of the standard hypothesis, some considerations regarding the superior colliculus are in order here. The superior colliculus is a layered structure at the “roof” of the midbrain. It is called the optic tectum (meaning “roof”) in nonmammals and its circuitry (p.76) and function are believed to be conserved phylogenetically (Butler and Hodos 2005). Indeed, the colliculus is considered to be an ancient visual system that is found throughout vertebrate orders, including ray-finned fishes. It is generally described as having six (or seven) layers, the top two (or three) of which exhibit visual properties and constitute the “visual colliculus.” These superficial layers are the recipient of direct retinal input, and response latencies to visual stimuli there are quite short (40–70 ms). Response properties change considerably as one moves down to the deep layers. They become visual-motor and discharge in close temporal relation to saccadic eye movements (Wurtz and Albano 1980). In addition, responses to auditory and tactile stimulation are observed, and many cells are multimodal (Stein and Meredith 1993).

Not surprisingly, the connectivity of the superficial and deep colliculus is quite different. What is surprising, though, is how different they are. The superficial layers receive inputs from the retina, primary visual cortex and surrounding areas, as well as the frontal eye field. The deep layers receive fibers from these regions, in addition to inputs from frontal, parietal, and temporal cortex, as well as from regions in the basal ganglia.

Here, the deep colliculus is of interest insofar as it has been implicated in several defensive behaviors. A series of experiments by Peter Redgrave, Paul Dean, and colleagues in the 1980s led them to suggest that, in rodents, the superior colliculus generates avoidance or escape movements directed away from stimuli that signal emergency or danger (see Dean, Redgrave, and Westby 1989). For example, a rapidly approaching (looming) stimulus can trigger defensive responding that relies on the superior colliculus. The colliculus has a considerable number of descending projections to several structures in the midbrain and pons that are involved in these behaviors, allowing the deep layers to rapidly engender defensive responses.

One particularly interesting midbrain structure is the periaqueductal gray (also called central gray), which surrounds the cerebral aqueduct (a duct containing cerebrospinal fluid). This structure is involved in the integration of behavioral responses to threatening or aversive stimuli. Whereas specific sectors of the periaqueductal gray mediate active coping strategies (such as confrontation or flight), others mediate more passive coping strategies in the face of an inescapable stressful encounter (Bandler and Shipley 1994). Intriguingly, the deep layers of the superior colliculus are adjacent to the periaqueductal gray, leading to the suggestion that they may in fact be the “same” structure that happens to be split into two by coursing fiber tracts (Holstege 1991). Regardless of the exact relationship between the deep collicular layers and the periaqueductal gray, they are bidirectionally connected to one another, affording effective communication between them.

(p.77) The work discussed in the previous paragraphs was done in the rat (see also Brandão et al. 1994; Schenberg et al. 2005). The circuitry may be similar to that in other mammals and possibly primates—for instance, the descending projections of the superior colliculus appear to be similar in the rat and primate (Huerta and Harting 1984). In primates, cortex is essential for form vision, though basic stimulus features, including looming, may also engage the colliculus and lead to rapid downstream activation of defensive behaviors. Until recently, however, no evidence of the involvement of the primate superior colliculus in defense-like behaviors had been reported and colliculus-related defense mechanisms were assumed to be absent in primates. To the surprise of the authors, in a recent study, activation of the deep collicular layers in monkeys evoked cowering, escape-like responses, high-pitch vocalizations, and attack of objects (DesJardin et al. 2013). These initial findings, if confirmed, would reveal that the deep layers of the superior colliculus participate in defensive behavior in primates, too.

In this section, the discussion thus far has centered on the “nonvisual” deep layers of the colliculus. Interestingly, in rodents, there is evidence that the “visual” superficial layers project to the substantia nigra (which is part of the basal ganglia) and are critical for the transmission of short-latency visual information to this structure (Comoli et al., 2003). In rodents, Redgrave (in Smith et al. 2011, 16105) has suggested that “an unpredicted and biologically significant behavioral event causes a short-latency response in the superior colliculus, which is relayed to the basal ganglia.” However, the ability of this circuit to support fast affective responses remains unknown. But because the superior colliculus supports only very crude form vision, the circuit would not be capable of rapidly distinguishing between different shapes, thus severely limiting the types of responses that it can generate. To conclude, the superior colliculus is a heterogeneous structure with a superficial portion that is mostly visual and a deeper part that receives visual, auditory, and somatosensory signals from cortex, as well as inputs from the frontal eye field. The deeper colliculus is thus substantially multimodal and associational. Here, as in the case of the pulvinar, we find again that the part of the structure that is associational is the one that has the most relevance to emotional processing.

Direct Subcortical Inputs to Visual Cortex

More than twenty-five years ago, Jürgen and Margarete Tigges (1985, 353) reviewed findings regarding subcortical sources of pathways to visual cortex: “The inputs provided by these heterogeneous subcortical projections undoubtedly contribute to the complexity of intracortical processing of visual information. It will be very exciting to learn from future experiments how these many (p.78) and diversified subcortical projections influence the mode of operation of cortical columns which appear to be such a fundamental aspect of cortical organization.” They listed more than twenty structures with direct connections, including the locus coeruleus, raphe nucleus, reticular formation, lateral hypothalamus, basal forebrain, and claustrum, in addition to the amygdala and thalamic nuclei (including the lateral geniculate nucleus and the pulvinar).

Two such pathways to visual cortex, the first from the locus coeruleus (discussed further in chapter 9) and the second from the raphe nucleus, seem to hold special promise. Neurons in the locus coeruleus, which is located in the lower part of the brainstem (pons), respond to a variety of sensory stimuli, including visual ones (Aston-Jones and Cohen 2005; Berridge and Waterhouse 2003). The locus coeruleus may act as an integrative center for external sensory events and internal vegetative requirements. This integration, in turn, can potentially influence responses in visual cortex (Tigges and Tigges 1985). Because the (serotonergic) fibers from the raphe nucleus, also located in the brainstem, form impressive and profuse arborizations in layer IV-C of primary visual cortex, the major target of fibers from the visual thalamus, they are in a position to modulate very early visual processing. After nearly three decades, however, our understanding of the functions carried out by these and other subcortical connections to visual cortex remains rather limited.


The evidence reviewed in this chapter suggests that the idea of a subcortical pathway specialized for the processing of emotional visual stimuli as maintained by the standard hypothesis is much in need of revision. The multiple waves model proposed by Ralph Adolphs and me (Pessoa and Adolphs 2010) effects such a revision, with several implications for the characterization of amygdala function in the processing of emotional visual information, as outlined next. The amygdala plays significant functions in a wide array of networks. Though the precise contribution of the amygdala in these networks is still unknown, it does not map specifically onto emotion but, instead, corresponds to broader and more abstract dimensions of information processing, including salience, ambiguity, unpredictability (Whalen 1998; Sander, Grafman, and Zalla 2003; Adolphs 2008; Hsu et al. 2005), and other aspects of biological value.

Indeed, as I argued in chapter 2, the amygdala promotes selective information processing and thus plays an important attentional role. It serves to allocate resources to stimuli, at least in part by influencing (through its connectivity) the anatomical components required to prioritize particular features of information (p.79) processing in a given situation. Such a function would come into play most prominently for affectively significant stimuli. Notably, the amygdala may not be unique in this respect since there are other, largely parallel, networks with architectures that enable related functions, too—such as the network comprising cortex and the pulvinar. The role of the amygdala in the proposed multiple waves model is consistent with a large body of findings and can in fact accommodate several views of amygdala function (Aggleton 1992, 2000; Whalen and Phelps 2009). It is also well understood that the amygdala contributes to several aspects of emotional expression and mobilization of bodily resources (via the central nucleus). Among others, this is an important distinction between the roles of the amygdala and pulvinar during processing of affective visual stimuli.

The multiple waves model also stresses the speed and temporal dispersion of cortical processing, rendering moot the assumed need for a fast subcortical route. Many visual properties can be established very rapidly by the initial wave of cortical response. This implies that there is ample time for feedback to influence ongoing signals. Consequently, unraveling the flow of visual information within cortex and subcortex presents formidable difficulties, difficulties we must overcome if we are to advance knowledge of affective processing. Ultimately, the fate of a biologically relevant stimulus should not be understood in terms of a “low road” versus a “high road,” but in terms of the “multiple roads” that lead to the expression of observed behaviors.

There is an enormous amount of literature implicating the amygdala in affective dysfunction in nearly all psychiatric illnesses, most notably, mood disorders. In this respect, the proposed multiple waves model suggests that, rather than focusing on neurons within the amygdala, we should focus on connections within cortex and between cortex and subcortical structures such as the amygdala. In large part, the substrate of brain function is to be found not so much within neurons as within networks—a theme that will be developed in detail in chapter 8.


(1) . It is noteworthy that the “circumscribed lesions” in Weiskrantz 1956 were in fact fairly broad and included parts of the temporal pole. Thus “even within the medial temporal region, there is no certainty that damage to the amygdaloid complex, as such, is essential” (Weiskrantz 1956, 389). However, given subsequent work on the amygdala, Weiskrantz’s findings have been typically associated with this structure.

(2) . In neuron-to-neuron communication, researchers at times differentiate between “driving inputs” and “modulatory inputs.” As suggested by Sherman and Guillery (1998, 7121): “The former carry the message, defining the essential patterns of activity, whereas the latter can alter the effectiveness of the drive without contributing significantly to the general pattern of the (p.80) message.” In general, this distinction is not clear cut, however, and often difficult to establish. For further discussion, see also Markov and Kennedy (2013).

(3) . “Neglect” is a common and disabling condition following brain damage in which patients fail to be aware of items to one side of space. For example, individuals with right-sided brain damage often fail to be aware of objects to their left.

(4) . I thank Alan Anticevic for discussions of this point.

(5) . Diffusion tensor imaging is an MRI technique that capitalizes on the diffusion of water molecules along fiber tracks to estimate anatomical connectivity between brain regions.

(6) . Two regions are functionally connected if their responses are correlated. See chapter 8 for further discussion.

(7) . Based on these considerations, Shi and Davis (2001) argued that even the auditory subcortical pathway to the amygdala might not typically drive affective information in intact animals that have undergone fear conditioning.

(8) . “Agnosia” refers to the difficulty or inability to recognize visual stimuli, such as animals or man-made objects, whereas “prosopagnosia” refers to the difficulty or inability to recognize faces.

(9) . Note that areas along the ventral surface were not included in the diagram in figure 3.10 (plate 5) but are also connected with the pulvinar, including areas TE and TEO (Webster, Bachevalier, and Ungerleider 1993).

(10) . The “magnocellular system” (or “magno channel”) is particularly sensitive to moving stimuli and exhibits faster cell responses.