Non-retinotopic processing

In which coordinate frame are objects coded in the human brain? Intuition may suggest a Euclidean, external-world based coordinate system, such as Lausanne is on latitude 46°30′57″ N and longitude 6°37′58″ E. However, this is for obvious reasons not the case (or do you know Lausanne’s coordinates?). Most models of neuroscience rely on a retinotopic coordinate system because neighboring points in the visual field are mapped onto neighboring photoreceptors in the retina and this retinotopic coding principle is maintained in the early visual areas. However, with every eye and head movement, the retinotopic representation of the world is displaced, necessitating non-retinotopic representations to achieve stability. Without these representations, we would be dizzy as it is the case when we move our heads very quickly. Almost nothing is known about how the human brain transforms retinotopic representations into a non-retinotopic coordinate frame. We have developed two behavioral paradigms to pit retinotopic directly against non-retinotopic processing.

Figure 1

Figure 1: Ternus-Pikler display. (A) In the ‘‘no-flankers’’ condition, two black disks are presented at the same locations separated by an inter-stimulus interval (ISI). A white dot is perceived to be moving up-down in the left disk, and left-right in the right disk. (B) In the group motion condition, a third disk is added either to the left or right of the center disks. When the ISI is sufficiently long, e.g., 100 ms, three disks are perceived to move back and forth as a group. The group-motion creates a non-retinotopic reference frame (see arrows that were not shown in the actual display). Features are integrated according to this non-retinotopic reference frame. A rotating dot is perceived in the center disk which results from the non-retinotopic integration of the retinotopic dot motions of the two center-most disks. The only differences between A and B lie in the outer disks. The percept of rotation is illusory in the sense that, retinotopically, only up-down and left-right motions are present. It is nearly impossible to attend to the retinotopic up-down or left-right motions in the group motion condition. (C) The ‘‘illusory’’ dot motion disappears when the ISI is reduced to 0 ms. Contrary to the group motion condition, the disks now are perceived at four positions: the two center disks at the center positions and the outer disk jumping from the far left to the far right position. As in the no-flankers condition, up-down motion is perceived in the center-left disk and right-left motion in the center-right disk. Figure and Legend adapted from Clarke AM, Repnow M, Öğmen H, Herzog MH (2013).

To this end, we have first modified a demonstration of the Gestaltists Ternus and Pikler, which is shown on the LPSY website and in Figure 1. In the No-Motion condition, motion is perceived according to integration within a retinotopic reference frame. This reference frame and perception strongly change when we added an «uninformative» additional disk alternating either to the left or right. Now, three disks are perceived moving back and forth horizontally. The horizontal motion creates a new, non-retinotopic reference frame, in which a dot rotates within the central disk. However, this rotation is illusory and is the product of the non-retinotopic integration of the up-down and left-right retinotopic motions. It is surprising that the retinotopic motion trajectories are almost invisible (Boi et al., 2009; Lauffs et al., 2018, 2019).

The Ternus-Pikler is, in general, a versatile test of whether a certain paradigm is processed retino- or non-retinotopically. It is like a pizza party. One can just put the stimuli of interest on the disk and pit directly retinotopic against non-retinotopic processing as in the case of the motion example shown above. We found that surprisingly all visual paradigms we tested, except for adaptation, were processed non-retinotopically (Key publications: Boi et al., 2009; Ogmen et al., 2006; Boi et al., 2011; Lauffs et al., 2107).

Neurophysiology and Modelling. At which neural level does non-retinotopic processing occur? We conducted an fMRI study, which showed that the early visual areas encode motion information in a retinotopic fashion, i.e., neurons were activated in a similar fashion for both reference frames (No-Motion and Group Motion condition). The first area that showed non-retinotopic encoding was hMT+ (Thunell et al., 2016a). We propose that the motion of the reference frame is immediately discounted from the motion of the dots at the hMT+ stage. This might be a reason why it is impossible to see the «true» retinotopic motion trajectories. A subsequent EEG study provided further evidence for this hypothesis Thunell et al., 2016b), which we put into a mathematical framework of vector fields (Ogmen et al., 2010; Clarke et al., 2016; Agaoglu et al., 2016). This model can easily be linked with visual short term memory (Ogmen & Herzog, 2016).

In a second paradigm, we expanded a classical metacontrast paradigm as shown in Figure 2. Interestingly, observers cannot consciously perceive the preceding vernier. Still, the offset of the invisible vernier is visible at the non-offset flanking lines. Hence, as with the Ternus-Pikler Display, features are integrated within an object based dynamic reference frame (Key publication: Otto et al., 2006). Within this reference frame vernier offsets are unconsciously integrated in an almost linear fashion (Otto et al., 2006). Changes in motion grouping lead to changes in integration (Otto et al., 2008). These results have provoked us to propose a new theory of consciousness. Invisibility of features does not occur because temporal or spatial limitations of the human brain but because the brain does not «want» to see certain interpretations of the external world. Invisibility is interpretation (Herzog et al., 2014).

Other experiments use anorthoscopic perception and are not further described here.

Figure 2

Figure 2: Sequential metacontrast paradigm. (A) The central vernier was randomly offset to the left or right followed by a sequence of non-offset, flanking lines (stream). Observers were asked to attend to one stream of lines, for example, the leftward stream (indicated by the arrow, not shown in actual display). The task was to discriminate the offset direction perceived in the stream. Responses were assessed with respect to their accordance with the target-offset. Although the central vernier was rendered invisible by sequential metacontrast, observers could discriminate the target-offsets very well at the flanking lines. (B) Performance strongly changes by inserting an anti-offset to the penultimate line in the attended motion stream indicating a combination of the two offsets. (C) Performance, compared to panel A, is almost not affected by an anti-offset presented in the non-attended motion stream. In the schematics, offset elements are highlighted in black (in the experimental display, all elements had the same luminance). From: Otto et al., 2006.

Publications

Reviews
  • Herzog MH, Hermens F, Öğmen H (2014). Invisibility and interpretation. Frontiers in Psychology, 5:975, p1-9.
  • Herzog MH, Otto TU, Öğmen H (2012). The Fate of Visible Features of Invisible Elements. Frontiers in Psychology, 3(119).
  • Öğmen H, Otto TU, Herzog MH (2014). Microstructure of motion correspondence. In: S. Gepshtein and L. Maloney (Eds). The Oxford Handbook of Computational Perceptual Organization.
  • Herzog MH, Otto TU (2009). Feature Inheritance. In: T. Bayne, A. Cleeremans, P. Wilken (Eds.). The Oxford Companion to Consciousness. Oxford University Press.

Ternus-Pikler display
Modelling
Sequential metacontrast
Anorthoscopic perception