We have previously shown that people are more sensitive at detecting asynchrony between a self-generated movement and delayed visual feedback when the perspective of the movement matches the ‘natural view’ suggesting an internal, visual, canonical body representation (Hoover and Harris, 2011). Is there a similar variation in sensitivity for parts of the body that cannot be seen in a first-person perspective? To test this, participants made movements with their hands and head (viewing their face or the back of their head) under four viewing conditions: (1) the natural (or direct) view, (2) mirror-reversed, (3) inverted, and (4) inverted and mirror-reversed. Participants indicated which of two periods (one with a minimum delay, the other with an added delay of 33–264 ms) was delayed and their sensitivity to delay was calculated. A significant linear trend was found when comparing sensitivity to detect cross-modal asynchrony in the ‘natural’ or ‘direct’ view condition across body parts; where sensitivity was greatest when viewing body parts seen most often (hands), intermediary for viewing body parts that are seen only indirectly (moving head while viewing face), and least for viewing body parts that are never seen at all (moving head while viewing back of the head). Further, dependency on viewpoint was most evident for body parts that are seen most often or indirectly, but not for body parts that are never seen. Results are discussed in terms of a visual representation of the body.
Aesthetic preferences for different shapes have been investigated in vision but not in touch. Here the visual and tactile pleasantness of 3-dimensional shapes has been studied in two experiments. In Experiment 1, the participants evaluated the tactile pleasantness of two shapes (a sphere and a cube), manipulated with right, left or both hands. The results revealed that the sphere was preferred over the cube in all the exploration conditions. Moreover, the cube was preferred more when explored bimanually than when explored unimanually. In Experiment 2, the participants evaluated pleasantness and angularity of 11 shapes, presented under visual and haptic exploration conditions. Curved shapes (e.g., cylinder) evoked more pleasant ratings compared to angular shapes in both sensorial conditions. Interestingly, certain shapes (e.g., triangle, rhombus) were preferred significantly more when visually presented than when haptically explored. These results provide some preliminary insights into the cognitive and neural differences in the processing of the aesthetic qualities of visual and tactile stimuli.
Multisensory experiences influence subsequent memory performance and brain responses. Studies have thus far concentrated on semantically congruent pairings, leaving unresolved the influence of stimulus pairing and memory sub-types. Here, we paired images with unique, meaningless sounds during a continuous recognition task to determine if purely episodic, single-trial multisensory experiences can incidentally impact subsequent visual object discrimination. Psychophysics and electrical neuroimaging analyses of visual evoked potentials (VEPs) compared responses to repeated images either paired or not with a meaningless sound during initial encounters. Recognition accuracy was significantly impaired for images initially presented as multisensory pairs and could not be explained in terms of differential attention or transfer of effects from encoding to retrieval. VEP modulations occurred at 100-130 ms and 270-310 ms and stemmed from topographic differences indicative of network configuration changes within the brain. Distributed source estimations localized the earlier effect to regions of the right posterior temporal gyrus (STG) and the later effect to regions of the middle temporal gyrus (MTG). Responses in these regions were stronger for images previously encountered as multisensory pairs. Only the later effect correlated with performance such that greater MTG activity in response to repeated visual stimuli was linked with greater performance decrements. The present findings suggest that brain networks involved in this discrimination may critically depend on whether multisensory events facilitate or impair later visual memory performance. More generally, the data support models whereby effects of multisensory interactions persist to incidentally affect subsequent behavior as well as visual processing during its initial stages.
We examined the effects of the rubber hand illusion on representing tactile stimuli using the Simon effect. In a tactile Simon effect task, participants are instructed to make intensity judgments (using foot pedal responses) to tactile stimuli presented to the hands. Participants are faster when the tactile stimulus and response foot are on the same versus opposite side of the body, regardless of whether the limbs are crossed or uncrossed. Furthermore, participants are faster overall when the hands are crossed versus uncrossed. In this study, participants engaged in a tactile Simon effect experiment with rubber hands positioned directly above the participants’ hidden hands, and with real and rubber hands stroked before each experimental block. Each participant was tested in four blocks, manipulating real and rubber hand position (crossed or uncrossed). First, we found that participants responded faster with real or rubber hands crossed, demonstrating that crossing the hands (real or rubber) can hasten tactile intensity judgments. Furthermore, on trials when the rubber hands were crossed, high ownership ratings for the rubber hand were significantly correlated with faster reaction times. Finally, we found a significantly more robust Simon effect when the rubber hands (but not real hands) were crossed. We discuss these findings with reference to how integration of rubber hands into the body schema influences how we represent the location of tactile stimuli.
Functional networks are comprised of neuronal ensembles bound through synchronization across multiple intrinsic oscillatory frequencies. Various coupled interactions between brain oscillators have been described (e.g., phase–amplitude coupling), but with little evidence that these interactions actually influence perceptual sensitivity. Here, electroencephalographic recordings were made during a sustained-attention task to demonstrate that cross-frequency coupling, driven by cross-sensory cuing, has significant consequences for perceptual outcomes (i.e., whether participants detect a near-threshold visual target). Our results reveal that phase-detection relationships at higher frequencies are entirely dependent on the phase of lower frequencies, such that higher frequencies alternate between periods when their phase is strongly predictive of visual-target detection and periods when their phase has no influence whatsoever. These data thus bridge the crucial gap between complex oscillatory phenomena and perceptual outcomes. Accounting for cross-frequency coupling between lower (i.e., delta and theta) and higher frequencies (e.g., beta and gamma), we show that visual-target detection fluctuates dramatically as a function of pre-stimulus phase, with performance swings of as much as 80%.
A recent study (Tsakiris et al. , 2011) suggested that lower interoceptive sensitivity, as assessed by heat-rate estimation, predicts malleability of body representations, as measured by proprioceptive drift and ownership in a rubber hand illusion (RHI) task. The authors suggested that one explanation of their finding is linked to the notion of limited attentional resources: individuals with high interoceptive sensitivity are more aware of internal states and, in turns, they have less attentional resources available for multisensory processing. If this is the case, the competition between interoceptive and multisensory processing should be strongest when they are concurrent. Here we tested this prediction using a visuo-proprioceptive conflict produced through prismatic goggles, without affecting body ownership (unlike the RHI). In three experiments, participants looked at their own hand while wearing neutral or prismatic goggles (visual field shifted 20° leftwards). Meanwhile, they performed a concurrent counting tasks on interoceptive (Exp. 1–2: heart-beats; Exp. 3: breaths) or exteroceptive signals (pure-tones). A no-task condition was also included. We measured proprioceptive drift in each condition an indicator of illusion strength. All experiments documented a significant drift of perceived hand position after prism exposure. This bodily illusion, however, was never affected by the concurrent task, regardless of whether it involved interoceptive or exteroceptive signals. These result reveal that multisensory integration underlying body perception is unaffected by concurrent tasks capturing attentional resources, strongly suggesting a low-level and automatic phenomenon. Furthermore, they indicate that the origin of increased body malleability in individuals with low interoceptive awareness is not competition for attentional resources.
We report two experiments aiming to define how experience and stimulus enactment affect multisensory temporal integration for ecologically-valid stimuli. In both experiments, a number of different dance steps were used as audiovisual displays at a range of stimulus onset asynchronies using the method of constant stimuli. Participants were either professional dancers or non-dancers. In Experiment 1, using a simultaneity judgment (SJ) task, we aimed at defining — for the first time — the temporal window of integration (TWI) for dancers and non-dancers and the role of experience in SJ performance. Preliminary results showed that dancers had smaller TWI in comparison to non-dancers for all stimuli tested, with higher complexity (participant rated) dance steps requiring larger auditory leads for both participant groups. In Experiment 2, we adapted a more embodied point of view by examining how enactment of the stimulus modulates the TWIs. Participants were presented with simple audiovisual dance steps that could be synchronous or asynchronous and were asked to synchronize with the audiovisual display by actually performing the step indicated. A motion capture system recorded their performance at a millisecond level of accuracy. Based on the optimal integration hypothesis, we are currently looking at the data in terms of which modality will be dominant, considering that dance is a spatially (visual) and temporally (audio) coordinated action. Any corrective adjustments, accelerations–decelerations, hesitations will be interpreted as indicators of the perception of ambiguity in comparison to their performance at the synchronous condition, thus, for the first time, an implicit SJ response will be measured.
Artificial auditory devices such as cochlear implants (CIs) and auditory brainstem implants (ABIs) have become standard means to manage profound sensorineural hearing loss. However, because of their structural limitations compared to the cochlea and the cochlear nucleus, the generated auditory sensations are still imperfect. Recipients need postoperative auditory rehabilitation. To improve these rehabilitation programs, this study evaluated the effects of bimodal (audio–visual) training under seven experimental conditions of distorted speech sound, named noise-vocoded speech sound (NVSS), which is similarly processed with a speech processor of CI/ABI. Word intelligibilities under the seven conditions of two-band noise-vocoded speech were measured for auditory (A), visual (V) and auditory–visual (AV) modalities after a few hours of bimodal (AV) training. The experiment was performed with 56 subjects with normal hearing. Performance of A and AV word recognition was significantly different under the seven auditory conditions. The V word intelligibility was not influenced by the condition of combined auditory cues. However, V word intelligibility was correlated with AV word recognition under all frequency conditions. Correlation between A and AV word intelligibilities was ambiguous. These findings suggest the importance of visual cues in AV speech perception under extremely degraded auditory conditions, and underscore the importance of the possible effectiveness of bimodal audio–visual training in postoperative rehabilitation for patients with postlingual deafness who have undergone artificial auditory device implantation.
It has previously been shown that older adults may be less efficient than younger adults at processing multisensory information, and that older adults with a history of falling may be less efficient than a healthy cohort when processing audio–visual stimuli (Setti et al., 2011). We investigated whether body stance has an effect on older adults’ ability to efficiently process multisensory information and also whether being presented with multisensory stimuli while standing may affect an individual’s balance. This experiment was performed by 44 participants, including both fall-prone older adults and a healthy control cohort. We tested their susceptibility to a sound-induced flash illusion (i.e., Shams et al., 2002), during both sitting and standing positions while measuring balance parameters using body-worn sensors. The results suggest that balance control in fall prone-adults was compromised relative to adults with no falls history, and this was particularly evident whilst they were presented with the auditory-flash illusion but not the non-illusory condition. Also, when the temporal window of the stimulus onset asynchrony was narrow (70 ms) fall-prone adults were more susceptible to the illusion during the standing position compared with their performance while seated, while the performance of older adults with no history of falling was unaffected by a change in position. These results suggest a link between efficient multisensory integration and balance control and have implications for interventions when fall-prone adults encounter complex multisensory information in their environment.
The neurobiology of synaesthesia is receiving growing attention in the search for insights into consciousness, such as the binding problem. One way of decoding the neurocognitive mechanisms underlying this phenomenon is to investigate the induction of synaesthesia via neurochemical agents, as commonly occurs with psychedelic substances. How synaesthesia is affected by drugs can also help inform us of the neural mechanisms underlying this condition. To address these questions we surveyed a sample of recreational drug users regarding the prevalence, type and frequency of synaesthesia under the influence of psychedelics and other psychoactive substances. The results indicate that synaesthesia is frequently experienced following the consumption of serotonergic agonists such as LSD and psilocybin and that these same drugs appear to augment synaesthesia in congenital synaesthetes. These results implicate the serotonergic system in the experience of synaesthesia.
The multisensory integration capabilities of superior colliculus (SC) neurons are normally acquired during early postnatal life and adapted to the environment in which they will be used. Recent evidence shows that they can even be acquired in adulthood, and require neither consciousness nor any of the reinforcement contingencies generally associated with learning. This process is believed to be based on Hebbian mechanisms, whereby the temporal coupling of multiple sensory inputs initiates development of a means of integrating their information. This predicts that co-activation of those input channels is sufficient to induce multisensory integration capabilities regardless of the specific spatiotemporal properties of the initiating stimuli. However, one might expect that the stimuli to be integrated should be consonant with the functional role of the neurons involved. For the SC, this would involve stimuli that can be localized. Experience with a non-localizable cue in one modality (e.g., ambient sound) and a discrete stimulus in another (e.g., a light flash) should not be sufficient for this purpose. Indeed, experiments with cats reared in omnidirectional sound (effectively masking discrete auditory events) reveal that the simple co-activation of two sensory input channels is not sufficient for this purpose. The data suggest that experience with the kinds of cross-modal events that facilitate the role of the SC in detecting, locating, and orienting to localized external events is a guiding factor in this maturational process. Supported by NIH grants NS 036916 and EY016716.
Remote pointing devices like the Wii remote and Smart TV remote controllers have a wide range of applications and are becoming more important for the manipulation of and interactions with information on a distant display due to wide-screen display and wireless technology in electric home appliances as well as the most advanced computing technology of virtual environment and augmented reality. Because remote pointing devices are used without external support, however, muscular tremors and motional disparity between the display and motor space can result in usability problems of mouse jitters and instability. In the present study, we propose a solution using feedforward technology with multisensory stimulation in which a user is provided with predictive information while approaching a target. Using a psychophysical experiment for user behavioral effectiveness and a survey for subjective rating for user satisfaction and difficulty, the feedforward technique was found to be more effective than was typical feedback. Also, the modality variations in feedforward were discussed. In particular, the feedforward with auditory and tactile stimulation was effective in user experience. The findings can be used to improve user interfaces for remote pointing controllers.
Most of the published research on the perception of food and drink has focused on what happens in-mouth during consumption. It is, however, important to note that people’s judgments are also profoundly influenced by other sensory cues, such as haptic input, be it their direct (oral-somatosensory) contact with the food itself, or their indirect contact with the product packaging, plateware, or cutlery as well. A series of experiments are reported which together demonstrate that people also evaluate the sensory characteristics, and even the quality and estimated price, of foods and beverages based on attributes, such as the weight or the texture, of the items we utilize during consumption (be it the cutlery, the tableware, or the product packaging). For instance, yoghurt samples were rated as being significantly more dense and more satiating when consumed from a heavy bowl than when exactly the same yoghurt was consumed from an identical bowl that was somewhat lighter. In another study, the texture of a yoghurt pot was shown to influence participants’ ratings of certain of the textural attributes of foods. We have also investigated the effect of the weight of the cutlery. These results suggest that the haptic cues associated with the consumption of food and drink can also influence our in-mouth perception of their textural properties. Given that the participants did not touch the food directly with their hands, the phenomenon observed might reflect an example of ‘sensation transference’ between what participants feel in their hands and what they perceive in their mouths.
How do people remember the location of objects? Location is always relative, and thus depends on a reference frame. There are two types of reference frames: egocentric (or observer-based) and allocentric (or environmental-based). Here we investigated the reference frame people used to remember object locations in a large room. We also examined whether the choice of a given reference frame is dictated by visual experience. Thus we tested congenitally blind, late blind, and sighted blindfolded participants. Objects were organized in a structured configuration and then explored one-by-one with participants walking back and forth from a single point. After the exploration of the locations, a spatial memory test was conducted. The memory test required participants to imagine being inside the array of objects, being oriented along a given heading, and then pointing towards the required object. Crucially the headings were either aligned to the allocentric structure of the configuration, that is rows and columns, or aligned to the egocentric route walked during the exploration of the objects. The spatial representation used by the participants can be revealed by better performance when the imagined heading in the test matches the spatial representation used. We found that participants with visual experience, that is late blind and blindfolded sighted, were better with headings aligned to the allocentric structure of the configuration. On the contrary, congenitally blind were more accurate with headings aligned to the egocentric walked routes. This suggests that visual experience during early development determines a preference for an allocentric frame of reference.
The majority of research on audio–visual interaction focused on spatio-temporal factors and synesthesia-like phenomena. Especially, research on synesthesia-like phenomena has been advanced by Marks et al., and they found synesthesia-like correlation between brightness and size of visual stimuli and pitch of auditory stimuli (Marks, 1987). It seems that main interest of research on synesthesia-like phenomena is what perceptual similarity/difference between synesthetes and non-synesthetes is. We guessed that cross-modal phenomena of non-synesthetes on perceptual level emerge as a function to complement the absence or ambiguity of a certain stimulus. To verify the hypothesis, we investigated audio–visual interaction using movement (speed) of an object as visual stimuli and sine-waves as auditory stimuli. In this experiment objects (circles) moved at a fixed speed in one trial and the objects were masked in arbitrary positions, and auditory stimuli (high, middle, low pitch) were given simultaneously with the disappearance of objects. Subject reported the expected position of the objects when auditory stimuli stopped. Result showed that correlation between the position, i.e., the movement speed, of the object and pitch of sound was found. We conjecture that cross-modal phenomena on non-synesthetes tend to occur when one of sensory stimuli are absent/ambiguous.
In number-form synaesthesia, numbers become explicitly mapped onto portions of space in the mind’s eye or around the body. However, non-synaesthetes are also known to map number onto space, though in an implicit way. For example, those who are literate in a language that is written in a left-to-right direction are likely to assign small numbers to the left side of space and large numbers to the right side of space (e.g., Dehaene et al., 1993). In non-synaesthetes, this mapping is flexible (e.g., numbers map onto a circular form if the participant is primed to do so by the appearance of a clock-face), which has been interpreted as a response to task demands (e.g., Bächtold et al., 1998) or as evidence of a linguistically-mediated, rather than a direct, link between number and space (e.g., Proctor and Cho, 2006). We investigated whether synaesthetes’ number forms show the same flexibility during an odd-or-even judgement task that tapped linguistic associations between number and space (following Gevers et al., 2010). Synaesthetes and non-synaesthetes alike mapped small numbers to the verbal label ‘left’ and large numbers to the verbal label ‘right’. This surprising result may indicate that synaesthetes’ number forms are also the result of a linguistic link between number and space, instead of a direct link between the two, or that performance on tasks such as these is not mediated by the number form.
We investigated whether top-down attentional control settings can specify task-relevant features in different sensory modalities (vision and audition). Two audiovisual search tasks were used where a spatially uninformative visual singleton cue preceded a target search array. In different blocks, participants searched for a visual target (defined by colour or shape in Experiments 1 and 2, respectively), or target defined by a combination of visual and auditory features (e.g., red target accompanied by a high-pitch tone). Spatial cueing effects indicative of attentional capture by target-matching visual singleton cues in the unimodal visual search task were reduced or completely eliminated when targets were audiovisually defined. The N2pc component (i.e. index attentional target selection in vision) triggered by these cues was reduced and delayed during search for audiovisual as compared to unimodal visual targets. These results provide novel evidence that the top-down control settings which guide attentional selectivity can include perceptual features from different sensory modalities.
The body is represented in a somatotopic framework such that adjacent body parts are represented next to each other in the brain. We utilised the organisation of the somatosensory cortex to study the generalisation pattern of tactile perceptual learning. Perceptual learning refers to the process of long-lasting improvement in the performance of a perceptual task following persistent sensory exposure. In order to test if perceptual learning generalises to neighbouring brain/body areas, 12 participants were trained on a tactile discrimination task on one fingertip (using tactile oriented gratings) over the course of four days. Thresholds for tactile acuity were estimated prior to, and following, the training for the ‘trained’ finger and three additional fingers: ‘adjacent’, ‘homologous’ (the same finger as trained but on the opposite hand) and ‘other’ (which was neither adjacent nor homologous to the trained finger). Identical threshold estimating with no training was also carried out for a control group. Following training, tactile thresholds were improved (as compared to the control group). Importantly, improved performance was not exclusive for the trained finger; it generalised to the adjacent and homologous fingers, but not the other finger. We found that perceptual learning indeed generalises in a way that can be predicted by the topography of the somatosensory cortex, suggesting that sensory experience is not necessary for perceptual learning. These findings may be translated to rehabilitation procedures that train the partially-deprived cortex using similar principles of perceptual learning generalisation, such as following amputation or blindness in adults.
