Sensory substitution and augmentation depend on the brain's capacity to incorporate new sensory cues into perception, but the flexibility of underlying computations remains unclear. We compared how a newly learned depth cue is integrated, relative to familiar cues, after 1 h of training. Using forced-choice depth judgments in 78 observers, we assessed three indicators of integration for familiar and novel cues: (1) cue combination, predicting precision gains; (2) re-weighting, predicting reliability-based biases; and (3) congruence sensitivity, predicting increased sensitivity for learned mappings. Familiar cues were combined near-optimally, while only some people combined the novel and familiar cues. Test-retest measures suggested that this reflected reliable individual differences. Nonetheless, both familiar and novel cue pairs were re-weighted by reliability and showed sensitivity to incongruence. These results indicate that although novel cues can be rapidly mapped to depth and weighted appropriately, their integration varies across individuals, suggesting that individuals may vary in perceptual flexibility.

Human perception is often characterised by efficient combination of sensory signals (cues). In recent studies, people could also improve precision via newly learned cues, with applications to enhance perception in healthy and clinical groups. However, it is unclear whether new cues can enhance manual object interactions. To study how new cues are used for object weight perception, people compared weights of containers. With haptic information plus the familiar visual cue of volume, participants showed precision improvements indicating cue combination. By contrast, a group of participants briefly trained with a novel visual cue to weight (line orientation) did not show improvements expected from combination. We then asked whether prolonged training (12 hours) with the novel cue would promote combination, testing for significant precision gains individually in six participants. Half of participants showed combination benefits, but these were not clearly related to training, as some combined cues before training. Using an illusion analogous to the size-weight illusion, we also asked whether the novel cue would become an automatic predictor of weight: two participants were susceptible to the illusion. We conclude that weight perception is susceptible to some enhancement, but subject to training effects and individual differences that are not yet understood.

Efficient manual interactions rely on predictive cues to object weight. While familiar cues (e.g., size) readily support motor planning, it is unclear whether newly learned cues can enhance performance in demanding tasks. Understanding the predictive utility of novel cues has implications for sensory augmentation. We investigated this using a goal-directed hitting task, where participants earned a performance-based reward for squashing virtual bugs. Targets appeared briefly at unpredictable locations on a touchscreen, and participants used light and heavy containers to squash the bugs. Container weight was signaled with a familiar (visual volume), novel (checkerboard patterns), or no cue (identical containers). The number of bugs hit was highest with the familiar cue, followed by the novel cue, and smallest with no cue. Response time followed a corresponding pattern: movements were fastest with the familiar cue, followed by the novel cue, and slowest with no cue. No differences were found in precision, but constant bias was highest with the novel cue, followed by no cue, and smallest with the familiar cue. These findings demonstrate that novel cues, though less effective than familiar ones, can be rapidly integrated to guide efficient motor behavior, suggesting their potential for augmenting human performance in real-world tasks.

Human perception is often characterised by efficient combination of sensory signals (cues). In recent studies, people could also improve precision via newly learned cues, with applications to enhance perception in healthy and clinical groups. However, it is unclear whether new cues can enhance manual object interactions. To study how new cues are used for object weight perception, people compared weights of containers. With haptic information plus the familiar visual cue of volume, participants showed precision improvements indicating cue combination. By contrast, a group of participants briefly trained with a novel visual cue to weight (line orientation) did not show improvements expected from combination. We then asked whether prolonged training (12 h) with the novel cue would promote combination, testing for significant precision gains individually in six participants. Half of participants showed combination benefits, but these were not clearly related to training, as some combined cues before training. Using an illusion analogous to the size-weight illusion, we also asked whether the novel cue would become an automatic predictor of weight: two participants were susceptible to the illusion. We conclude that weight perception is susceptible to some enhancement, but subject to training effects and individual differences that are not yet understood.

When a person picks up an object, naturalistic cues inform fine motor planning that is reflected in early spikes in force rate changes. Naturalistic cues to weight can also create an illusion whereby a signal to being heavier leads to the object being perceived as lighter - for example, the size-weight illusion. The present study asked to what extent an arbitrary new auditory cue, one that signals object weight, participates in these effects. In Experiment 1, participants used the new signal to adjust both their peak grip force rates and peak load force rates while lifting an object, consistent with using it for efficient motor planning. This matched how they used a naturalistic visual size cue. In Experiment 2, a new audio cue to heavier weight led to a heavier reported weight - the opposite of a size-weight illusion, and opposite to how the same participants used a naturalistic visual size cue. Thus, while the newly learned audio-weight mapping had similar functional properties to its more familiar perceptual counterpart, it did not show the same signature of automatic processing. These results have implications for understanding the flexible use of new cues and for targeting the underlying mechanisms in order to augment human abilities.

New wearable devices and technologies provide unprecedented scope to augment or substitute human perceptual abilities. However, the flexibility to reorganize brain processing to use novel sensory signals during early sensitive periods in infancy is much less evident at later ages, making integration of new signals into adults’ perception a significant challenge. We believe that an approach informed by cognitive neuroscience is crucial for maximizing the true potential of new sensory technologies. Here, we present a framework for measuring and evaluating the extent to which new signals are integrated within existing structures of perception and experience. As our testbed, we use laboratory tasks in which healthy volunteers learn new, augmented perceptual-motor skills. We describe a suite of measures of (i) perceptual function (psychophysics), (ii) neural representations (fMRI/decoding), and (iii) subjective experience (qualitative interview/micro-phenomenology) targeted at testing hypotheses about how newly learned signals become integrated within perception and experience. As proof of concept, we provide example data showing how this approach allows us to measure changes in perception, neural processing, and subjective experience. We argue that this framework, in concert with targeted approaches to optimizing training and learning, provides the tools needed to develop and optimize new approaches to human sensory augmentation and substitution.

BACKGROUND Sensory substitution and augmentation systems (SSASy) seek to either replace or enhance existing sensory skills by providing a new route to access information about the world. Tests of such systems have largely been limited to untimed, unisensory tasks. OBJECTIVE To test the use of a SSASy for rapid, ballistic motor actions in a multisensory environment. METHODS Participants played a stripped-down version of air hockey in virtual reality with motion controls (Oculus Touch). They were trained to use a simple SASSy (novel audio cue) for the puck's location. They were tested on ability to strike an oncoming puck with the SASSy, degraded vision, or both. RESULTS Participants coordinated vision and the SSASy to strike the target with their hand more consistently than with the best single cue alone, t(13) = 9.16, p <.001, Cohen's d = 2.448. CONCLUSIONS People can adapt flexibly to using a SSASy in tasks that require tightly timed, precise, and rapid body movements. SSASys can augment and coordinate with existing sensorimotor skills rather than being limited to replacement use cases - in particular, there is potential scope for treating moderate vision loss. These findings point to the potential for augmenting human abilities, not only for static perceptual judgments, but in rapid and demanding perceptual-motor tasks.

It is clear that people can learn a new sensory skill-a new way of mapping sensory inputs onto world states. It remains unclear how flexibly a new sensory skill can become embedded in multisensory perception and decision-making. To address this, we trained typically sighted participants (N = 12) to use a new echo-like auditory cue to distance in a virtual world, together with a noisy visual cue. Using model-based analyses, we tested for key markers of efficient multisensory perception and decision-making with the new skill. We found that 12 of 14 participants learned to judge distance using the novel auditory cue. Their use of this new sensory skill showed three key features: (a) It enhanced the speed of timed decisions; (b) it largely resisted interference from a simultaneous digit span task; and (c) it integrated with vision in a Bayes-like manner to improve precision. We also show some limits following this relatively short training: Precision benefits were lower than the Bayes-optimal prediction, and there was no forced fusion of signals. We conclude that people already embed new sensory skills in flexible multisensory perception and decision-making after a short training period. A key application of these insights is to the development of sensory augmentation systems that can enhance human perceptual abilities in novel ways. The limitations we reveal (sub-optimality, lack of fusion) provide a foundation for further investigations of the limits of these abilities and their brain basis. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Studying how sensory signals from different sources (sensory cues) are integrated within or across multiple senses allows us to better understand the perceptual computations that lie at the foundation of adaptive behaviour. As such, determining the presence of precision gains – the classic hallmark of cue combination – is important for characterising perceptual systems, their development and functioning in clinical conditions. However, empirically measuring precision gains to distinguish cue combination from alternative perceptual strategies requires careful methodological considerations. Here, we note that the majority of existing studies that tested for cue combination either omitted this important contrast, or used an analysis approach that, unknowingly, strongly inflated false positives. Using simulations, we demonstrate that this approach enhances the chances of finding significant cue combination effects in up to 100% of cases, even when cues are not combined. We establish how this error arises when the wrong cue comparator is chosen and recommend an alternative analysis that is easy to implement but has only been adopted by relatively few studies. By comparing combined-cue perceptual precision with the best single-cue precision, determined for each observer individually rather than at the group level, researchers can enhance the credibility of their reported effects. We also note that testing for deviations from optimal predictions alone is not sufficient to ascertain whether cues are combined. Taken together, to correctly test for perceptual precision gains, we advocate for a careful comparator selection and task design to ensure that cue combination is tested with maximum power, while reducing the inflation of false positives.

Efficient decision-making requires accounting for sources of uncertainty (noise, or variability). Many studies have shown how the nervous system is able to account for perceptual uncertainty (noise, variability) that arises from limitations in its own abilities to encode perceptual stimuli. However, many other sources of uncertainty exist, reflecting for example variability in the behaviour of other agents or physical processes. Here we review previous studies on decision making under uncertainty as a function of the different types of uncertainty that the nervous system encounters, showing that noise that is intrinsic to the perceptual system can often be accounted for near-optimally (i.e. not statistically different from optimally), whereas accounting for other types of uncertainty can be much more challenging. As an example, we present a study in which participants made decisions about multisensory stimuli with both intrinsic (perceptual) and extrinsic (environmental) uncertainty and show that the nervous system accounts for these differently when making decisions: they account for internal uncertainty but under-account for external. Human perceptual systems may be well equipped to account for intrinsic (perceptual) uncertainty because, in principle, they have access to this. Accounting for external uncertainty is more challenging because this uncertainty must be learned. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Cue combination describes the use of two sensory cues together to increase perceptual precision. Internal relative bias describes a situation in which two cues to the same state of the world are perceived as signaling different states of the world on average. Current theory and evidence have difficulty accounting for many instances where cue combination is absent, such as in children under 10 years old, and in a variety of tasks. Here we show that internal relative biases between cues could be a key explanatory factor. Experiment 1, studying children's three-dimensional (slant) perception via disparity and texture, found a negative cross-sectional correlation between internal relative bias and cue combination behavior in 7- to 10-year-olds. Strikingly, children who had below-median levels of internal relative bias were able to combine cues, unlike the typical result for that age range. Experiment 2, studying adults’ visual-auditory localization, found that cue combination behavior increased after an intervention designed to decrease internal relative bias. We interpret this as strong but preliminary evidence that internal relative bias can disrupt cue combination behavior. This provides a plausible mechanism to explain why children under 10 generally do not combine cues and why the audiovisual cue combination is so inconsistent in adults. Moving forward, we suggest that researchers who fail to find an expected cue combination effect should further investigate the possibility of issues with internal relative bias. Decreasing internal relative bias may also be an important goal for rehabilitation and sensory substitution or augmentation approaches to promoting efficient multisensory perception.

Reliability-weighted averaging of multiple perceptual estimates (or cues) can improve precision. Research suggests that newly learned statistical associations can be rapidly integrated in this way for efficient decision-making. Yet, it remains unclear if the integration of newly learned statistics into decision-making can directly influence perception, rather than taking place only at the decision stage. In two experiments, we implicitly taught observers novel associations between shape and color. Observers made color matches by adjusting the color of an oval to match a simultaneously presented reference. As the color of the oval changed across trials, so did its shape according to a novel mapping of axis ratio to color. Observers showed signatures of reliability-weighted averaging—a precision improvement in both experiments and reweighting of the newly learned shape cue with changes in uncertainty in Experiment 2. To ask whether this was accompanied by perceptual effects, Experiment 1 tested for forced fusion by measuring color discrimination thresholds with and without incongruent novel cues. Experiment 2 tested for a memory color effect, observers adjusting the color of ovals with different axis ratios until they appeared gray. There was no evidence for forced fusion and the opposite of a memory color effect. Overall, our results suggest that the ability to quickly learn novel cues and integrate them with familiar cues is not immediately (within the short duration of our experiments and in the domain of color and shape) accompanied by common perceptual effects.

When the illumination falling on a surface change, so does the reflected light. Despite this, adult observers are good at perceiving surfaces as relatively unchanging-an ability termed colour constancy. Very few studies have investigated colour constancy in infants, and even fewer in children. Here we asked whether there is a difference in colour constancy between children and adults; what the developmental trajectory is between six and 11 years; and whether the pattern of constancy across illuminations and reflectances differs between adults and children. To this end, we developed a novel, child-friendly computer-based object selection task. In this, observers saw a dragon's favourite sweet under a neutral illumination and picked the matching sweet from an array of eight seen under a different illumination (blue, yellow, red, or green). This set contained a reflectance match (colour constant; perfect performance) and a tristimulus match (colour inconstant). We ran two experiments, with two-dimensional scenes in one and three-dimensional renderings in the other. Twenty-six adults and 33 children took part in the first experiment; 26 adults and 40 children took part in the second. Children performed better than adults on this task, and their performance decreased with age in both experiments. We found differences across illuminations and sweets, but a similar pattern across both age groups. This unexpected finding might reflect a real decrease in colour constancy from childhood to adulthood, explained by developmental changes in the perceptual and cognitive mechanisms underpinning colour constancy, or differences in task strategies between children and adults. HIGHLIGHTS: Six- to 11-year-old children demonstrated better performance than adults on a colour constancy object selection task. Performance decreased with age over childhood. These findings may indicate development of cognitive strategies used to overcome automatic colour constancy mechanisms.

Mature perceptual systems can learn new arbitrary sensory signals (novel cues) to properties of the environment, but little is known about the extent to which novel cues are integrated into normal perception. In normal perception, multiple uncertain familiar cues are combined, often near-optimally (reliability-weighted averaging), to increase perceptual precision. We trained observers to use abstract novel cues to estimate horizontal locations of hidden objects on a monitor. In experiment 1, 4 groups of observers each learned to use a different novel cue. All groups benefited from a suboptimal but significant gain in precision using novel and familiar cues together after short-term training (3 ∼1.5 hr sessions), extending previous reports of novel-familiar cue combination. In experiment 2, we tested whether 2 novel cues may also be combined with each other. One pair of novel cues could be combined to improve precision but the other could not, at least not after 3 sessions of repeated training. Overall, our results provide extensive evidence that novel cues can be learned and combined with familiar cues to enhance perception, but mixed evidence for whether perceptual and decision-making systems can extend this ability to the combination of multiple novel cues with only short-term training. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Vision undergoes major development during infancy and childhood, demonstrated in improvements in both detection and recognition tasks. Classically, developmental vision research has focussed on sensitivity improvements in early visual channels. However, in recent years, decision-theoretic approaches have formalised how changes in visual performance could also result from more efficient use of available information, for example by optimising decision rules, cost functions, and priors. Using these quantitative frameworks, we are beginning to understand how these factors contribute to childhood vision. For example, improved depth perception in late childhood reflects a shift from processing depth cues independently to combining them in visual cortex, as demonstrated by the emergence of fMRI evidence for fused depth-cue representations within neural detectors in area V3B. Similarly, development of visual motion-, location-, and object perception, in part reflects more efficient combining of stimulus features (e.g., averaging dots across displays) besides greater sensitivity to these features’ properties (e.g., single dot motion). Thus, rather than greater sensitivity to basic visual information, substantial improvements in visual discrimination and detection may reflect better inferential capacities. This also applies to visually-guided movement tasks that emulate real-life action under risk: while adults can rapidly identify visuomotor strategies that minimise risk and uncertainty in new situations with complex cost factors, children up to age 10 years do not. Together, these studies show that improved decision-making plays a major role in visual development in childhood, and that modelling this role is needed to gain computational-level insight in the driving factors of human visual plasticity.