Designing multi-sensor experiments to study interactive communication is costly, complex, and often poorly documented, particularly in terms of implementation challenges and the rationale behind decision-making. Here, we aim to demystify the process by detailing the development of a custom multi-sensor lab designed to monitor comprehensive behavioural and physiological responses during interactive communication. To illustrate the system architecture, we use an implemented paradigm as a worked example: two adults engaged in natural conversation while listening to realistic background noise scenes delivered via open Sennheiser HD-800 headphones. Multi-sensor data were collected using DPA headset microphones, Tobii Pro Glasses 3, a Vicon motion capture system (with Tobii integration), BIOPAC amplifiers (PPG, EDA, ECG, respiration, temperature), a Fitbit Sense 2 smartwatch, and six Logitech BRIO 4K ultra HD video cameras recording via Open Broadcaster Software (OBS). Due to high bandwidth demands and varying sampling rates (1 Hz to 48 kHz) among the different devices, all systems recorded independently which posed a significant synchronisation challenge. A central RME soundcard delivered trigger pulses to the Tobii, BIOPAC, OBS, and Vicon systems to mark stimulus onset/offset. Redundant synchronisation mechanisms were implemented to mitigate the risk of trigger failure and bespoke processing pipelines were developed to synchronise data streams. We share key challenges and creative solutions encountered in building such a lab, to help researchers understand and anticipate common pain points and make more informed decisions when implementing their own multi-sensor setups.

Global Navigation Satellite Systems (GNSS) are essential to the operation of many modern services, but their signals are inherently vulnerable to Radio Frequency Interference (RFI). The Extrapolated Single Propagation Particle Filter (ESP-PF) has proven effective in enhancing RFI geo-localization accuracy, but its computational demands pose challenges for real-time embedded systems deployment. This paper presents a hardware-software coimplementation of the ESP-PF algorithm on the AMD XCZU28DR System-on-Chip (SoC) platform, designed to achieve an optimal balance between computational performance and hardware resource efficiency. The algorithm is partitioned into eight Computation Cores, each strategically mapped to either the Programmable Logic (PL) or the Processing System (PS) to optimize execution speed and resource utilization. Experimental results show that the proposed co-design achieves a processing rate of 16 Hz, delivering speed-ups of 21% and 97% over hardware-only and software-only implementations, respectively, while maintaining comparable geolocalization accuracy. These findings highlight the potential of the proposed architecture for real-time, GNSS RFI geo-localization embedded systems deployment.

Bilateral cochlear implant (BiCI) users do not perform as well as typical hearing listeners when discriminating the direction of sound motion. This is likely due to the lack of synchronization between two independently operating sound processors. The use of bilaterally linked processors can improve the timing of electrical stimulation across the ears. However, these improvements are potentially counteracted by spectral peak-picking in sound coding strategies that may activate different electrodes across the ears, thereby reducing the fidelity of the binaural cues that BiCI users rely on for sound localization. To improve interaural synchrony, a bilateral peak-picking strategy that guarantees coordinated stimulation across the ears was developed and tested using the CCi-MOBILE, a bilaterally linked research processor. The auditory motion tracking abilities of nine BiCI users were measured using the bilateral peak-picking strategy and compared to performance with commercially available, unsynchronized processors. Results showed no effect of bilateral peak-picking. However, a small benefit was observed when tracking the range of motion of a sound when listening with bilateral synchronization which appears to be due to reduced interaural level difference changes with auditory motion. These findings suggest that dynamic auditory perception with BiCI may likely need bilaterally-synchronized hardware.

Growing numbers of children and adults who are deaf are eligible to receive cochlear implants (CI), which provide access to everyday sound. CIs in both ears (bilateral CIs or BiCIs) are becoming standard of care in many countries. However, their effectiveness is limited because they do not adequately restore the acoustic cues essential for sound localization, particularly interaural time differences (ITDs) at low frequencies. The cochlea, the auditory sensory organ, typically transmits ITDs more effectively at the apical region, which is specifically "tuned" to low frequencies. We hypothesized that effective restoration of robust ITD perception through electrical stimulation with BiCIs depends on targeting cochlear locations that transmit information most effectively. Importantly, we show that these locations can occur anywhere along the cochlea, even on the opposite end of the frequency map from where ITD cues are most dominantly encoded in an acoustic hearing system.

Cochlear implants deliver binaural cues primarily in the signal envelope through two unsynchronized sound processors, presenting challenges to bilateral cochlear implant (BiCI) users to (re)gain binaural hearing. Using a custom-designed acoustic complex to elicit multi-channel stimulation through the cochlear implant sound processors, this study measured sensitivity to interaural time difference (ITD) and interaural level difference (ILD) using a discrimination task while concurrently tracking eye gaze positions to reveal decision-making delays during binaural processing. Two groups of BiCI users were tested with either perilingual or postlingual onset of deafness. Results show all BiCI users exhibited ILD sensitivity and a majority (67% of the perilingual and 78% of the postlingual) demonstrated ITD sensitivity, though those with perilingual onset showed higher (poorer) ITD sensitivity compared to postlingual users. Additionally, ILD processing at suprathreshold magnitudes led to faster decision making as accuracy increased for both groups. However, ITD processing was less salient, with faster ITD processing observed only in the postlingual group. These findings provide insights into the differences in binaural processing in users based on the timing of deafness onset.

Global Navigation Satellite Systems (GNSS) are vulnerable to Radio Frequency Interference (RFI). Prior work has explored the use of Particle Filter (PF) for geo-locating RFI sources, but their high computational cost remains a challenge. This work presents a comparative study of hardware implementations of PF and its efficient variant, Extrapolated Single Propagation Particle Filter (ESP-PF), for RFI source tracking. Results show that the hardware implementation of PF performs slower than ESP-PF and has higher resource consumption. The work highlights the trade-offs between computational complexity and hardware resource utilisation, offering insights for optimised source tracking.

This paper introduces a methodology for hardware-software co-implementation, designed to simplify the System-on-Chip (SoC) design process when attempting to time-multiplex hardware. The approach streamlines the transformation of complex algorithms into efficient SoC implementations by providing a recipe for seamless hardware-software partitioning. As a case study, the methodology is applied to the Extrapolated Single Propagation Particle Filter algorithm for geo-locating sources of radio frequency interference and results demonstrate a considerable speed-up when using a time-multiplexed implementation.

Bilateral cochlear implants (BiCIs) improve spatial-hearing outcomes relative to a single cochlear implant (CI), but the extent varies across patients. One limiting factor may be interaurally asymmetric spectro-temporal representations of sounds. The present study simulated interaural spectro-temporal asymmetries to explore how they affect binaural processing in listeners with normal hearing. We simulated CI stimulation using high-rate (500 pps) band limited acoustic pulse trains with a 3 mm bandwidth and 4 or 6.5 kHz center frequency. Second-order amplitude modulation (AM) was applied at a 100 Hz modulation rate, 20% or 50% modulation depth, and sinusoidal or sawtooth shape. Intracranial lateralization was used to assess utility of whole-waveform interaural time differences (ITDs). Results demonstrated that lateralization of envelope ITDs was limited by the ear with smaller AM depth. This effect depended upon the sharpness of temporal onsets and the extent to which frequencies overlapped in each ear. These results suggest that interaural asymmetry in spectro-temporal representations may limit binaural outcomes of patients with BiCIs. Two well-established models of binaural processing that assume interaural symmetry and rely on coincidence-detection were used to predict performance. Models were mostly predictive of performance, but made consistent errors, suggesting that models of BiCI stimulation using coincidence-detection should be improved to account for asymmetries.

High-Level Synthesis (HLS) is a common approach for programming Field Programmable Gate Arrays (FPGAs) across various applications. HLS tools enable novice hardware designers to synthesize Register Transfer Level (RTL) representations from high-level software programming languages. However, adhering to a specific coding style and understanding the performance disparities post-synthesis can challenge new HLS users. This work introduces an Instruction Block Diagram (IBD) framework using inputs from the Vitis HLS Schedule Viewer to enhance the understanding of HLS compiler interpretation for the user code and guide optimization to improve efficiency.

IntroductionBilateral cochlear implants (BiCIs) do not restore sound localization abilities to the full extent exhibited by typical hearing (TH) listeners, partly due to poor encoding of interaural time differences (ITDs). ITD cues have been provided and investigated using synchronized research processors that ensure the precise delivery of ITD cues. These studies have been conducted in a direct stimulation setting, which bypasses the processor microphones and, in most cases, removes interaural level difference cues (ILDs). To our knowledge, this is the first study that evaluated the efficacy of synchronized stimulation in restoring sensitivity to ITDs in a free field localization experiment. This was made possible by the CCi-MOBILE, a portable and real-time processing research platform that allows for synchronizing microphone inputs.MethodsFourteen BiCI listeners were tested with experimental real-time coding strategies in comparison to unsynchronized clinical processors. We calculated the binaural cues from the acoustic stimuli at the level of microphone input.ResultsThe recordings show that the experimental coding strategies in this study deliver ITDs with greater precision than the clinical strategy. However, psychophysical testing did not show the benefit of an ITD-encoding strategy in improving localization in a free field. The ITD encoding strategies preserved ITDs, which better differentiated unique loudspeaker locations than interaural level differences (ILDs), suggesting that listeners could achieve improved performance if they accessed these cues. As expected, ILDs were similar across all strategies, including the ITD encoding strategies. The lack of improvement in localization performance is likely because ILDs remained to be the dominant cue in acute localization testing, even when ITD cues were available.DiscussionProviding BiCI listeners with adequate experiences with ITD cues may be necessary to shift their reliance from ILD dominance to a combined reliance on ILD and ITD cues in free-field conditions. The CCi-MOBILE could enable take-home practice with novel stimulation strategies for extended experiences and long-term evaluation in real-world listening environments.

Background/Objective: Bilaterally implanted cochlear implant (CI) users do not consistently have access to interaural time differences (ITDs). ITDs are crucial for restoring the ability to localize sounds and understand speech in noisy environments. Lack of access to ITDs is partly due to lack of communication between clinical processors across the ears and partly because processors must use relatively high rates of stimulation to encode envelope information. Speech understanding is best at higher stimulation rates, but sensitivity to ITDs in the timing of pulses is best at low stimulation rates. Methods: We implemented a practical “mixed rate” strategy that encodes ITD information using a low stimulation rate on some channels and speech information using high rates on the remaining channels. The strategy was tested using a bilaterally synchronized research processor, the CCi-MOBILE. Nine bilaterally implanted CI users were tested on speech understanding and were asked to judge the location of a sound based on ITDs encoded using this strategy. Results: Performance was similar in both tasks between the control strategy and the new strategy. Conclusions: We discuss the benefits and drawbacks of the sound coding strategy and provide guidelines for utilizing synchronized processors for developing strategies.

Global Navigation Satellite Systems (GNSS) are vulnerable to Radio Frequency Interference (RFI). Prior work proposed the use of an Extrapolated Single Propagation Particle Filter (ESP-PF) to geo-locate RFI sources. This work describes a hardware implementation of the ESP-PF on a Xilinx Field Programmable Gate Array (FPGA). It provides a detailed analysis of fixed-point ESP-PF modelling and discusses ways to minimise its impact. Performance analysis shows that the hardware implementation produces position estimates that are comparable to and sometimes exceed that of the floating-point implementation. The hardware implementation enables ESP-PF to be used in real-world applications to geo-locate multiple RFI sources via time-multiplexing.

Comparing outcomes between patients with bilateral hearing aids (HAs), bilateral cochlear implants (CIs), and bimodal fittings (CI + HA) are difficult because of the variations in hearing performance between the different devices and patient groups. This may impact patient counselling, device selection, and hearing outcomes for listeners with more severe hearing loss. This study seeks to identify the factors impacting the listening abilities of adults fitted with bilateral HAs, bilateral CIs, and bimodal fittings in noisy environments by comparing outcomes from commonly used clinical tests and a new task that emphasises realistic listening in background noise. By identifying these limiting factors from a variety of tests, optimising hearing aid fittings, and comparing between device configuration groups, the study will help understand where hearing aid devices no longer meet satisfactory individual outcomes, and a cochlear implant may improve long-term performance. This approach pushes toward considering real-world outcomes through realistic test measures when informing clinical counsel and device implantation or configuration recommendations. As such, adults with hearing loss can receive more tailored advice that validates their daily listening concerns and clinicians are granted a better understanding of how to improve the quality of life for their clients.

Beamforming systems that rely only on microphone arrays may suffer from poor performance in complex acoustic environments with multiple sources and room reverberation. To overcome this limitation, we propose a novel method that combines a planar microphone array and a 360° camera to steer the beamformer. The camera provides visual information about the location and orientation of sound sources, which can be used to adjust the beamformer parameters. We will present results that compare the signal-to-noise ratio (SNR) of the proposed method with a conventional beamforming microphone-based method in a simulated room with different source positions, orientations, and noise levels to investigate the benefit of visual information for beamforming in different environments. The proposed method has the potential to improve the performance of beamforming systems in applications such as speech enhancement, speech recognition, and speaker localization.

Difficulty communicating is the most challenging consequence of living with hearing loss, substantially affecting personal and professional relationships. While hearing devices help to redress this challenge, there is often a mismatch between performance measures obtained in clinical and laboratory settings and observed real-world behaviour. This discrepancy is likely due to an array of parameters, with the most notable being unrealistic speech stimuli (e.g., contrived sentence materials), artificial background noise, and tasks that do not reflect real-world communication behaviour or scenarios (e.g., sentence recall). To bridge this gap, we used sensor-fusion to understand communication difficulties in familiar communication partners engaged in natural, unrestricted conversations while listening to different levels of realistic background noise. We tallied communication breakdowns as a robust, overt metric of communication difficulty and fused data from an array of sensors including microphones, eye and motion trackers, and wearables that detect autonomic nervous system activity to objectively index communication difficulty. Our approach aims to find biomarkers that may predict the communication difficulties faced by individuals with hearing loss in the real-world. Ultimately, this research will contribute to enhancing the effectiveness of hearing devices, leading to improved social connection and quality of life for people with hearing loss.

Listening is the primary gateway for children to learn in the mainstream classroom, but the dynamics and noise of modern classrooms can make listening challenging. This is especially true for children with hearing loss, language and communication difficulties, attention deficits, autism, other learning needs, and/or those communicating in a language other than their native language. It is, therefore, critical for researchers to realistically assess how children listen to learn in the classroom and to understand how listening can be improved to enhance children’s learning and wellbeing. The Listen to Learn for Life (L3) assessment framework is a tool to use when conducting this research which holistically incorporates frameworks from health, speech and hearing sciences, and education sectors. The L3 assessment framework has three main components: characterisation of activity (perceptual setting of lecture, group work, or individual learning activity), functioning assessment (hearing, listening, comprehending, or communicating), and impact (learning and wellbeing). These are affected by External and Internal Influences. Here, we present selected examples of how to apply the framework to assess children’s listening, learning, and wellbeing during different classroom activities as well as determine the effectiveness of a chosen intervention.

Bilateral cochlear implant (BICI) listeners do not have full access to the binaural cues that normal hearing (NH) listeners use for spatial hearing tasks such as localization. When using their unsynchronized everyday processors, BICI listeners demonstrate sensitivity to interaural level differences (ILDs) in the envelopes of sounds, but interaural time differences (ITDs) are less reliably available. It is unclear how BICI listeners use combinations of ILDs and envelope ITDs, and how much each cue contributes to perceived sound location. The CCi-MOBILE is a bilaterally synchronized research processor with the untested potential to provide spatial cues to BICI listeners. In the present study, the CCi-MOBILE was used to measure the ability of BICI listeners to perceive lateralized sound sources when single pairs of electrodes were presented amplitude-modulated stimuli with combinations of ILDs and envelope ITDs. Young NH listeners were also tested using amplitude-modulated high-frequency tones. A cue weighting analysis with six BICI and ten NH listeners revealed that ILDs contributed more than envelope ITDs to lateralization for both groups. Moreover, envelope ITDs contributed to lateralization for NH listeners but had negligible contribution for BICI listeners. These results suggest that the CCi-MOBILE is suitable for binaural testing and developing bilateral processing strategies.

While listeners with bilateral cochlear implants (BiCIs) are able to access information in both ears, they still struggle to perform well on spatial hearing tasks when compared to normal hearing listeners. This performance gap could be attributed to the high stimulation rates used for speech representation in clinical processors. Prior work has shown that spatial cues, such as interaural time differences (ITDs), are best conveyed at low rates. Further, BiCI listeners are sensitive to ITDs with a mixture of high and low rates. However, it remains unclear whether mixed-rate stimuli are perceived as unitary percepts and spatially mapped to intracranial locations. Here, electrical pulse trains were presented on five, interaurally pitch-matched electrode pairs using research processors, at either uniformly high rates, low rates, or mixed rates. Eight post-lingually deafened adults were tested on perceived intracranial lateralization of ITDs ranging from 50 to 1600 μs. Extent of lateralization depended on the location of low-rate stimulation along the electrode array: greatest in the low- and mixed-rate configurations, and smallest in the high-rate configuration. All but one listener perceived a unitary auditory object. These findings suggest that a mixed-rate processing strategy can result in good lateralization and convey a unitary auditory object with ITDs.

Global Navigation Satellite System (GNSS) signals are vulnerable to intentional or unintentional Radio Frequency Interference (RFI) due to their low-received signal power levels. In order to geo-locate and track the RFI source(s), distributed network of sensors are used. However, the sensor measurements tend to be corrupted or degraded due to the environment and system noise. In addition, the non-linear nature of the position estimation based on Angle of Arrival (AOA) and Time Difference of Arrival (TDOA) measurements further complicates the problem. This paper presents the hardware design of an extrapolated single-propagation particle filter (ESP-PF). The ESP-PF was proposed to overcome the challenges in RFI source localization based on hybrid AOA and TDOA measurements from and between the sensor nodes, respectively. The performance of this hardware implementation is evaluated and compared with its fully-software counterpart in terms of position estimation accuracy. The results show that the hardware implementation performs comparably with that of the floating-point software-only implementation.

Objectives: Bilateral cochlear implant (BiCI) listeners use independent processors in each ear. This independence and lack of shared hardware prevents control of the timing of sampling and stimulation across ears, which precludes the development of bilaterally-coordinated signal processing strategies. As a result, these devices potentially reduce access to binaural cues and introduce disruptive artifacts. For example, measurements from two clinical processors demonstrate that independently-running processors introduce interaural incoherence. These issues are typically avoided in the laboratory by using research processors with bilaterally-synchronized hardware. However, these research processors do not typically run in real-time and are difficult to take out into the real-world due to their benchtop nature. Hence, the question of whether just applying hardware synchronization to reduce bilateral stimulation artifacts (and thereby potentially improve functional spatial hearing performance) has been difficult to answer. The CI personal digital assistant (ciPDA) research processor, which uses one clock to drive two processors, presented an opportunity to examine whether synchronization of hardware can have an impact on spatial hearing performance. Design: Free-field sound localization and spatial release from masking (SRM) were assessed in 10 BiCI listeners using both their clinical processors and the synchronized ciPDA processor. For sound localization, localization accuracy was compared within-subject for the two processor types. For SRM, speech reception thresholds were compared for spatially separated and co-located configurations, and the amount of unmasking was compared for synchronized and unsynchronized hardware. There were no deliberate changes of the sound processing strategy on the ciPDA to restore or improve binaural cues. Results: There was no significant difference in localization accuracy between unsynchronized and synchronized hardware (p = 0.62). Speech reception thresholds were higher with the ciPDA. In addition, although five of eight participants demonstrated improved SRM with synchronized hardware, there was no significant difference in the amount of unmasking due to spatial separation between synchronized and unsynchronized hardware (p = 0.21). Conclusions: Using processors with synchronized hardware did not yield an improvement in sound localization or SRM for all individuals, suggesting that mere synchronization of hardware is not sufficient for improving spatial hearing outcomes. Further work is needed to improve sound coding strategies to facilitate access to spatial hearing cues. This study provides a benchmark for spatial hearing performance with real-time, bilaterally-synchronized research processors.