Automating the verbal paired associates test to assess memory remotely and at scale

Background Verbal neuropsychological tests are sensitive to the early signs of neurodegeneration in older adults. However, their dependence on in‐person testing and manual scoring make them unsuitable for large‐scale screening and home‐based monitoring. To address this, we have developed a fully automated verbal paired associate learning (VPA) test. Here we describe the program of work to refine and validate this task, and ensure suitability of the test for remote, repeat administration. Methods Two phases of data collection were carried out. In the first, data was collected from 3,264 participants aged 17–86 years (M = 34.5, SD = 12.32) at home and on their own devices. The automated VPA task comprises eight word‐pairs, learned over three attempts. Automatic speech recognition (ASR) scores responses in real‐time, using the Neurovocalix platform. Following this phase, the task was refined to better enable repeat testing. This included producing a large corpus of algorithmically generated pairs, controlling for important linguistic features (e.g. frequency and concreteness), and adding brief familiarisation to the first visit to increase reliability and stability. We assessed: 1) the accuracy of automated scoring 2) the impact of participant characteristics and task design on performance and consistency 3) participant feedback. Results Data from the first phase showed excellent correspondence between human and automated scoring of VPA, with a correlation .93 ( p < 0.001), and the expected sensitivity to ageing number of task performance. Participant demographics and technology choice did not impact correspondence between ASR scoring and manual scoring. In the second phase data collection, we compared VPA performance in participants with (N = 190) and without (N = 186) prior familiarisation. Familiarisation dramatically improved consistency of participant performance, eliminating the significant effect of task order on performance (F > 100, p < 0.001). Participants reported that they found the task engaging and easy to understand, and repeated testing over five sessions in older adults was well tolerated. Conclusions Together, these results demonstrate that remote, automated, voice‐based, cognitive assessments are feasible and acceptable. These methods may therefore be useful in the context of home‐based screening or monitoring in remote clinical trials, enabling longitudinal assessment of memory in AD.