Reliability and Validity of Ecological Momentary Assessment of Cognition in Clinical and Community Samples (Preprint)

BACKGROUND Current methods of evaluating cognitive functioning typically rely on a single timepoint to assess and characterize an individual’s performance. However, cognitive functioning fluctuates within-individuals over time in relation to environmental, psychological, and physiological contexts. This limits the generalizability and diagnostic utility of single-timepoint assessments, particularly among individuals who may exhibit large variations in cognition depending on physiological or psychological context (e.g., those with type 1 diabetes (T1D), who may have fluctuating glucose concentrations throughout the day). OBJECTIVE We report the reliability and validity of cognitive ecological momentary assessment (cognitive EMA) as a method for understanding between-person differences and capturing within-person variation in cognition over time, in a community and a T1D sample. METHODS Cognitive performance was measured three times a day for 15 days (T1D sample, n=198, recruited through endocrinology clinics) and 10 days (community sample, n=128, recruited from TestMyBrain.org, an online citizen science platform) using ultra-brief cognitive tests developed for cognitive EMA. Our cognitive EMA platform allowed for remote, automated assessment in participants’ natural environments, enabling measurement of within-person cognitive variation without the burden of repeated laboratory or clinic visits. This allowed us to evaluate reliability and validity in samples that differed in their expected degree of cognitive variability as well as method of recruitment. RESULTS Results demonstrate excellent between-person reliability (ranging from 0.95 to 0.99) and construct validity of cognitive EMA in both the T1D and community samples. Within-person reliability in both samples (ranging from 0.20 to 0.80) was comparable to previous studies in healthy aging adults. As expected, full-length baseline and EMA versions of TestMyBrain (TMB) tests correlated highly with one another and loaded on expected cognitive domains when using exploratory factor analysis. Interruptions had higher negative impacts on accuracy-based outcomes ( = -.34 to -.26; all ps < 0.001) than reaction-time based outcomes ( = -.07 to -.02; p < 0.001 to p = 0.4). CONCLUSIONS We demonstrate that ultra-brief mobile assessments are both reliable and valid across two very different clinic versus community samples, despite the conditions in which cognitive EMAs are administered, which are often associated with more noise and variability. The psychometric characteristics described here should be leveraged appropriately depending on the goals of cognitive assessment (e.g., diagnostic vs everyday functioning) and population being studied.