The ability to flexibly learn the structure of one’s surroundings (structure learning) is crucial for adaptive behaviour. Use of an inaccurate model of the environment can lead to incorrect inferences, and thus maladaptive actions. Despite this, relatively little is understood about how structure learning occurs in human cognition, with only a handful of experiments investigating the phenomenon. To address this, we built on existing approaches to create an online clustering algorithm, in which we included a working memory component, allowing belief update about past stimuli (retrospective inference) in contrast with the more widespread fully online approach. We used this model to simulate behaviour on a novel structure learning task, where optimal performance required estimating the number and properties of discrete clusters of continuous stimuli. In this work we show how our algorithm outperforms a parametric one (i.e. with fixed number of clusters). We further demonstrate how retrospective inference benefits structure learning, with performance increasing with working memory capacity. We finally discuss trial-by trial measures that can be derived from our model, which provide testable predictions for future empirical studies.

Learning about the statistics of one’s environment is a fundamental requirement of adaptive behaviour. In this experiment we probe whether pupil dilation in response to brief auditory stimuli reflects automatic statistical learning about the underlying stimulus distributions. Specifically, we consider whether pupil dilation reflects automatic (task-irrelevant) learning about the precision of Gaussian distributions of pitch in a sequence of tones. We provide clear evidence, both by comparing responses to perceptually identical probe tones in low and high precision blocks, and using a novel model-based analysis, that subjects did indeed track the precision of the stimulus distribution. This extends previous work looking at electrophysiological effects of precision (or, equivalently, variance) learning, and provides new evidence that the putatively noradrenergic processes underlying pupil dilation reflect rapidly updated information about distributions of sensory stimuli. In addition, our study represents a validation of our model-based approach to analysing pupillometry data, which we believe has considerable promise for future studies.

BACKGROUND:Psychotic experiences emerge from abnormalities in perception and belief formation and occur more commonly in those experiencing childhood trauma. However, which precise aspects of belief formation are atypical in psychosis is not well understood. We used a computational modeling approach to characterize belief updating in young adults in the general population, examine their relationship with psychotic outcomes and trauma, and determine the extent to which they mediate the trauma-psychosis relationship.METHODS:We used data from 3360 individuals from the Avon Longitudinal Study of Parents and Children birth cohort who completed assessments for psychotic outcomes, depression, anxiety, and two belief updating tasks at age 24 and had data available on traumatic events assessed from birth to late adolescence. Unadjusted and adjusted regression and counterfactual mediation methods were used for the analyses.RESULTS:Basic behavioral measures of belief updating (draws-to-decision and disconfirmatory updating) were not associated with psychotic experiences. However, computational modeling revealed an association between increased decision noise with both psychotic experiences and trauma exposure, although <3% of the trauma-psychotic experience association was mediated by decision noise. Belief updating measures were also associated with intelligence and sociodemographic characteristics, confounding most of the associations with psychotic experiences. There was little evidence that belief updating parameters were differentially associated with delusions compared with hallucinations or that they were differentially associated with psychotic outcomes compared with depression or anxiety.CONCLUSIONS:These findings challenge the hypothesis that atypical belief updating mechanisms (as indexed by the computational models and behavioral measures we used) underlie the development of psychotic phenomena.

Probabilistic models of cognition typically assume that agents make inferences about current states by combining new sensory information with fixed beliefs about the past, an approach known as Bayesian filtering. This is computationally parsimonious, but, in general, leads to suboptimal beliefs about past states, since it ignores the fact that new observations typically contain information about the past as well as the present. This is disadvantageous both because knowledge of past states may be intrinsically valuable, and because it impairs learning about fixed or slowly changing parameters of the environment. For these reasons, in offline data analysis it is usual to infer on every set of states using the entire time series of observations, an approach known as (fixed-interval) Bayesian smoothing. Unfortunately, however, this is impractical for real agents, since it requires the maintenance and updating of beliefs about an ever-growing set of states. We propose an intermediate approach, finite retrospective inference (FRI), in which agents perform update beliefs about a limited number of past states. (Formally, this represents online fixed-lag smoothing with a sliding window.) This can be seen as a form of bounded rationality in which agents seek to optimise the accuracy of their beliefs subject to computational and other resource costs. We show through simulation that this approach has the capacity to significantly increase the accuracy of both inference and learning, using a simple variational scheme applied to both randomly generated Hidden Markov models (HMMs), and a specific application of the HMM, in the form of the widely used probabilistic reversal task. Our proposal thus constitutes a theoretical contribution to normative accounts of bounded rationality, which makes testable empirical predictions that can be explored in future work.

Successful behaviour depends on the right balance between maximising reward and soliciting information about the world. Here, we show how different types of information-gain emerge when casting behaviour as surprise minimisation. We present two distinct mechanisms for goal-directed exploration that express separable profiles of active sampling to reduce uncertainty. 'Hidden state' exploration motivates agents to sample unambiguous observations to accurately infer the (hidden) state of the world. Conversely, 'model parameter' exploration, compels agents to sample outcomes associated with high uncertainty, if they are informative for their representation of the task structure. We illustrate the emergence of these types of information-gain, termed active inference and active learning, and show how these forms of exploration induce distinct patterns of 'Bayes-optimal' behaviour. Our findings provide a computational framework for understanding how distinct levels of uncertainty systematically affect the exploration-exploitation trade-off in decision-making.

The nascent field computational psychiatry has undergone exponential growth since its inception. To date, much of the published work has focused on choice behaviors, which are primarily modeled within a reinforcement learning framework. While this initial normative effort represents a milestone in psychiatry research, the reality is that many psychiatric disorders are defined by disturbances in subjective states (e.g., depression, anxiety) and associated beliefs (e.g., dysmorphophobia, paranoid ideation), which are not considered in normative models. In this paper, we present interoceptive inference as a candidate framework for modeling subjective-and associated belief-states in computational psychiatry. We first introduce the notion and significance of modeling subjective states in computational psychiatry. Next, we present the interoceptive inference framework, and in particular focus on the relationship between interoceptive inference (i.e., belief updating) and emotions. Lastly, we will use drug craving as an example of subjective states to demonstrate the feasibility of using interoceptive inference to model the psychopathology of subjective states.

Determining the best model or models for a particular data set, a process known as Bayesian model comparison, is a critical part of probabilistic inference. Typically, this process assumes a fixed model-space (that is, a fixed set of candidate models). However, it is also possible to perform Bayesian inference over model-spaces themselves, thus determining which spaces provide the best explanation for observed data. Model-space inference (MSI) allows the effective exclusion of poorly performing models (a process analogous to Automatic Relevance Detection), and thus mitigates against the well-known phenomenon of model dilution, resulting in posterior probability estimates that are, on average, more accurate than those produced when using a fixed model-space. We focus on model comparison in the context of multiple independent data sets (as produced, for example, by multi-subject behavioural or neuroimaging studies), and cast our proposal as a development of random-effects Bayesian Model Selection, the current state-of-the-art in the field. We demonstrate the increased accuracy of MSI using simulated behavioural and neuroimaging data, as well as by assessing predictive performance in previously-acquired empirical data. Additionally, we explore other applications of MSI, including formal testing for a diversity of models within a population, and comparison of model-spaces between populations. Our approach thus provides an important new tool for model comparison.

Learning about the uncertainty of environmental stimuli is a fundamental requirement of adaptive behaviour. In this experiment we probe whether pupil dilation in response to brief auditory stimuli reflects statistical learning about the underlying stimulus distributions. Specifically, we consider whether pupil dilation reflects automatic (task-irrelevant) learning about the precision of Gaussian distributions of tones. By comparing responses to perceptually identical outlier and standard tones in low and high precision blocks, we provide clear evidence that subjects do indeed learn about precision, as reflected by increased responses to surprising (outlier) tones during high precision blocks. This extends previous work looking at electrophysiological effects of precision learning, and provides new evidence that the putatively noradrenergic processes underlying pupil dilation reflect learning about the uncertainty of stimulus distributions. In addition, we use our data to test a new convolution-based approach for analysing pupillometry data, which we believe has considerable promise for this and future studies.

Distinguishing between meaningful and meaningless sensory information is fundamental to forming accurate representations of the world. Dopamine is thought to play a central role in processing the meaningful information content of observations, which motivates an agent to update their beliefs about the environment. However, direct evidence for dopamine's role in human belief updating is lacking. We addressed this question in healthy volunteers who performed a model-based fMRI task designed to separate the neural processing of meaningful and meaningless sensory information. We modeled participant behavior using a normative Bayesian observer model and used the magnitude of the model-derived belief update following an observation to quantify its meaningful information content. We also acquired PET imaging measures of dopamine function in the same subjects. We show that the magnitude of belief updates about task structure (meaningful information), but not pure sensory surprise (meaningless information), are encoded in midbrain and ventral striatum activity. Using PET we show that the neural encoding of meaningful information is negatively related to dopamine-2/3 receptor availability in the midbrain and dexamphetamine-induced dopamine release capacity in the striatum. Trial-by-trial analysis of task performance indicated that subclinical paranoid ideation is negatively related to behavioral sensitivity to observations carrying meaningful information about the task structure. The findings provide direct evidence implicating dopamine in model-based belief updating in humans and have implications for understating the pathophysiology of psychotic disorders where dopamine function is disrupted.

Older adults struggle in dealing with changeable and uncertain environments across several cognitive domains. This has been attributed to difficulties in forming adequate task representations that help navigate uncertain environments. Here, we investigate how, in older adults, inadequate task representations impact on model-based reversal learning. We combined computational modeling and pupillometry during a novel model-based reversal learning task, which allowed us to isolate the relevance of task representations at feedback evaluation. We find that older adults overestimate the changeability of task states and consequently are less able to converge on unequivocal task representations through learning. Pupillometric measures and behavioral data show that these unreliable task representations in older adults manifest as a reduced ability to focus on feedback that is relevant for updating task representations, and as a reduced metacognitive awareness in the accuracy of their actions. Instead, the data suggested older adults' choice behavior was more consistent with a guidance by uninformative feedback properties such as outcome valence. Our study highlights that an inability to form adequate task representations may be a crucial factor underlying older adults' impaired model-based inference.

The aberrant salience hypothesis of schizophrenia proposes that symptoms such as paranoia arise when behavioural salience is attributed to neutral stimuli. Mesolimbic dopamine dysfunction is thought to be central to this mechanism; building on findings that activity in this pathway conveys a (signed) reward prediction error signal. Given that many psychotic symptoms are not explicitly related to reward learning, it is relevant that recent studies in rodents have demonstrated a role for midbrain dopamine neurons in value-neutral associative learning. Direct evidence for this role in humans, however, is lacking. In this study we asked whether the mesolimbic dopamine circuit is involved in encoding the value-neutral meaningful information of observations, using a model-based functional magnetic resonance imaging (fMRI) task and dopamine positron emission tomography (PET). We define ‘meaningful information’ as the degree to which an observation results in a belief-update to an agent’s internal model of the environment (Kullback-Leibler divergence from prior to posterior beliefs; ‘Bayesian surprise’). Participants were tasked to infer the current (hidden) state of the environment, using partially-informative observations at each trial, and then report their belief at the end of each trial. Participant beliefs were modelled using a Hidden Markov Model of the task and iterative application of Bayes’ rule, allowing us to quantify the Bayesian surprise (meaningful information content) associated with a trial observation. Crucially, our task de-correlated Bayesian surprise from both the pure sensory unexpectedness of an observation (unexpected but meaningless information) and its signed reward prediction error. 39 healthy participants (22M, mean age 26y) performed 180 task trials within an fMRI scanner. 36 participants also had a [11C]-(+)-4-propyl-9-hydroxy-naphthoxazine (PHNO) PET scan to quantify dopamine-2/3 receptor (D2/3R) availability. 17 participants additionally had a second PET scan 3hrs post 0.5mg/kg oral dexamphetamine, to quantify striatal dopamine release capacity. Neuroimaging analyses were restricted to the bilateral substantia nigra/ventral tegmental area (SN/VTA) and ventral striatum (VS). Our computational model closely predicted participant behaviour (R2= .67), and there was a negative correlation between subclinical paranoia and the degree to which participant behaviour approximated normative Bayesian performance (rho = -.60, P<0.001). Neuronal activation encoding the meaningful information content of an observation (Bayesian surprise) was present in SN/VTA and VS (both P(peak)<0.05, SVC), whereas no such encoding was present for sensory unexpectedness or reward-prediction error. Crucially, activation encoding Bayesian surprise was inversely correlated with D2/3R availability in the SN/VTA (rho = -.43, P=0.009), consistent with a tonic inhibitory role for midbrain D2/3Rs. Moreover, activation encoding Bayesian surprise was inversely related to dopamine release capacity in the VS (rho = -.66, P=0.005), indicating that subjects with high dopamine release capacity showed blunted striatal activation in response to belief-changing information, as is also found in schizophrenia. We provide direct evidence in humans that a mesolimbic dopamine circuit is involved in encoding the meaningful information content of observations, distinct from its involvement in processing signed reward prediction error. These results implicate dopamine in a wider range of function than reward learning, including updating a predictive associative model of the world, and are therefore relevant for the aberrant salience hypothesis of schizophrenia.

Our environment abounds in objects of different orientations. When we want to interact with these objects, e.g. by grasping, we can do so provided we accurately perceive their orientation. The orientation of an object can be characterized along three independent spatial axes: x (or slant), y (or tilt), and z (or depth). How does the human brain achieve the percept of object orientation, is it by processing each axis separately, or by processing them conjointly? Also, do different (combinations of) monocular and binocular depth cues affect our perception of depth differently? To answer these questions, we designed a behavioral task in which participants were asked to reproduce the orientation of a target array by adjusting the tilt and slant of a test array. On each trial, random combinations of target orientations on both X and Y were generated. Error magnitudes for each trial and for each axis were recorded. Over the course of three behavioral experiments, we coupled this task with an incremental number of depth cues, such that: i) only texture gradients were available; ii) texture gradients were coupled with line vergeance; iii) texture gradients, line vergeance and binocular disparity were used together. Our results appear to indicate a differential coupling strength between X and Y orientation perception according to which depth cues are available. In a follow-up fMRI study using the same behavioral procedure and texture gradients only, we presented not only trials where X and Y orientations were generated together, but also trials where only one of these axes was manipulated. The neuroimaging data have been analyzed through both effective and functional approaches, to further specify the brain areas involved in object orientation perception, as well as their coupling. Meeting abstract presented at VSS 2018

Considerable recent evidence indicates that angular gyrus dysfunction in humans does not result in amnesia, but does impair a number of aspects of episodic memory. Patients with parietal lobe lesions have been reported to exhibit a deficit when freely recalling autobiographical events from their pasts, but can remember details of the events when recall is cued by specific questions. In apparent contradiction, inhibitory brain stimulation targeting angular gyrus in healthy volunteers has been found to have no effect on free recall or cued recall of word pairs. The present study sought to resolve this inconsistency by testing free and cued recall of both autobiographical memories and word-pair memories in the same healthy male and female human participants following continuous theta burst stimulation (cTBS) of angular gyrus and a vertex control location. Angular gyrus cTBS resulted in a selective reduction in the free recall, but not cued recall, of autobiographical memories, whereas free and cued recall of word-pair memories were unaffected. Additionally, participants reported fewer autobiographical episodes as being experienced from a first-person perspective following angular gyrus cTBS. The findings add to a growing body of evidence that a function of angular gyrus within the network of brain regions responsible for episodic recollection is to integrate memory features within an egocentric framework into the kind of first-person perspective representation that enables the subjective experience of remembering events from our personal pasts.SIGNIFICANCE STATEMENT In seeking to understand the role played by the angular gyrus region of parietal cortex in human memory, interpreting the often conflicting findings from neuroimaging and neuropsychology studies has been hampered by differences in anatomical specificity and localization between methods. In the present study, we address these limitations using continuous theta burst stimulation in healthy volunteers to disrupt function of angular gyrus and a vertex control region. With this method, we adjudicate between two competing theories of parietal lobe function, finding evidence that is inconsistent with an attentional role for angular gyrus in memory, supporting instead an account in terms of integrating memory features within an egocentric framework into a first-person perspective representation that enables the subjective experience of remembering.

Dopamine may be critical for processing the meaningful information content of observations, which drives belief updating about the (hidden) states of the world [1–3]. In human functional magnetic resonance imaging (fMRI) studies, neuronal activity encoding the magnitude of belief updates has been detected in dopamine-rich midbrain regions, namely the substantia nigra and ventral tegmental area (SN/VTA) [1-3]. Yet, there is no direct evidence linking dopamine function to belief updating signals in humans. This question is relevant to our understanding of the relationship between dopamine, aberrant inference and the generation of psychotic symptoms such as paranoia [4]. We investigated the dopaminergic basis of belief update signals using fMRI and positron emission tomography (PET) imaging. 39 healthy participants (mean age 26y, 22 male) performed an fMRI task that decomposed sensory information into information-theoretic surprise (unexpected but not necessarily meaningful information content) and Bayesian surprise (magnitude of belief updates due to meaningful information) [1]. A Bayesian observer computational model was fitted to participants' behavior, allowing the derivation of trial-wise regressors for information-theoretic and Bayesian surprise for fMRI analysis. 36 participants also had a [11C]-(+)-4-propyl-9-hydroxy-naphthoxazine ([11C]-(+)-PHNO) PET scan to quantify dopamine-2/3 receptor (D2/3R) availability. 17 participants additionally had a second [11C]-(+)-PHNO PET scan 3hrs following 0.5mg/kg oral dexamphetamine, to quantify striatal dopamine release capacity. Our analyses focused on the relationship between measures of neuronal activity (fMRI) and dopamine function (PET) in the bilateral SN/VTA complex and ventral striatum (VS). Our computational model closely predicted participant behaviour (R2= .67), and there was a negative correlation between subclinical paranoid thoughts and the degree to which participant behavior approximated the predictions of an ideal Bayesian observer (rho = -0.60, P<0.001). The magnitude of belief updates (Bayesian surprise) was encoded in the SN/VTA and VS (both PFWE<0.05), putatively reflecting dopaminergic activity. Neuronal activation encoding Bayesian surprise was inversely correlated with D2/3R availability in the SN/VTA (rho = -0.43, P=0.009), consistent with studies indicating that midbrain D2/3R inhibit dopamine neurons [5]. Moreover, neuronal activation encoding Bayesian surprise was inversely related to dopamine release capacity in the VS (rho = -0.66, P=0.005), indicating that subjects with high dopamine release capacity showed blunted striatal activation in response to belief-changing information (as is also found in schizophrenia [4]). Neuronal activation encoding information-theoretic surprise was not present in the SN/VTA or VS, and showed no association with dopaminergic measures. Our results indicate that the SN/VTA and VS are involved in processing the meaningful information of observations, as reflected in Bayesian belief updating [1-3]. Crucially, we provide direct evidence that neuronal signals encoding (unsigned) belief updates depend on dopaminergic activity, at variance with classical interpretations of dopamine function in terms of (signed) reward prediction errors. Moreover, our results suggest that a reduced behavioural sensitivity to meaningful information is related to subclinical paranoia. Together, these results are highly relevant for dopaminergic theories of psychosis, such as the aberrant salience hypothesis, which posit that mesostriatal dopamine dysfunction results in the formation of false beliefs about the causes of sensory information, leading to hallucinations and delusions [4].

Normative models of human cognition often appeal to Bayesian filtering, which provides optimal online estimates of unknown or hidden states of the world, based on previous observations. However, in many cases it is necessary to optimise beliefs about sequences of states rather than just the current state. Importantly, Bayesian filtering and sequential inference strategies make different predictions about beliefs and subsequent choices, rendering them behaviourally dissociable. Taking data from a probabilistic reversal task we show that subjects' choices provide strong evidence that they are representing short sequences of states. Between-subject measures of this implicit sequential inference strategy had a neurobiological underpinning and correlated with grey matter density in prefrontal and parietal cortex, as well as the hippocampus. Our findings provide, to our knowledge, the first evidence for sequential inference in human cognition, and by exploiting between-subject variation in this measure we provide pointers to its neuronal substrates.

Neural models describe brain activity at different scales, ranging from single cells to whole brain networks. Here, we attempt to reconcile models operating at the microscopic (compartmental) and mesoscopic (neural mass) scales to analyse data from microelectrode recordings of intralaminar neural activity. Although these two classes of models operate at different scales, it is relatively straightforward to create neural mass models of ensemble activity that are equipped with priors obtained after fitting data generated by detailed microscopic models. This provides generative (forward) models of measured neuronal responses that retain construct validity in relation to compartmental models. We illustrate our approach using cross spectral responses obtained from V1 during a visual perception paradigm that involved optogenetic manipulation of the basal forebrain. We find that the resulting neural mass model can distinguish between activity in distinct cortical layers – both with and without optogenetic activation – and that cholinergic input appears to enhance (disinhibit) superficial layer activity relative to deep layers. This is particularly interesting from the perspective of predictive coding, where neuromodulators are thought to boost prediction errors that ascend the cortical hierarchy.

This paper offers an active inference account of choice behaviour and learning. It focuses on the distinction between goal-directed and habitual behaviour and how they contextualise each other. We show that habits emerge naturally (and autodidactically) from sequential policy optimisation when agents are equipped with state-action policies. In active inference, behaviour has explorative (epistemic) and exploitative (pragmatic) aspects that are sensitive to ambiguity and risk respectively, where epistemic (ambiguity-resolving) behaviour enables pragmatic (reward-seeking) behaviour and the subsequent emergence of habits. Although goal-directed and habitual policies are usually associated with model-based and model-free schemes, we find the more important distinction is between belief-free and belief-based schemes. The underlying (variational) belief updating provides a comprehensive (if metaphorical) process theory for several phenomena, including the transfer of dopamine responses, reversal learning, habit formation and devaluation. Finally, we show that active inference reduces to a classical (Bellman) scheme, in the absence of ambiguity.

This article describes a process theory based on active inference and belief propagation. Starting from the premise that all neuronal processing (and action selection) can be explained by maximizing Bayesian model evidence—or minimizing variational free energy—we ask whether neuronal responses can be described as a gradient descent on variational free energy. Using a standard (Markov decision process) generative model, we derive the neuronal dynamics implicit in this description and reproduce a remarkable range of well-characterized neuronal phenomena. These include repetition suppression, mismatch negativity, violation responses, place-cell activity, phase precession, theta sequences, theta-gamma coupling, evidence accumulation, race-to-bound dynamics, and transfer of dopamine responses. Furthermore, the (approximately Bayes’ optimal) behavior prescribed by these dynamics has a degree of face validity, providing a formal explanation for reward seeking, context learning, and epistemic foraging. Technically, the fact that a gradient descent appears to be a valid description of neuronal activity means that variational free energy is a Lyapunov function for neuronal dynamics, which therefore conform to Hamilton’s principle of least action.