We combine extant theories of evidence accumulation and multi-modal integration to develop an integrated framework for modeling multimodal integration as a process that unfolds in real time. Many studies have formulated sensory processing as a dynamic process where noisy samples of evidence are accumulated until a decision is made. However, these studies are often limited to a single sensory modality. Studies of multimodal stimulus integration have focused on how best to combine different sources of information to elicit a judgment. These studies are often limited to a single time point, typically after the integration process has occurred. We address these limitations by combining the two approaches. Experimentally, we present data that allow us to study the time course of evidence accumulation within each of the visual and auditory domains as well as in a bimodal condition. Theoretically, we develop a new Averaging Diffusion Model in which the decision variable is the mean rather than the sum of evidence samples and use it as a base for comparing three alternative models of multimodal integration, allowing us to assess the optimality of this integration. The outcome reveals rich individual differences in multimodal integration: while some subjects’ data are consistent with adaptive optimal integration, reweighting sources of evidence as their relative reliability changes during evidence integration, others exhibit patterns inconsistent with optimality.

When people make decisions, do they give equal weight to evidence arriving at different times? A recent study (Kiani et al., 2008) using brief motion pulses (superimposed on a random moving dot display) reported a primacy effect: pulses presented early in a motion observation period had a stronger impact than pulses presented later. This observation was interpreted as supporting the bounded diffusion (BD) model and ruling out models in which evidence accumulation is subject to leakage or decay of early-arriving information. We use motion pulses and other manipulations of the timing of the perceptual evidence in new experiments and simulations that support the leaky competing accumulator (LCA) model as an alternative to the BD model. While the LCA does include leakage, we show that it can exhibit primacy as a result of competition between alternatives (implemented via mutual inhibition), when the inhibition is strong relative to the leak. Our experiments replicate the primacy effect when participants must be prepared to respond quickly at the end of a motion observation period. With less time pressure, however, the primacy effect is much weaker. For 2 (out of 10) participants, a primacy bias observed in trials where the motion observation period is short becomes weaker or reverses (becoming a recency effect) as the observation period lengthens. Our simulation studies show that primacy is equally consistent with the LCA or with BD. The transition from primacy-to-recency can also be captured by the LCA but not by BD. Individual differences and relations between the LCA and other models are discussed.

In perceptual decision-making, ideal decision-makers should bias their choices toward alternatives associated with larger rewards, and the extent of the bias should decrease as stimulus sensitivity increases. When responses must be made at different times after stimulus onset, stimulus sensitivity grows with time from zero to a final asymptotic level. Are decision makers able to produce responses that are more biased if they are made soon after stimulus onset, but less biased if they are made after more evidence has been accumulated? If so, how close to optimal can they come in doing this, and how might their performance be achieved mechanistically? We report an experiment in which the payoff for each alternative is indicated before stimulus onset. Processing time is controlled by a "go" cue occurring at different times post stimulus onset, requiring a response within msec. Reward bias does start high when processing time is short and decreases as sensitivity increases, leveling off at a non-zero value. However, the degree of bias is sub-optimal for shorter processing times. We present a mechanistic account of participants' performance within the framework of the leaky competing accumulator model [1], in which accumulators for each alternative accumulate noisy information subject to leakage and mutual inhibition. The leveling off of accuracy is attributed to mutual inhibition between the accumulators, allowing the accumulator that gathers the most evidence early in a trial to suppress the alternative. Three ways reward might affect decision making in this framework are considered. One of the three, in which reward affects the starting point of the evidence accumulation process, is consistent with the qualitative pattern of the observed reward bias effect, while the other two are not. Incorporating this assumption into the leaky competing accumulator model, we are able to provide close quantitative fits to individual participant data.

Single neurons in cortical area LIP are known to carry information relevant to both sensory and value-based decisions that are reported by eye movements. It is not known, however, how sensory and value information are combined in LIP when individual decisions must be based on a combination of these variables. To investigate this issue, we conducted behavioral and electrophysiological experiments in rhesus monkeys during performance of a two-alternative, forced-choice discrimination of motion direction (sensory component). Monkeys reported each decision by making an eye movement to one of two visual targets associated with the two possible directions of motion. We introduced choice biases to the monkeys' decision process (value component) by randomly interleaving balanced reward conditions (equal reward value for the two choices) with unbalanced conditions (one alternative worth twice as much as the other). The monkeys' behavior, as well as that of most LIP neurons, reflected the influence of all relevant variables: the strength of the sensory information, the value of the target in the neuron's response field, and the value of the target outside the response field. Overall, detailed analysis and computer simulation reveal that our data are consistent with a two-stage drift diffusion model proposed by Diederich and Bussmeyer [1] for the effect of payoffs in the context of sensory discrimination tasks. Initial processing of payoff information strongly influences the starting point for the accumulation of sensory evidence, while exerting little if any effect on the rate of accumulation of sensory evidence.

Performance on serial tasks is influenced by first-and higher-order sequential effects, respectively, due to the immediately previous and earlier trials. As response-to-stimulus interval (RSI) increases, the pattern of reaction times transits from a benefit-only mode, traditionally ascribed to automatic facilitation (AF), to a cost-benefit mode, due to strategic expectancy (SE). To illuminate the sources of such effects, we develop a connectionist network of two mutually inhibiting neural decision units subject to feedback from previous trials. A study of separate biasing mechanisms shows that residual decision unit activity can lead to only first-order AF, but higher-order AF can result from strategic priming mediated by conflict monitoring, which we instantiate in two distinct versions. A further mechanism mediates expectation-related biases that grow during RSI toward saturation levels determined by weighted repetition (or alternation) sequence lengths. Equipped with these mechanisms, the network, consistent with known neurophysiology, accounts for several sets of behavioral data over a wide range of RSIs. The results also suggest that practice speeds up all the mechanisms rather than adjusting their relative strengths.

Neurons in diverse brain areas can respond to the interruption of a regular stimulus pattern by firing bursts of spikes. Here we describe a simple model, which permits such responses to periodic stimuli over a substantial frequency range. Focusing on the omitted stimulus response (OSR) in isolated retinas subjected to periodic patterns of dark flashes, we develop a biophysically-realistic model which accounts for resonances in ON bipolar cells. The bipolar cell terminal contains an LRC oscillator whose inductance is modulated by a transient calcium concentration, thus adjusting its resonant frequency to approximately match that of the stimulus. The model reproduces ganglion cell outputs, which sum the ON and OFF bipolar pathways, and it responds to omitted flashes with approximately constant latencies, as observed experimentally.

It is observed that retinae subjected to periodic patterns of light flashes in the range of 6-20 Hz can respond to an omitted flash by ganglion cell spikes after a precisely fixed amount of time [1]. We explore the mechanism underlying the omitted stimulus response (OSR) with emphasis on the function of ON bipolar cells. We model ON bipolar cell terminals with LRC circuits in which an inductor represents the voltagedependent conductances. Two models are developed and tested: a single adaptive oscillator model and a resonator bank model. In the former there is only one LRC oscillator whose inductance is determined by calcium concentration which effectively leads to the dependency of oscillator’s resonant frequency on stimulus frequency. In the latter multiple oscillators are involved which individually dominate the response at each stimulus frequency. The matching of terminal oscillation frequency with flash stimuli frequency implies a fixed latency from OSR to omitted flash as observed in experiments (See Fig. 1). For the resonator bank model the effects of gap junction coupling and synchronization are examined. We also compare model predictions and empirical observations in term of OSR size and latency when flash number and duration vary. We find that the adaptive oscillator model is more robust and can better encompass the frequency range in which OSR is observed, suggesting that calcium dynamics and the nonlinear conductance-voltage relation play important roles.

We study the dynamics and bifurcations of noise-free neurons coupled by gap junctions and inhibitory synapses, using both delayed delta functions and alpha functions to model the latter. We focus on the case of two cells, as in the studies of Chow and Kopell (2000) and Lewis and Rinzel (2003), but also show that stable asynchronous splay states exist for globally coupled networks of N cells dominated by subthreshold electrical coupling. Our results agree with those of Lewis and Rinzel (2003) in the weak coupling range, but our Poincaré map analysis yields more information about global behavior and domains of attraction, and we show that the explicit discontinuous maps derived using delayed delta functions compare well with the continuous history-dependent, implicitly-defined maps derived from alpha functions. We find that increased bias currents, super-threshold electrical coupling and synaptic delays promote synchrony, while sub-threshold electrical coupling and fast synapses promote asynchrony. We compare our analytical results with simulations of an ionic current model of spiking cells, and briefly discuss implications for stimulus response modes of locus coeruleus and for central pattern generators.

We review simple connectionist and firing rate models for mutually inhibiting pools of neurons that discriminate between pairs of stimuli. Both are two-dimensional nonlinear stochastic ordinary differential equations, and although they differ in how inputs and stimuli enter, we show that they are equivalent under state variable and parameter coordinate changes. A key parameter is gain: the maximum slope of the sigmoidal activation function. We develop piecewise-linear and purely linear models, and one-dimensional reductions to Ornstein–Uhlenbeck processes that can be viewed as linear filters, and show that reaction time and error rate statistics are well approximated by these simpler models. We then pose and solve the optimal gain problem for the Ornstein–Uhlenbeck processes, finding explicit gain schedules that minimize error rates for time-varying stimuli. We relate these to time courses of norepinephrine release in cortical areas, and argue that transient firing rate changes in the brainstem nucleus locus coeruleus may be responsible for approximate gain optimization.

There is increasing evidence from in vivo recordings in monkeys trained to respond to stimuli by making left- or rightward eye movements, that firing rates in certain groups of neurons in oculo-motor areas mimic drift-diffusion processes, rising to a (fixed) threshold prior to movement initiation. This supplements earlier observations of psychologists, that human reaction-time and error-rate data can be fitted by random walk and diffusion models, and has renewed interest in optimal decision-making ideas from information theory and statistical decision theory as a clue to neural mechanisms. We review results from decision theory and stochastic ordinary differential equations, and show how they may be extended and applied to derive explicit parameter dependencies in optimal performance that may be tested on human and animal subjects. We then briefly describe a biophysically-based model of a pool of neurons in locus coeruleus, a brainstem nucleus implicated in widespread norepinephrine release. This neurotransmitter can effect transient gain changes in cortical circuits of the type that the abstract drift-diffusion analysis requires. We also describe how optimal gain schedules can be computed in the presence of time-varying noisy signals. We argue that a rational account of how neural spikes give rise to simple behaviors is beginning to emerge.