The visual environment is filled with complex, multi-dimensional objects that vary in their value to an observer's current goals. When faced with multi-dimensional stimuli, humans may rely on biases to learn to select those objects that are most valuable to the task at hand. Here, we show that decision making in a complex task is guided by the sparsity bias: the focusing of attention on a subset of available features. Participants completed a gambling task in which they selected complex stimuli that varied randomly along three dimensions: shape, color, and texture. Each dimension comprised three features (e.g., color: red, green, yellow). Only one dimension was relevant in each block (e.g., color), and a randomly-chosen value ranking determined outcome probabilities (e.g., green > yellow > red). Participants were faster to respond to infrequent probe stimuli that appeared unexpectedly within stimuli that possessed a more valuable feature than to probes appearing within stimuli possessing a less valuable feature. Event-related brain potentials recorded during the task provided a neurophysiological explanation for sparsity as a learning-dependent increase in optimal attentional performance (as measured by the N2pc component of the human event-related potential) and a concomitant learning-dependent decrease in prediction errors (as measured by the feedback-elicited reward positivity). Together, our results suggest that the sparsity bias guides human reinforcement learning in complex environments.

The scientific literature on automated gait analysis for human recognition has grown dramatically over the past 15 years. A number of sensing modalities including those based on vision, sound, pressure, and accelerometry have been used to capture gait information. For each of these modalities, a number of methods have been developed to extract and compare human gait information, resulting in different sets of features. This paper provides an extensive overview of the various types of features that have been utilized for each sensing modality and their relationship to the appearance and biomechanics of gait. The features considered in this work include (a) static and dynamic (temporal) features; (b) model-based and model-free visual features; (c) ground reaction force-based and finely resolved underfoot pressure features; (d) wearable sensor features; and (e) acoustic features. We also review the factors that impact gait recognition, and discuss recent work on gait spoofing and obfuscation. Finally, we enumerate the challenges and open problems in the field of gait recognition.

Rule-based machine translation is more data efficient than the big data-based machine translation approaches, making it appropriate for languages with low bilingual corpus resources – i.e., minority languages. However, the rule-based approach has declined in popularity relative to its big data cousins primarily because of the extensive training and labour required to define the language rules. To address this, we present a semantic representation that 1) treats all bits of meaning as individual concepts that 2) modify or further specify one another to build a network that relates entities in space and time. Also, the representation can 3) encapsulate propositions and thereby define concepts in terms of other concepts, supporting the abstraction of underlying linguistic and ontological details. These features afford an exact, yet intuitive semantic representation aimed at handling the great variety in language and reducing labour and training time. The proposed natural language generation, parsing, and translation strategies are also amenable to probabilistic modeling and thus to learning the necessary rules from example data.

Underfoot pressure is becoming a fully-fledged gait biometric modality with the advent of modular, high-resolution sensing floor tiles because they capture rich gait signals without requiring the subject's cooperation. The literature on underfoot pressure-based gait recognition has posed a variety of features for both the barefoot and shod walking cases. In this paper, we compare and combine a large number of these with three use case scenarios in mind: 1) barefoot gait recognition and shod-foot recognition when the shoe used in training is 2) the same and 3) different from the test shoe. Our results show a difference in the types of features most useful in each scenario and their combination yields barefoot recognition (EER of 2.2%) that approaches the performance levels of biometrics currently in use, a good showing for this emerging technology.

Biological systems are capable of learning that certain stimuli are valuable while ignoring the many that are not, and thus perform feature selection. In machine learning, one effective feature selection approach is the least absolute shrinkage and selection operator (LASSO) form of regularization, which is equivalent to assuming a Laplacian prior distribution on the parameters. We review how such Bayesian priors can be implemented in gradient descent as a form of weight decay, which is a biologically plausible mechanism for Bayesian feature selection. In particular, we describe a new prior that offsets or “raises” the Laplacian prior distribution. We evaluate this alongside the Gaussian and Cauchy priors in gradient descent using a generic regression task where there are few relevant and many irrelevant features. We find that raising the Laplacian leads to less prediction error because it is a better model of the underlying distribution. We also consider two biologically relevant online learning tasks, one synthetic and one modeled after the perceptual expertise task of Krigolson et al. (2009). Here, raising the Laplacian prior avoids the fast erosion of relevant parameters over the period following training because it only allows small weights to decay. This better matches the limited loss of association seen between days in the human data of the perceptual expertise task. Raising the Laplacian prior thus results in a biologically plausible form of Bayesian feature selection that is effective in biologically relevant contexts.

When retrospective revaluation phenomena (e.g., unovershadowing: AB+, then A−, then test B) were discovered, simple elemental models were at a disadvantage because they could not explain such phenomena. Extensions of these models and novel models appealed to within-compound associations to accommodate these new data. Here, we present an elemental, neural network model of conditioning that explains retrospective revaluation apart from within-compound associations. In the model, previously paired stimuli (say, A and B, after AB+) come to activate similar ensembles of neurons, so that revaluation of one stimulus (A−) has the opposite effect on the other stimulus (B) through changes (decreases) in the strength of the inhibitory connections between neurons activated by B. The ventral striatum is discussed as a possible home for the structure and function of the present model.

Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.

In reinforcement learning, the goal is to seek rewards and avoid punishments. A simple scalar captures the value of a state or of taking an action, where expected future rewards increase and punishments decrease this quantity. Naturally an agent should learn to predict this quantity to take beneficial actions, and many value function approximators exist for this purpose. In the present work, however, we show how value function approximators can cause confusion between predictors of an outcome of one valence (e.g., a signal of reward) and the inhibitor of the opposite valence (e.g., a signal canceling expectation of punishment). We show this to be a problem for both linear and non-linear value function approximators, especially when the amount of data (or experience) is limited. We propose and evaluate a simple resolution: to instead predict reward and punishment values separately, and rectify and add them to get the value needed for decision making. We evaluate several function approximators in this slightly different value function approximation architecture and show that this approach is able to circumvent the confusion and thereby achieve lower value-prediction errors.

Biological organisms need to accurately infer which features of their environment predict future rewards and punishments for survival sake. This problem resembles linear regression, which finds parameter values expressing the linear relationship between features and an outcome. The least mean squares regression method generalizes well when there is little system noise and at least as many training data points (experiences) as input features. When this is not the case, feature selection may be applied to eliminate irrelevant features and improve generalization. Here, we show a biologically plausible approach to feature selection that computes the maximum likelihood estimate of Pearl's “Noisy OR” model. We show that this results in highlighting the features that are most correlated with the outcome at the expense of the least correlated. We extend this “relative correlation” approach to represent global inhibitory features and show that as additive noise and the number of irrelevant features are increased, relative correlation leads to substantially less prediction error on test data than does least means squares in a simple linear regression task. We demonstrate how relative correlation can be implemented in a dual pathway neural network and discuss some similarities between it and the basal ganglia.

Hyperactivity is a key symptom in those diagnosed with Attention Deficit Hyperactivity Disorder. In the present work, we model hyperactivity in terms of a twoarmed bandit task from Reinforcement Learning, where initial state-values are set abnormally high. Extinction of these state-values when neither action is very rewarding induces repetitive switching between actions over a series of trials with a frequency that is proportional to the initial state-value. Here we propose that although setting initial values may be a useful exploration strategy, switching can become overly frequent or “hyperactive” when they are set too high. Keywords— ADHD, Hyperactivity, Reinforcement Learning, Exploration Strategy

The striatum has long been implicated in reinforcement learning and has been suggested by several neurophysiological studies as the substrate for encoding the reward value of stimuli. Reward prediction error (RPE) has been used in several basal ganglia models as the underlying learning signal, which leads to Pavlovian conditioning abilities that can be simulated by the Rescorla-Wagner model.Lateral inhibition between striatal projection neurons was once thought to have a winner-take-all function, useful in selecting between possible actions. However, it has been noted that the necessary reciprocal connections for this interpretation are too few, and the relative strength of these synaptic connections is weak. Still, modeling studies show that lateral inhibition does have an overall suppression effect on striatal activity and may play an important role in striatal processing.Neurophysiological recordings show task-relevant ensembles of responsive neurons at specific points in a behavioral paradigm (Barnes et al., 2005), which appear to be induced by lateral inhibition (see Ponzi and Wickens, 2010). We have developed a similarly responding, RPE-based model of the striatum by incorporating lateral inhibition. Model neurons are assigned to either the direct or the indirect pathway but lateral connections occur within and between these groups, leading to competition between both the individual neurons and their pathways. We successfully applied this model to the simulation of Pavlovian phenomena beyond those of the Rescorla-Wagner model, including negative patterning, unovershadowing, and external inhibition.

The field of Reinforcement Learning (RL) in machine learning relates significantly to the domains of classical and instrumental conditioning in psychology, which give an understanding of biology’s approach to RL. In recent years, there has been a thrust to correlate some machine learning RL algorithms with brain structure and function, a benefit to both fields. Our focus has been on one such structure, the striatum, from which we have built a general model. In machine learning terms, this model is equivalent to a value-function approximator (VFA) that learns according to Temporal Difference error. In keeping with a biological approach to RL, the present work seeks to evaluate the robustness of this striatum-based VFA using biological criteria. We selected five classical conditioning tests to expose the learning accuracy and efficiency of the VFA for simple state-value associations. Manually setting the VFA’s many parameters to reasonable values, we characterize it by varying each parameter independently and repeatedly running the tests. The results show that this VFA is both capable of performing the selected tests and is quite robust to changes in parameters. Test results also reveal aspects of how this VFA encodes reward value.

Automatic target recognition (ATR) of objects in side scan sonar imagery typically employs image processing techniques (e.g. segmentation, Fourier transform) to extract features describing the objects. The features are used to discriminate between sea floor clutter and targets (e.g. sea mines). These methods are typically developed for a specific sonar, and are computationally intensive. The present work used the Restricted Boltzmann Machine (RBM) to discriminate between images of targets and clutter, achieving a 90

The purpose of this work is to identify features that can successfully classify objects that appear in sidescan sonar imagery as belonging to one of 3 mine classes or a non-mines class. Naval mine hunters identify mines in the imagery primarily using the size and shape of signature bright and dark regions, referred to as the highlight and shadow respectively. A data set of real sidescan sonar imagery was provided by Defence Research and Development Canada. Many feature sets, some novel, were tested for their ability to discriminate between mines and non-mines, as well as between the different types of mines (cylinder, truncated cone, and sphere) and the non-mines. Classification was performed using a linear discriminant function. Ultimately, several good features representing certain size and shape qualities were identified. These include measures of object height, shadow elongation, shadow 2-rotational symmetry, and particular shadow shapes (using Fourier descriptors).