Resonator circuits: a neural network for e ciently solving factorization problems

For the brain to make sense of knowledge, it must not only be able to represent features about sensory inputs, but also it must be able to store and manipulate this information within data structures. Connectionist theories have long been criticized because it is hard to imagine how complex, hierarchical structures could be represented and processed by neural networks. The connectionist literature proposed “vector-symbolic architectures” (VSA) to solve this problem. Such models define operations between vectors that are dimensionality-preserving. These define an algebra that can be used for building hierarchical representations and computing with data structures (4; 1; 3). In the VSA literature, however, deep hierarchical relationships for solving computational problems are rarely utilized. This is because the problem of simultaneous inference, or factorization, arises. Previous solutions required enumerating an unwieldy combinatoric search to solve the inference problem. In this work, we detail an algorithm that makes computation with compound distributed representations significantly more practical, because it can e ciently solve factorization problems without directly enumerating all combinations of components. Rather, the principle of superposition allows a set of recurrent neural networks, called a “resonator circuit”, to iteratively search through the space to resolve factors in a compound representation. The resonator circuit is demonstrated in a scene understanding task, where the system is asked to identify and localize simple objects. In the connectionist literature, a key issue emerged that made the creation and manipulation of symbols and hierarchical data structures challenging. The problem was related to the formation of conjunctive representations. How could one represent the conjunction of features, such as ‘red triangle’, in a fashion so that they also would not get confused with another conjunction, such as ‘green square’. One solution was to have a neuron that would only fire to the presence of a particular combination of features. However, this requires a neuron for every combination of features, and this is problematic if one desires to represent deep hierarchical structure. This is because the dimensionality of the representation grows exponentially with increasing depth. Another solution is provided by a family of models known collectively as Vector-Symbolic Architectures (VSA) (4; 1). Building on the concept of reduced representations (2), VSAs utilize the properties of high-dimensional random vectors to define an algebra that allows one to create hierarchical data-structures in a distributed representation with fixed dimensionality. This requires defining a new type of operation between neural populations, called the “binding” operation, which is typically a form of multiplication between high-dimensional vectors. In the VSA literature, however, deep hierarchical relationships for solving computational problems are rarely utilized. This is because the problem of simultaneous inference, or factorization, arises. In this situation, an unknown conjunction of features needs to be inverted, but there is no knowledge of any feature. The VSA representation of ‘red triangle’ looks nothing like the representation for ‘red’ or for ‘triangle’. Finding the individual features of a conjunction amounts to a factorization problem. In this work, we detail an algorithm that makes computation with compound distributed representations significantly more practical, because it can e ciently solve factorization problems. The principle of superposition allows a set of recurrent neural networks, called a “resonator circuit”, to iteratively search through a combinatoric space to resolve factors in a compound representation. The resonator circuit is demonstrated with a scene understanding task. In these scenes (Fig. 1A), three objects appear that have compositional properties. Each object is a discrete letter with a discrete color and a continuous location. The scene is encoded into a distributed complex-valued high-dimensional vector using FHRR (4). Each color channel (RGB) is assigned an FHRR vector stored in the codebook R = [r,g,b] 2 CN⇥3, where N = 20, 000. FHRR vectors for horizontal position and vertical position, h 2 CN and v 2 CN , serve to encode location information. Each pixel location is indexed by exponentiating the vector by the location scalars x and y, and the vectors hx, vy are stored in codebooks H and V for each pixel value of x and y. The scene is thus encoded into a high-dimensional vector: