We introduce LayerLock, a simple yet effective approach for self-supervised visual representation learning, that gradually transitions from pixel to latent prediction through progressive layer freezing. First, we make the observation that during training of video masked-autoencoding (MAE) models, ViT layers converge in the order of their depth: shallower layers converge early, deeper layers converge late. We then show that this observation can be exploited to accelerate standard MAE by progressively freezing the model according to an explicit schedule, throughout training. Furthermore, this same schedule can be used in a simple and scalable approach to latent prediction that does not suffer from "representation collapse". We apply our proposed approach, LayerLock, to large models of up to 4B parameters with results surpassing those of non-latent masked prediction on the 4DS perception suite.

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99 as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75 categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models.

Scaling has not yet been convincingly demonstrated for pure self-supervised learning from video. However, prior work has focused evaluations on semantic-related tasks x2013 action classification, ImageNet classification, etc. In this paper we focus on evaluating self-supervised learning on non-semantic vision tasks that are more spatial (3D) and temporal (+1D = 4D), such as camera pose estimation, point and object tracking, and depth estimation. We show that by learning from very large video datasets, masked auto-encoding (MAE) with transformer video models actually scales, consistently improving performance on these 4D tasks, as model size increases from 20M all the way to the largest by far reported self-supervised video model x2013 22B parameters. Rigorous apples-to-apples comparison with many recent image and video models demonstrates the benefits of scaling 4D representations. Pretrained models are available at https://github.com/google-deepmind/representations4d .

To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent's location/viewpoint, as the two variables jointly give rise to the agent's observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input alone: a set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well as a set of "frame" latents, corresponding to global time-varying elements such as viewpoint. This factorization of latents allows our model, SIMONe, to represent object attributes in an allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajectories as time-abstracted, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.

Despite decades of research, little is known about how people visually perceive object shape. We hypothesize that a promising approach to shape perception is provided by a "visual perception as Bayesian inference" framework which augments an emphasis on visual representation with an emphasis on the idea that shape perception is a form of statistical inference. Our hypothesis claims that shape perception of unfamiliar objects can be characterized as statistical inference of 3D shape in an object-centered coordinate system. We describe a computational model based on our theoretical framework, and provide evidence for the model along two lines. First, we show that, counterintuitively, the model accounts for viewpoint-dependency of object recognition, traditionally regarded as evidence against people's use of 3D object-centered shape representations. Second, we report the results of an experiment using a shape similarity task, and present an extensive evaluation of existing models' abilities to account for the experimental data. We find that our shape inference model captures subjects' behaviors better than competing models. Taken as a whole, our experimental and computational results illustrate the promise of our approach and suggest that people's shape representations of unfamiliar objects are probabilistic, 3D, and object-centered. (PsycINFO Database Record

The format of high-level object representations in temporal-occipital cortex is a fundamental and as yet unresolved issue. Here we use fMRI to show that human lateral occipital cortex (LOC) encodes novel 3-D objects in a multisensory and part-based format. We show that visual and haptic exploration of objects leads to similar patterns of neural activity in human LOC and that the shared variance between visually and haptically induced patterns of BOLD contrast in LOC reflects the part structure of the objects. We also show that linear classifiers trained on neural data from LOC on a subset of the objects successfully predict a novel object based on its component part structure. These data demonstrate a multisensory code for object representations in LOC that specifies the part structure of objects.

In the past few years, deep convolutional neural networks (CNNs) trained on large image data sets have shown impressive visual object recognition performances. Consequently, these models have attracted the attention of the cognitive science community. Recent studies comparing CNNs with neural data from cortical area IT suggest that CNNs may—in addition to providing good engineering solutions—provide good models of biological visual systems. Here, we report evidence that CNNs are, in fact, not good models of human visual perception. We show that a 3D shape inference model explains human performance on an object shape similarity task better than CNNs. We argue that deep neural networks trained on large amounts of image data to maximize object recognition performance do not provide adequate models of human vision.

The world is multimodal.1 We sense our environments using inputs from multiple sensory modalities. Similarly, digital information is increasingly available through multiple media. In this extended abstract, we present a general computational framework for understanding multimodal learning and perception that builds on the analysis-by-synthesis approach [2, 3]. The analysis-by-synthesis approach makes use of generative models describing how causes in the environment give rise to percepts. It achieves an analysis of the input as a synthesis of these causes by inverting this generative model. This approach has been successfully applied in a wide range of fields from vision[2] to language[3]. Our framework is a natural extension of this approach to multimodal perception. From the perspective of Marr’s tri-level hypothesis2, our framework can be understood as a computational level analysis of multimodal perception. It offers a conceptual analysis of multimodal perception that enables a unified treatment of a diverse set of questions. As a proof of concept of our framework, we apply it to object shape perception via visual and haptic modalities, and show that it captures people’s behavior in this task with high accuracy.

People learn modality-independent, conceptual representations from modality-specific sensory signals. Here, we hypothesize that any system that accomplishes this feat will include three components: a representational language for characterizing modality-independent representations, a set of sensory-specific forward models for mapping from modality-independent representations to sensory signals, and an inference algorithm for inverting forward models—that is, an algorithm for using sensory signals to infer modality-independent representations. To evaluate this hypothesis, we instantiate it in the form of a computational model that learns object shape representations from visual and/or haptic signals. The model uses a probabilistic grammar to characterize modality-independent representations of object shape, uses a computer graphics toolkit and a human hand simulator to map from object representations to visual and haptic features, respectively, and uses a Bayesian inference algorithm to infer modality-independent object representations from visual and/or haptic signals. Simulation results show that the model infers identical object representations when an object is viewed, grasped, or both. That is, the model’s percepts are modality invariant. We also report the results of an experiment in which different subjects rated the similarity of pairs of objects in different sensory conditions, and show that the model provides a very accurate account of subjects’ ratings. Conceptually, this research significantly contributes to our understanding of modality invariance, an important type of perceptual constancy, by demonstrating how modality-independent representations can be acquired and used. Methodologically, it provides an important contribution to cognitive modeling, particularly an emerging probabilistic language-of-thought approach, by showing how symbolic and statistical approaches can be combined in order to understand aspects of human perception.

Transfer of object shape knowledge across visual and haptic modalities Goker Erdogan (gerdogan@bcs.rochester.edu) Ilker Yildirim (iyildirim@bcs.rochester.edu) Robert A. Jacobs (robbie@bcs.rochester.edu) Department of Brain & Cognitive Sciences, University of Rochester, Rochester, NY 14627 USA Abstract We investigate the hypothesis that multisensory representa- tions mediate the crossmodal transfer of shape knowledge across visual and haptic modalities. In our experiment, par- ticipants rated the similarities of pairs of synthetic 3-D objects in visual, haptic, cross-modal, and multisensory settings. Our results offer two contributions. First, we provide evidence for a single multisensory shape representation common to both vi- sual and haptic modalities. Second, our analyses suggest that these representations are part-based, representing objects as compositions of subparts. Keywords: multisensory perception, visual perception, haptic perception, object perception, shape representation Introduction Imagine the following simple scenario. You see an object, and later you are asked to find that object among a set of objects by using only your sense of touch. How might the visual information about the object be transferred to the hap- tic modality to achieve object recognition in this task? One hypothesis is that haptic input is mapped to a visual shape rep- resentation, maybe like a form of visual imagery, and object recognition is achieved in this visual shape space. It is also plausible that an analogous haptic imagery process is at play, and object perception takes place in a haptic shape space. The third alternative and the hypothesis we are arguing for is the Multisensory Hypothesis. This hypothesis states that peo- ple use sensory representations of objects to infer amodal or multisensory representations characterizing objects’ intrinsic properties, and object perception is mediated by these multi- sensory representations. The first question we address is exactly this question of whether people use modality-specific (vision-specific and haptic-specific) object representations or whether they use modality-independent (multisensory) representations. The second question concerns the fine-grained structure of object representations. If multisensory representations underlie our object perception, what can we say about the nature of these representations? An influential hypothesis in the cognitive science literature is that object representations are part-based, meaning that objects are represented in terms of their parts and the spatial relations among these parts (Marr & Nishi- hara, 1978; Biederman, 1987). To address these questions, we collected similarity judg- ments from people about pairs of novel objects when objects are viewed, when objects are grasped, when one object is viewed and the other is grasped, and when both objects are viewed and grasped. We found that participants gave similar ratings in all experimental conditions, providing evidence for the existence of multisensory object representations. More- over, our analyses suggest that our multisensory object repre- sentations are part-based. Related Research Previous studies provide behavioral and neurophysiological evidence for the existence of multisensory representations. Quiroga (2012), for example, reported the existence of “con- cept cells” which are neurons that respond selectively to par- ticular persons or objects regardless of the modality used to sense those persons or objects. One neuron, for instance, re- sponded when a person viewed an image of the television host Oprah Winfrey, viewed her written name, or heard her spoken name (Quiroga, Kraskov, Koch, & Fried, 2009). Additionally, brain imaging studies (Amedi, Jacobson, Hendler, Malach, & Zohary, 2002) show that LOtv, a neural region within the hu- man lateral occipital complex, is activated both by viewing and touching objects. Behavioral results are consistent with neurophysiological findings. Konkle, Wang, Hayward, and Moore (2009) re- ported that motion aftereffects transferred between vision and touch—when adapted to visual motion in a certain direc- tion, people felt tactile motion aftereffects in the opposite direction, and vice versa. Such a finding provides strong evidence for a shared representation underlying visual and tactile motion perception. In another study, Lacey, Pappas, Kreps, Lee, and Sathian (2009) found that subjects initially showed viewpoint-dependent object recognition in both vi- sual and haptic modalities. However, following unimodal training with either visual or haptic stimuli, people’s object recognition performances became viewpoint-independent in both modalities. A set of studies by Wallraven, B¨ulthoff, and colleagues also provide evidence for common object rep- resentations underlying visual and haptic object perception. In these studies (Cooke, J¨akel, Wallraven, & B¨ulthoff, 2007; Gaissert, Wallraven, & B¨ulthoff, 2010; Gaissert, B¨ulthoff, & Wallraven, 2011; Gaissert & Wallraven, 2012), subjects pro- vided similarity judgments for different sets of objects, both artificial and natural, in vision alone, haptic alone, and vision- haptic conditions. It was found that subjects’ similarity rat- ings were similar in all three sensory conditions, thereby sug- gesting that these ratings were based on shared, multisensory representations. The experiment reported here uses a similar procedure, but extends this earlier work by focusing on the part-based nature of these representations.

This paper presents a computational model of concept learning using Bayesian inference for a grammatically structured hypothesis space, and test the model on multisensory (visual and haptics) recognition of 3D objects. The study is performed on a set of artificially generated 3D objects known as fribbles, which are complex, multipart objects with categorical structures. The goal of this work is to develop a working multisensory representational model that integrates major themes on concepts and concepts learning from the cognitive science literature. The model combines the representational power of a probabilistic generative grammar with the inferential power of Bayesian induction.