The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.

General-purpose robots need a deep understanding of the physical world, advanced reasoning, and general and dexterous control. This report introduces the latest generation of the Gemini Robotics model family: Gemini Robotics 1.5, a multi-embodiment Vision-Language-Action (VLA) model, and Gemini Robotics-ER 1.5, a state-of-the-art Embodied Reasoning (ER) model. We are bringing together three major innovations. First, Gemini Robotics 1.5 features a novel architecture and a Motion Transfer (MT) mechanism, which enables it to learn from heterogeneous, multi-embodiment robot data and makes the VLA more general. Second, Gemini Robotics 1.5 interleaves actions with a multi-level internal reasoning process in natural language. This enables the robot to "think before acting" and notably improves its ability to decompose and execute complex, multi-step tasks, and also makes the robot's behavior more interpretable to the user. Third, Gemini Robotics-ER 1.5 establishes a new state-of-the-art for embodied reasoning, i.e., for reasoning capabilities that are critical for robots, such as visual and spatial understanding, task planning, and progress estimation. Together, this family of models takes us a step towards an era of physical agents-enabling robots to perceive, think and then act so they can solve complex multi-step tasks.

Recent advancements in large multimodal models have led to the emergence of remarkable generalist capabilities in digital domains, yet their translation to physical agents such as robots remains a significant challenge. This report introduces a new family of AI models purposefully designed for robotics and built upon the foundation of Gemini 2.0. We present Gemini Robotics, an advanced Vision-Language-Action (VLA) generalist model capable of directly controlling robots. Gemini Robotics executes smooth and reactive movements to tackle a wide range of complex manipulation tasks while also being robust to variations in object types and positions, handling unseen environments as well as following diverse, open vocabulary instructions. We show that with additional fine-tuning, Gemini Robotics can be specialized to new capabilities including solving long-horizon, highly dexterous tasks, learning new short-horizon tasks from as few as 100 demonstrations and adapting to completely novel robot embodiments. This is made possible because Gemini Robotics builds on top of the Gemini Robotics-ER model, the second model we introduce in this work. Gemini Robotics-ER (Embodied Reasoning) extends Gemini's multimodal reasoning capabilities into the physical world, with enhanced spatial and temporal understanding. This enables capabilities relevant to robotics including object detection, pointing, trajectory and grasp prediction, as well as multi-view correspondence and 3D bounding box predictions. We show how this novel combination can support a variety of robotics applications. We also discuss and address important safety considerations related to this new class of robotics foundation models. The Gemini Robotics family marks a substantial step towards developing general-purpose robots that realizes AI's potential in the physical world.

As the variety of media content increases, consumer services are being asked for more ways to access them. For a better consumer experience, more media attributes should be revealed to connect consumers with media content. Various methods have been studied to extract complicated contexts in media content like entity detection and image/video captioning. Among them, this paper focuses on the movie description. A movie description model manifests comprehensive descriptions of movies concerning story context. Thus, consumer services can provide rich information about movies to users. This paper proposes a novel movie description model with the story background to generate detailed descriptions. The story background entities are included in the movie script during the pre-production stage. These entities have a significant effect on the scene portrayed. We define the story background using the location and time information of the scenes. The models with shot structures extract the story background from a keyframe of each scene to generate the scene description. In experiments with the LSMDC dataset, the proposed model achieves 0.0141 of BLEU@4 and 0.1313 CIDEr, which is about 9% over the baselines. Qualitatively, the description generated through the proposed model provides richer contextual information compared to previous studies.

In this paper, we discuss an algorithm for the problem of undirected st-connectivity that is deterministic and log-space, namely that of Reingold within his 2008 paper "Undirected Connectivity in Log-Space". We further present a separate proof by Rozenman and Vadhan of USTCONN∈ L and discuss its similarity with Reingold's proof. Undirected st-connectively is known to be complete for the complexity class SL–problems solvable by symmetric, non-deterministic, log-space algorithms. Likewise, by Aleliunas et. al., it is known that undirected st-connectivity is within the RL complexity class, problems solvable by randomized (probabilistic) Turing machines with one-sided error in logarithmic space and polynomial time. Finally, our paper also shows that undirected st-connectivity is within the L complexity class, problems solvable by deterministic Turing machines in logarithmic space. Leading from this result, we shall explain why SL = L and discuss why is it believed that RL = L.

With the rapid rise of various Over- The- Top (OTT) and streaming services like Netflix and YouTube, the number of viewers of video content has increased exponentially. In addition, the growth of the video content services market has been accelerated by the pandemic. Especially Korean content has drawn attention globally such as “The Squid Game”, “The Parasite”. A keyword like “Mukbang” which originated in the Korean language has commonly been used among global viewers and grown interest in Korean food. Therefore, we constructed Korean Food in Video Contents (KFVC) dataset and measured Korean food detection performance using the object detection model to observe the feasibility of searching Korean foods introduced in video contents.

Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to real-world settings, such as in robotics, where such interaction is expensive. In this work we investigate ways to minimize online interactions in a target task, by reusing a suboptimal policy we might have access to, for example from training on related prior tasks, or in simulation. To this end, we develop two RL algorithms that can speed up training by using not only the action distributions of teacher policies, but also data collected by such policies on the task at hand. We conduct a thorough experimental study of how to use suboptimal teachers on a challenging robotic manipulation benchmark on vision-based stacking with diverse objects. We compare our methods to offline, online, offline-to-online, and kickstarting RL algorithms. By doing so, we find that training on data from both the teacher and student, enables the best performance for limited data budgets. We examine how to best allocate a limited data budget - on the target task - between the teacher and the student policy, and report experiments using varying budgets, two teachers with different degrees of suboptimality, and five stacking tasks that require a diverse set of behaviors. Our analysis, both in simulation and in the real world, shows that our approach is the best across data budgets, while standard offline RL from teacher rollouts is surprisingly effective when enough data is given.

Persistent memory (PM) technologies offer performance close to DRAM with persistence. Persistent memory enables programs to directly modify persistent data through normal load and store instructions bypassing heavyweight OS system calls for persistency. However, these stores are not made immediately made persistent, the developer must manually flush the corresponding cache lines to force the data to be written to persistent memory. While state-of-the-art testing tools can help developers find and fix persistency bugs, prior studies have shown fixing persistency bugs on average takes a couple of weeks for PM developers. The developer has to manually inspect the execution to identify the root cause of the problem. In addition, most of the existing state-of-the-art testing tools require heavy user annotations to detect bugs without visible symptoms such as a segmentation fault. In this paper, we present robustness as a sufficient correctness condition to ensure that program executions are free from missing flush bugs. We develop an algorithm for checking robustness and have implemented this algorithm in the PSan tool. PSan can help developers both identify silent data corruption bugs and localize bugs in large traces to the problematic memory operations that are missing flush operations. We have evaluated PSan on a set of concurrent indexes, persistent memory libraries, and two popular real-world applications. We found 48 bugs in these benchmarks that 17 of them were not reported before.

We study the problem of robotic stacking with objects of complex geometry. We propose a challenging and diverse set of such objects that was carefully designed to require strategies beyond a simple "pick-and-place" solution. Our method is a reinforcement learning (RL) approach combined with vision-based interactive policy distillation and simulation-to-reality transfer. Our learned policies can efficiently handle multiple object combinations in the real world and exhibit a large variety of stacking skills. In a large experimental study, we investigate what choices matter for learning such general vision-based agents in simulation, and what affects optimal transfer to the real robot. We then leverage data collected by such policies and improve upon them with offline RL. A video and a blog post of our work are provided as supplementary material.

This paper proposes a novel scene boundary detector by considering different features appropriate for definition changes of scenes according to target services or tasks. In the proposed method, the information in shots is categorized into two groups: relational and sequential information. Relational information is acquired by the multi-layered convolution neural networks by merging and embedding similarity vectors from visual and audio features. Sequential information that contains particular patterns of continuous shots is handled with dual recurrent neural networks. The different definitions of scenes are reflected in the proposed method by supervised parameter estimation with a sampling method. Scene boundaries are rarely observed in video content. Thus, it results in skewed class distribution. The sampling method tries to expand instances in scene boundary using reverse order shots, while it reduces the number of non-boundary shots by variance preserved shot filtering. A focal loss is finally adopted for the training process to lead better parameters from an imbalanced dataset. The proposed method is evaluated with three datasets constructed with real-world movies. We empirically proved that different definitions of scene boundary could affect the performance of scene boundary detection through experiments. The proposed deep neural networks with both relational and sequential information show the ability to handle diverse scene definitions in experiments. With supervised learning, the proposed method can reflect the definition bias in each dataset. As a result, the proposed method shows its effectiveness in handling different types of information and adopting other scene definitions by achieving state-of-the-art performances in two benchmark datasets.

As the development of deep learning techniques is growing, applications using deep learning have been spreading. Among various applications, images and videos related applications are the most common example of the practical deep learning application. The performances in those applications have been boosted by adopting deep learning techniques. To achieve performance, securing a large amount of data-oriented to target tasks is crucial. In this paper, we have designed the experiments to examine the effect of generated data on both where the dataset can be easily collected and hard to secure. We use state-of-the-art generative model, MCnet, to enlarge the Sexually Harmful Contents dataset and UCF-101. By training C3D with augmented data, we measure the classification performance. The generated labeled data have increased the performance by 7% on harmful content detection.

Deep reinforcement learning (RL) algorithms can use high-capacity deep networks to learn directly from image observations. However, these high-dimensional observation spaces present a number of challenges in practice, since the policy must now solve two problems: representation learning and task learning. In this work, we tackle these two problems separately, by explicitly learning latent representations that can accelerate reinforcement learning from images. We propose the stochastic latent actor-critic (SLAC) algorithm: a sample-efficient and high-performing RL algorithm for learning policies for complex continuous control tasks directly from high-dimensional image inputs. SLAC provides a novel and principled approach for unifying stochastic sequential models and RL into a single method, by learning a compact latent representation and then performing RL in the model's learned latent space. Our experimental evaluation demonstrates that our method outperforms both model-free and model-based alternatives in terms of final performance and sample efficiency, on a range of difficult image-based control tasks. Our code and videos of our results are available at our website.

We have built the scene-segmented video information annotation system and upgraded it to version 2.0. The system imports the video by user selection and splits into the scene units. Each scene clips are annotated by the integration of visual features derived by state-of-the-art deep learning techniques. The proposed system uses the multiview deep convolutional neural network for video segmentation and a supervised movie caption model for video annotation. Each functionality has been installed in two different sub-systems and connected through the web interface. The web interface allows connecting to external content providers in order to expand the capability of the system.

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.

Surface condition plays an important role in the optical performance of semiconductor materials. As new types of semiconductors, the emerging metal-halide perovskites are promising for next-generation optoelectronic devices. We discover significantly improved light-emission efficiencies in lead halide perovskites due to surface oxygen passivation. The enhancement manifests close to 3 orders of magnitude as the perovskite dimensions decrease to the nanoscale, improving external quantum efficiencies from <0.02% to over 12%. Along with about a 4-fold increase in spontaneous carrier recombination lifetimes, we show that oxygen exposure enhances light emission by reducing the nonradiative recombination channel. Supported by X-ray surface characterization and theoretical modeling, we propose that excess lead atoms on the perovskite surface create deep-level trap states that can be passivated by oxygen adsorption.

We had designed the scene-segmented video information annotation system using video segmentation and information annotation. For video segmentation, the proposed system adapts the multiview deep convolution neural network. Segmented scenes are annotated by using the unsupervised sentence embedding model for closed captions. Both functionalities effectively work together with the web interface designed to tie not only our functionalities but also external content providers.

Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly predict the future from raw sensory readings, such as camera images. At test time, we explore three distinct goal specification methods: designated pixels, where a user specifies desired object manipulation tasks by selecting particular pixels in an image and corresponding goal positions, goal images, where the desired goal state is specified with an image, and image classifiers, which define spaces of goal states. Our deep predictive models are trained using data collected autonomously and continuously by a robot interacting with hundreds of objects, without human supervision. We demonstrate that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.

Prediction is an appealing objective for self-supervised learning of behavioral skills, particularly for autonomous robots. However, effectively utilizing predictive models for control, especially with raw image inputs, poses a number of major challenges. How should the predictions be used? What happens when they are inaccurate? In this paper, we tackle these questions by proposing a method for learning robotic skills from raw image observations, using only autonomously collected experience. We show that even an imperfect model can complete complex tasks if it can continuously retry, but this requires the model to not lose track of the objective (e.g., the object of interest). To enable a robot to continuously retry a task, we devise a self-supervised algorithm for learning image registration, which can keep track of objects of interest for the duration of the trial. We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and non-prehensile manipulation. Our real-world experiments demonstrate that a model trained with 160 robot hours of autonomously collected, unlabeled data is able to successfully perform complex manipulation tasks with a wide range of objects not seen during training.