It is crucial to efficiently execute instructions such as “Find an apple and a banana.” or “Get ready for a field trip,” which require searching for multiple objects or understanding context-dependent commands. This study addresses the challenging problem of determining which robot should be assigned to which part of a task when each robot possesses different situational on-site knowledge, specifically, room-wise object presence probabilities learned from the area designated to it by the user. We infer room-wise object presence probabilities via Bayesian inference using a spatial concept model. The inference results are then converted into prompts. Large language models (LLMs) use these prompts to decompose instructions into tasks and assign them to multiple robots. We designed a novel few-shot prompting strategy that enables LLMs to infer required objects from ambiguous commands and decompose them into appropriate subtasks. In our experiments, the proposed method achieved 47/50 successful assignments, outperforming random (28/50) and commonsense-based assignment (26/50). Furthermore, we conducted qualitative evaluations using two actual mobile manipulators. The results demonstrated that our framework could handle instructions, including underspecified instructions such as “Get ready for a field trip.”, by successfully performing task decomposition, assignment, sequential planning, and execution. For reproducibility, we release the full set of prompts on the project website at https://kentomurata0610.github.io/multi-robot-task-planning .

Emergent communication (EmCom) investigates how artificial agents collaboratively develop languages that exhibit compositional generalization and message alignment between agents. Existing approaches, such as the Inter-VAE+VAE model employing recurrent neural networks (RNNs), face limitations in compositionality and message alignment. Specifically, the sequential structure of RNNs leads to gradual information loss, making it difficult to consistently encode multiple attributes of referents across message sequences and maintain alignment with agents’ internal conceptual representations. To overcome these limitations, we propose Inter-TLM+VAE, a Transformer-based model integrated into the Metropolis-Hastings Naming Game (MHNG), designed to enhance compositional generalization and message alignment between two agents. Each agent consists of two modules: a Variational Autoencoder (VAE), which encodes visual objects into internal conceptual representations, and a Transformer-based Language Model (TLM), which generates messages conditioned directly on these representations. We evaluate our model on the dSprites and 3DShapes datasets, comparing it against baselines including referential games, reconstruction tasks, and the original Inter-VAE+VAE. Results demonstrate that Inter-TLM+VAE improves both compositionality and interagent alignment on a more complex dataset. An ablation study further reveals that self-attention alone substantially outperforms recurrent architectures across all metrics, while integrating both self- and cross-attention achieves the best overall performance. These findings underscore the role of attention mechanisms in advancing robust EmCom systems within the MHNG framework.

Robots operating in domestic and office environments must understand object ownership to correctly execute instructions such as “Bring me my cup.” However, ownership cannot be reliably inferred from visual features alone. To address this gap, we propose Active Ownership Learning (ActOwL), a framework that enables robots to actively generate and ask users ownership-related questions. ActOwL employs a probabilistic generative model to select questions that maximize information gain, thereby acquiring ownership knowledge to improve learning efficiency. Additionally, by leveraging commonsense knowledge from large language models (LLMs), objects are pre-classified as either shared or owned, and only owned objects are targeted for questioning. Through experiments in a simulated home environment and a real-world laboratory setting, ActOwL achieved significantly higher ownership clustering accuracy with fewer questions than baseline methods. These findings demonstrate the effectiveness of combining active inference with LLM-guided commonsense reasoning, advancing robots’ ability to acquire ownership knowledge for practical, socially appropriate task execution. The project page is available at https://hashimoto-saki.github.io/ActOwl/ .

Emergent communication in multi-agent systems poses scalability challenges, as conventional models are typically limited to dyadic interactions. This study introduces the Variational Bayes Naming Game (VBNG), a decentralized Bayesian inference framework for modeling symbol emergence among three or more agents. Based on the Collective Predictive Coding (CPC) hypothesis, VBNG enables agents to infer shared signs by minimizing free energy using their individual observations. In our experiment, we evaluated the proposed method against a conventional sampling-based approach (RMHNG) and a centralized topline model (VB) using both synthetic and patched MNIST image datasets. VBNG achieved accuracy comparable to the topline model. Although the conventional method failed under conditions where individual agents' observations were insufficient for identification, VBNG demonstrated high robustness (ARI > 0.6). The proposed method demonstrated significantly improved computational efficiency, converging 2-4 times faster than the conventional method in experiments with up to 10 agents. This performance was achieved while attaining high classification accuracy and high inter-agent agreement (kappa = 0.94) on the MNIST task. This study presents a robust, scalable, and theoretically coherent framework for symbol emergence research, demonstrating variational inference as an effective mechanism for consensus formation in decentralized multi-agent systems.

This study tackles robotic picking of multi-part deformable objects--common in warehouses yet underexplored in the literature--such as cable-attached appliances and pouch drinks, which comprise both rigid and deformable components. Their deformability poses a challenge to model-based 6D pose estimators, such as FoundationPose, that assume rigid bodies. To address this, we present PartPose, which estimates the 6D pose of the multi-part deformable objects by focusing on the rigid components. PartPose uses Bayesian optimization to select an appropriate region of interest (ROI) and then estimates its pose with a render-and-compare pipeline. We evaluate pose-estimation and picking success rates on nine multi-part deformable objects, counting a pose estimate as successful if the translational error is <30 mm and the rotational error is <0.3 radians. PartPose significantly outperforms a FoundationPose baseline, achieving success rates of 98.2% (translational), 96.4% (rotational), and 87.2% (picking), versus 47.9%, 35.9%, and 22.8%, respectively. Moreover, PartPose generalizes category-level semantic knowledge to new instances within the same category without performance degradation when those instances have semantically similar components. This capability is crucial for large logistics centers that handle diverse and novel objects.

Real-world communicative signals—such as gestures, vocalizations, and facial expressions—are inherently continuous and subtle. Research on emergent communication has been advanced as a constructive approach to elucidate the emergence of symbol systems enabling communication among agents through interactions mediated by physical signals, namely, signs. However, research on emergent communication using decentralized Bayesian inference has predominantly relied on discrete symbolic representations, struggling to capture inherent subtleties. To address this limitation, we propose a probabilistic framework called multimodal Gaussian process emergent communication (MGPEmCom), which enables the emergence of continuous symbols among decentralized agents. This framework integrates Gaussian process latent variable models to describe multimodal sensory observations and employs the Metropolis—Hastings naming game for decentralized inference, allowing agents to jointly infer shared continuous signs without accessing one another’s internal states. We validate our approach with a referential game task in which a speaker uses an emergent continuous symbol to communicate a target object to a listener. Experimental results demonstrate that MGPEmCom outperforms existing methods, including Inter-MDM and VAE+REINFORCE, particularly in scenarios that require fine-grained distinctions and generalization to unknown objects. These results indicate that our method enables more expressive and generalizable symbol emergence. This work establishes a principled Bayesian approach to decentralized symbol emergence using continuous latent representations, offering a foundation for modeling communication applicable to unknown objects.

Measuring the extent to which emergent languages encode the visual content of their inputs is an open problem. We refer to this property as visual reflection: the extent to which emergent messages preserve information about their source images that can be recovered without appeal to the speaker-listener pair that produced them. Existing metrics measure it only indirectly, through proxies such as human-defined concept inventories, natural-language captions, structural distance correlations, or Referential Game accuracy, each of which can either miss visual content the message encodes or credit content it does not. We propose EmCom-Diffusion, an evaluation framework that measures visual reflection directly: it reconstructs each input image from its emergent message and compares the reconstruction with the original image itself, rather than with human-defined targets. Concretely, it finetunes a pretrained text-to-image diffusion model on (image, emergent-message) pairs and scores visual reflection as the perceptual similarity between the reconstructed and original images, operating generatively rather than discriminatively. Instantiating it on MS-COCO with a Referential Game, we validate the metric against random and fixed-token baselines under three pretrained visual encoders, and compare it against four existing metrics (CBM, supervised translation, TopSim, and R@1). EmCom-Diffusion captures visual content the other metrics miss.

Unsupervised constituency parsing aims to accurately induce latent tree structures from raw text alone. Recent neural parameterizations of PCFGs achieve strong performance in both supervised and unsupervised parsing, yet rely on high-capacity black-box networks for rule scoring – as exemplified by the Neural PCFG family – leaving rule probabilities without an interpretable mathematical form. In this paper, we propose Holographic Neural PCFG (Hol-PCFG), which recasts PCFG rule scoring as algebraic relation modeling among grammar-symbol embeddings. Hol-PCFG adapts Holographic Embeddings (Nickel et al., 2016), which scores knowledge-graph triples via circular correlation, to the left-child, right-child, and lexical-emission relations over torus-constrained embeddings, giving every rule probability a closed form that carries the intrinsic structure of grammar rules by construction. Hol-PCFG achieves state-of-the-art parsing performance in six languages while cutting rule-scoring parameters by 99.94

This study explores collaborative music generation using multiple symbolic music generation AI agents, grounding the process in the Systems Model of Creativity. We propose the Metropolis-Hastings Music Generation Game (MH-MuG), which integrates latent diffusion models (LDMs) and formulates the collaboration as a decentralized Bayesian inference process. In MH-MuG, agents with distinct musical knowledge (pre-trained on Classical and Jazz) alternate between composer and listener roles. This interaction functions as a Markov Chain Monte Carlo (MCMC) method, enabling agents to collectively sample from a joint distribution that integrates their knowledge. We compared two variants: one without fine-tuning (w/o f.t.), modeling a fixed-knowledge game, and one with fine-tuning (w/ f.t.), modeling mutual adaptation. Our experiments yielded two key findings: (1) The (w/o f.t.) variant, functioning as a collaborative music generation game, successfully generated high-quality, stylistically fused music. (2) Conversely, the (w/ f.t.) variant led to a significant reduction in diversity. We interpret this not as a failure, but as a computational demonstration of the ‘‘siloing’’ phenomenon that occurs when creative interactions are limited to a closed loop.

We propose EasyControlEdge, adapting an image-generation foundation model to edge detection. In real-world edge detection (e.g., floor-plan walls, satellite roads/buildings, and medical organ boundaries), crispness and data efficiency are crucial, yet producing crisp raw edge maps with limited training samples remains challenging. Although image-generation foundation models perform well on many downstream tasks, their pretrained priors for data-efficient transfer and iterative refinement for high-frequency detail preservation remain underexploited for edge detection. To enable crisp and data-efficient edge detection using these capabilities, we introduce an edge-specialized adaptation of image-generation foundation models. To better specialize the foundation model for edge detection, we incorporate an edge-oriented objective with an efficient pixel-space loss. At inference, we introduce guidance based on unconditional dynamics, enabling a single model to control the edge density through a guidance scale. Experiments on BSDS500, NYUDv2, BIPED, and CubiCasa compare against state-of-the-art methods and show consistent gains, particularly under no-post-processing crispness evaluation and with limited training data.

According to the theory of constructed emotion, the brain actively forms emotion categories by integrating multimodal bodily signals, and constructs emotional experiences by using these categories to predict and interpret sensory inputs. While research has advanced in modeling individual emotion construction, the social process of co-construction-how a shared understanding of emotions emerges between individuals-remains computationally underexplored. This study investigates this process by modeling emergent communication between two embodied agents using the Metropolis-Hastings Naming Game (MHNG), grounded in the Collective Predictive Coding (CPC) framework. Our experiments, using visual, auditory, and simulated interoceptive inputs, yield two main findings. First, MHNG-based communication significantly improves the alignment, clarity, and inter-agent agreement of the learned emotion categories compared to non-communicative and non-selective baselines, with the alignment effect concentrated at the symbolic layer rather than the perceptual latent representation. Second, even when the two agents have systematically divergent interoceptive dynamics, communication still produces robust categorical alignment, with distinct, category-specific reshaping patterns of each agent's emotion categories-consistent with the constructed-emotion view that interoceptive heterogeneity is constitutive of, rather than an obstacle to, shared emotional meaning. These findings provide computational support for the co-constructionist view of emotion and extend the CPC framework from physical to socially-grounded domains.

Emotion is both a psychological phenomenon rooted in physiological fluctuations and a socially constructed phenomenon shaped by language and culture. However, traditional psychological constructivism has remained confined to individual brain processes, while social constructivism has lacked a bottom-up computational mechanism explaining how language and culture emerge from bodily emotional substrates and dynamically evolve. To bridge this theoretical gap, this paper proposes the Theory of Co-constructed Emotion (TCoE) via Collective Predictive Coding (CPC), a framework coupling physiological regulation with social symbol sharing within a single active inference formulation. We extend the conventional CPC model by explicitly incorporating interoception---which is indispensable for modeling emotion from a physiological perspective---into each agent's generative model, alongside exteroception and internal and external representations. In TCoE, the co-construction of emotion is described as a dual inference process: an internal loop of allostasis regulating individual bodily states, and a social loop of communication sharing emotional concepts with others. These loops mutually minimize prediction errors mediated by shared symbols (emotion words). We argue that this theoretical framework 1) aligns with psychological findings on emotional contagion and concept formation, 2) enables constructive verification in multi-agent robotic systems, and 3) provides a computational redefinition of psychiatric conditions such as alexithymia as a disconnection between interoception and social symbols. Thus, we offer a novel computational foundation for emotion research that unifies individual embodiment and social symbol systems.

Large Language Models (LLMs) have demonstrated a remarkable ability to capture extensive world knowledge, yet how this is achieved without direct sensorimotor experience remains a fundamental puzzle. This study proposes a novel theoretical solution by introducing the Collective World Model hypothesis. We argue that an LLM does not learn a world model from scratch; instead, it learns a statistical approximation of a collective world model that is already implicitly encoded in human language through a society-wide process of embodied, interactive sense-making. To formalize this process, we introduce generative emergent communication (Generative EmCom), a framework built on the Collective Predictive Coding (CPC). This framework models the emergence of language as a process of decentralized Bayesian inference over the internal states of multiple agents. We argue that this process effectively creates an encoder-decoder structure at a societal scale: human society collectively encodes its grounded, internal representations into language, and an LLM subsequently decodes these symbols to reconstruct a latent space that mirrors the structure of the original collective representations. This perspective provides a principled, mathematical explanation for how LLMs acquire their capabilities. The main contributions of this paper are: 1) the formalization of the Generative EmCom framework, clarifying its connection to world models and multi-agent reinforcement learning, and 2) its application to interpret LLMs, explaining phenomena such as distributional semantics as a natural consequence of representation reconstruction. This work provides a unified theory that bridges individual cognitive development, collective language evolution, and the foundations of large-scale AI.

Multi-robot coordination based on large language models (LLMs) has attracted growing attention, since LLMs enable the direct translation of natural language instructions into robot action plans by decomposing tasks and generating high-level plans. However, recovering from physical execution failures remains difficult, and tasks often stagnate due to the repetition of the same unsuccessful actions. While frameworks for remote robot operation using Mixed Reality were proposed, there have been few attempts to implement remote error resolution specifically for physical failures in multi-robot environments. In this study, we propose REPAIR (Robot Execution with Planned And Interactive Recovery), a human-in-the-loop framework that integrates remote error resolution into LLM-based multi-robot planning. In this method, robots execute tasks autonomously; however, when an irrecoverable failure occurs, the LLM requests assistance from an operator, enabling task continuity through remote intervention. Evaluations using a multi-robot trash collection task in a real-world environment confirmed that REPAIR significantly improves task progress (the number of items cleared within a time limit) compared to fully autonomous methods. Furthermore, for easily collectable items, it achieved task progress equivalent to full remote control. The results also suggested that the mental workload on the operator may differ in terms of physical demand and effort. The project website is https://emergentsystemlabstudent.github.io/REPAIR/.

Symbols are shared, but perception is private. We study emergent communication between heterogeneous visual agents through decentralized learning, asking what visual information can become shareable when agents have different visual representations. Instead of optimizing messages through a shared external communicative objective, our agents exchange only discrete token sequences and update their own models using local perceptual evidence. This setting focuses on an underexplored aspect of emergent communication, examining whether common symbols can arise without shared perceptual access, and how the similarity between private visual spaces constrains the content and symmetry of the resulting language. We instantiate this setting in the Metropolis-Hastings Captioning Game (MHCG), where two agents collaboratively form shared captions by exchanging proposed token sequences that a listener accepts or rejects using an MH-style criterion evaluated against its own visual features. We compare three pairings of frozen visual encoders, with agents starting from randomly initialized text modules. Experiments on MS-COCO show that MHCG produces visually informative shared token sequences that outperform a no-communication baseline in cross-agent alignment, visual-feature prediction, and image-text retrieval; all cross-agent metrics decline as encoder mismatch increases. Moderate encoder heterogeneity reduces the number of shared sequences while preserving per-sequence visual specificity, whereas stronger encoder heterogeneity yields fewer, coarser, and more asymmetric sequences. Ablations show that listener-side MH acceptance is critical for avoiding degenerate token formation. These results suggest that shared symbols can arise from local perceptual evaluation alone, with visual representational similarity across encoders shaping both the content and symmetry of the resulting language.

Service robots must infer object ownership to correctly interpret instructions such as "bring me my cup." However, ownership is a latent attribute that cannot be directly observed, and existing methods often rely on limited cues such as recent usage, making them unreliable in scenarios such as temporary sharing. We propose a framework for context-aware ownership inference with uncertainty-guided interaction (COIN). The method integrates user background information and object usage history using a large language model (LLM) to estimate ownership scores. To handle uncertainty, we apply conformal prediction to construct a set of plausible owners and selectively generate user queries when the prediction is uncertain. Experiments in a simulated home environment show that the proposed method consistently outperforms baseline approaches, achieving a Subset Accuracy of 0.988 and a Mean Jaccard index of 0.991. The method also maintains high performance in scenarios involving temporary use and shared ownership. The results demonstrate that combining contextual reasoning with uncertainty-aware interaction improves both estimation accuracy and robustness. The project page is available at https://emergentsystemlabstudent.github.io/COIN/.

We propose a fully decentralized multi-agent world model that enables both symbol emergence for communication and coordinated behavior through temporal extension of collective predictive coding. Unlike previous research that focuses on either communication or coordination separately, our approach achieves both simultaneously. Our method integrates world models with communication channels, enabling agents to predict environmental dynamics, estimate states from partial observations, and share critical information through bidirectional message exchange with contrastive learning for message alignment. Using a two-agent trajectory drawing task, we demonstrate that our communication-based approach outperforms non-communicative models when agents have divergent perceptual capabilities, achieving the second-best coordination after centralized models. Importantly, our decentralized approach with constraints preventing direct access to other agents' internal states facilitates the emergence of more meaningful symbol systems that accurately reflect environmental states. These findings demonstrate the effectiveness of decentralized communication for supporting coordination while developing shared representations of the environment.

To make robotic grasping scalable, vision-based control with high data efficiency and accuracy is needed. World models are capable of creating representations of physical environments from sensory information. In particular, NewtonianVAE is a world model that can control targets in physical environments by using proportional control in its latent space from input images. However, NewtonianVAE entangles information of each object in separate state subspaces making control unfeasible when trained with multiple objects. In this paper, we introduce Conditional NewtonianVAE, a novel framework designed to generate pre-grasping actions by disentangling object-type information from the state space in physical latent spaces. Our method incorporates a conditioning variable to achieve disentanglement, facilitating the use of the learned state space for control tasks. Through simulation and real-robot experiments, we demonstrate the effectiveness of Conditional NewtonianVAE in accurately positioning the end-effector into a pre-grasping pose, thereby enhancing the success rate of robotic grasping. Conditional NewtonianVAE achieves a grasping success rate of 83% for known objects and 78% for unseen objects in the real-robot experiments.

This article proposes a new conceptual framework called collective predictive coding as a model of science (CPC-MS) to formalize and understand scientific activities. Building on the idea of CPC originally developed to explain symbol emergence, CPC-MS models science as a decentralized Bayesian inference process carried out by a community of agents. The framework describes how individual scientists’ partial observations and internal representations are integrated through communication and peer review to produce shared external scientific knowledge. Key aspects of scientific practice like experimentation, hypothesis formation, theory development and paradigm shifts are mapped onto components of the probabilistic graphical model. This article discusses how CPC-MS provides insights into issues like social objectivity in science, scientific progress and the potential impacts of artificial intelligence on research. The generative view of science offers a unified way to analyse scientific activities and could inform efforts to automate aspects of the scientific process. Overall, CPC-MS aims to provide an intuitive yet formal model of science as a collective cognitive activity.