In this paper, we propose VistaDreamer, a novel framework to reconstruct a 3D scene from a single-view image. Recent diffusion models enable generating high-quality novel-view images from a single-view input image. However, maintaining consistency across generated novel views remains challenging, often leading to noisy or distorted 3D reconstructions. VistaDreamer addresses this problem by a two-stage pipeline. In the first stage, VistaDreamer builds a global coarse 3D scaffold by zooming out a little step with inpainted boundaries and an estimated depth map. Then, on this global scaffold, iterative RGB-D inpainting generates novel-view images to fill the holes, guided by several geometric consistency constraints. In the second stage, we further enhance the consistency between the generated novel-view images by a novel training-free Multi-view Consistency Sampling (MCS) that introduces multi-view consistency constraints in reverse sampling process of diffusion models. VistaDreamer framework is compatible with various geometric estimators, diffusion models and samplers. Extensive experiments show that, without task-specific training or fine-tuning, it achieves good reconstruction quality and strong generalization across diverse scenes and trajectories, with ablation studies confirming the effectiveness and trends of our designs.
Video Moment Retrieval (VMR), the task of identifying a specific moment in an untrimmed video from a natural language query, is a key capability for advanced video understanding. As an emerging research direction, point-supervised VMR offers a cost-effective solution by requiring minimal annotation efforts. In this paper, we explore the information asymmetries within this task: a semantic-positional asymmetry between the clarity of the visual cues and the imprecision of the annotated frame's position, and an instance-duration asymmetry between the certainty of a moment’s occurrence and the ambiguity of its duration. To mitigate these asymmetries, we propose a novel framework, Dual-Asymmetry Aligned Network (DAAN), featuring two specialized components: (1) the Center-Guided Calibration module (CGC) which mitigates positional imprecision by dynamically generating a pseudo temporal center and leveraging a decomposed cross-modal attention strategy to recalibrate the feature distribution to align semantic attention with the estimated event centroid; and (2) the Duration-Aware Proposal Discriminator (DPD) which alleviates duration ambiguity by distinguishing the most plausible temporal spans. It infers missing duration information by learning query-conditioned structural compatibility between proposal content and boundary representations. Extensive experiments on TACoS, ActivityNet Captions, and Charades-STA validate the effectiveness of our proposed framework in point-supervised VMR.
Existing methods enhance the training of detection transformers by incorporating an auxiliary one-to-many assignment. In this work, we treat the model as a multi-task framework, simultaneously performing one-to-one and one-to-many predictions. We investigate the roles of each component in the transformer decoder across these two training targets, including self-attention, cross-attention, and a feed-forward network. Our empirical results demonstrate that any independent component in the decoder can effectively learn both targets simultaneously, even when other components are shared. This finding leads us to propose a multi-route training mechanism, featuring a primary route for one-to-one prediction and two auxiliary training routes for one-to-many prediction. We propose a novel instructive self-attention mechanism, integrated into the first auxiliary route, which dynamically and flexibly guides object queries for one-to-many prediction. For the second auxiliary route, we introduce a route-aware Mixture-of-Experts (MoE) to facilitate knowledge sharing while mitigating potential conflicts between routes. Additionally, we apply an MoE to low-scale features in the encoder, optimizing the balance between efficiency and effectiveness. The auxiliary routes are discarded during inference. We conduct extensive experiments across various object detection baselines, achieving consistent improvements as demonstrated in 1. Our method is highly flexible and can be readily adapted to other tasks. To demonstrate its versatility, we conduct experiments on both instance segmentation and panoptic segmentation, further validating its effectiveness.
Sharing medical image sets is essential yet challenging due to limited resource and strict regulatory, which significantly hinder the development of medical foundation models and the transfer of medical domain knowledge. In this paper, we utilize medical image dataset distillation techniques to enable effective model construction and knowledge transfer without requiring downstream users to directly access the original data. Trajectory matching has become the mainstream distillation technique, yet our empirical findings on medical image datasets reveal two critical phenomena: instability of distillation optimization and redundancy of synthetic images. To investigate these phenomena, we conduct a theoretical analysis from the perspectives of bi-level optimization and neural tangent kernel, and formally define them as two core challenges: trajectory matching drift and intra-class diversity decay, which underscore the need for more stable and diversity-preserving trajectory matching methods. Guided by the analysis results, we propose a novel progressive and diverse dataset distillation (PD$^{3}$) method, which incorporates (1) a progressive trajectory matching paradigm that progressively unrolls the student training trajectory to extend the matching horizon, while adaptively selecting optimal expert matching trajectory to ensure stable and convergent distillation optimization; and (2) a dynamic diversity enhancing module that alleviates the redundancy of synthetic images by enhancing intra-class distribution and breaking redundancy accumulation. To facilitate fair evaluation, we establish a comprehensive medical image dataset distillation benchmark covering diverse medical image modalities. Extensive experiments show that the proposed method achieves an average absolute improvement of 10.20% and a relative gain of 20.09% over prior state-of-the-art methods, highlighting its effectiveness and practical potential.
Enabling embodied agents to follow natural language instructions is a core challenge for building intelligent robots. In this area, the key point lies in how to extract effective task representation from human instruction. While previous methods have attempted to learn task-related representations with large language models (LLMs), they are limited by highly relying on pre-collected task data or requiring extra training procedures. In this study, we explore a novel approach that leverages the internal knowledge of LLMs to extract effective task representations, and propose in-context learning embedding as task representation (InCLET) method. InCLET is grounded on a foundational finding that LLM in-context learning using trajectories can greatly help represent tasks. We thus firstly employ LLM to imagine task trajectories following the natural language instruction. Then use in-context learning of LLM to generate task representations. Finally, the task representation can be naturally used to train a human instruction-following agent with reinforcement learning. We conduct experiments on various embodied control scenarios, covering both simulation and real-world robot. Experiments on FrankaKitchen and CLEVR-Robot simulators, we present that InCLET creates effective task representations and improve the performance of policy. Besides, this representation can significantly improve the training efficiency, compared to the baseline methods. Experiments on real-world ALOHA robot, we show that InCLET can achieve a higher task completion rate compared to baselines.
