Abductive reasoning seeks the likeliest possible explanation for partial observations. Although being frequently employed in human daily reasoning, abduction is rarely explored in computer vision literature. In this article, we propose a new task, Visual Abductive Reasoning (VAR), that underpins the machine intelligence study of abductive reasoning in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the observed premise. We create the first large-scale VAR dataset, which contains a total of 9K examples. We further devise a transformer-based VAR model - Reasonerv2 - for knowledge-driven, causal-and-cascaded reasoning. Reasonerv2 first adopts a contextualized directional position embedding strategy in the encoder, to capture the causal-related temporal structure of the observations, and yield discriminative representations for the premises and hypotheses. Then, Reasonerv2 extracts condensed causal knowledge from external knowledge bases, for reasoning beyond observation. Finally, Reasonerv2 cascades multiple decoders so as to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR benchmarking results show that Reasonerv2 surpasses many famous video-language models, while still being far behind human performance.

Building autonomous agents that can replicate human behavior in the realistic 3D world is a key step toward artificial general intelligence. This requires agents to be holistic goal achievers and to naturally adapt to environmental dy namics. In this work, we introduce Actor, an agent capable of performing high-level, long-horizon, abstract goals in 3D households, guided by its internal value similar to those of humans. Actor operates in a perceive-plan-act cycle, extending the ungrounded, scene-agnostic LLM controller with deliberate goal decomposition and decision-making through actively searching the behavior space, generating activity choices based on a hierarchical prior, and evaluating these choices using customizable value functions to determine the subsequent steps. Furthermore, we introduce BehaviorHub, a large-scale human behavior simulation dataset in scene-aware, complicated tasks. Considering the unaffordable acquisition of human-authored 3D human behavior data, we construct BehaviorHub by exploring the commonsense knowledge of LLMs learned from large corpora, and automatically aligning motion resources with 3D scene for knowledgeable generation. Extensive experiments on our established benchmark demonstrate that the proposed architecture leads to effective behavior planning and simulation. BehaviorHub also proves beneficial for downstream task development.

Pixel-level Video Understanding requires effectively integrating three-dimensional data in both spatial and temporal dimensions to learn accurate and stable semantic information from continuous frames. However, existing advanced models on the VSPW dataset have not fully modeled spatiotemporal relationships. In this paper, we present our solution for the PVUW competition, where we introduce masked video consistency (MVC) based on existing models. MVC enforces the consistency between predictions of masked frames where random patches are withheld. The model needs to learn the segmentation results of the masked parts through the context of images and the relationship between preceding and succeeding frames of the video. Additionally, we employed test-time augmentation, model aggeregation and a multimodal model-based post-processing method. Our approach achieves 67.27 mIoU performance on the VSPW dataset, ranking 2nd place in the PVUW2024 challenge VSS track.

Video segmentation aims at partitioning video sequences into meaningful segments based on objects or regions of interest within frames. Current video segmentation models are often derived from image segmentation techniques, which struggle to cope with small-scale or class-imbalanced video datasets. This leads to inconsistent segmentation results across frames. To address these issues, we propose a training strategy Masked Video Consistency, which enhances spatial and temporal feature aggregation. MVC introduces a training strategy that randomly masks image patches, compelling the network to predict the entire semantic segmentation, thus improving contextual information integration. Additionally, we introduce Object Masked Attention (OMA) to optimize the cross-attention mechanism by reducing the impact of irrelevant queries, thereby enhancing temporal modeling capabilities. Our approach, integrated into the latest decoupled universal video segmentation framework, achieves state-of-the-art performance across five datasets for three video segmentation tasks, demonstrating significant improvements over previous methods without increasing model parameters.

We explore the task of language-guided video segmentation (LVS). Previous algorithms mostly adopt 3D CNNs to learn video representation, struggling to capture long-term context and easily suffering from visual-linguistic misalignment. In light of this, we present Locater (local-global context aware Transformer), which augments the Transformer architecture with a finite memory so as to query the entire video with the language expression in an efficient manner. The memory is designed to involve two components - one for persistently preserving global video content, and one for dynamically gathering local temporal context and segmentation history. Based on the memorized local-global context and the particular content of each frame, Locater holistically and flexibly comprehends the expression as an adaptive query vector for each frame. The vector is used to query the corresponding frame for mask generation. The memory also allows Locater to process videos with linear time complexity and constant size memory, while Transformer-style self-attention computation scales quadratically with sequence length. To thoroughly examine the visual grounding capability of LVS models, we contribute a new LVS dataset, A2D-S +, which is built upon A2D-S dataset but poses increased challenges in disambiguating among similar objects. Experiments on three LVS datasets and our A2D-S + show that Locater outperforms previous state-of-the-arts. Further, we won the 1st place in the Referring Video Object Segmentation Track of the 3rd Large-scale Video Object Segmentation Challenge, where Locater served as the foundation for the winning solution.

Recent advances in semi-supervised semantic segmentation have been heavily reliant on pseudo labeling to compensate for limited labeled data, disregarding the valuable relational knowledge among semantic concepts. To bridge this gap, we devise LogicDiag, a brand new neural-logic semi-supervised learning framework. Our key insight is that conflicts within pseudo labels, identified through symbolic knowledge, can serve as strong yet commonly ignored learning signals. LogicDiag resolves such conflicts via reasoning with logic-induced diagnoses, enabling the recovery of (potentially) erroneous pseudo labels, ultimately alleviating the notorious error accumulation problem. We showcase the practical application of LogicDiag in the data-hungry segmentation scenario, where we formalize the structured abstraction of semantic concepts as a set of logic rules. Extensive experiments on three standard semi-supervised semantic segmentation benchmarks demonstrate the effectiveness and generality of LogicDiag. Moreover, LogicDiag highlights the promising opportunities arising from the systematic integration of symbolic reasoning into the prevalent statistical, neural learning approaches.

Recent segmentation methods, such as OCR and CPNet, utilizing"class level"information in addition to pixel features, have achieved notable success for boosting the accuracy of existing network modules. However, the extracted class-level information was simply concatenated to pixel features, without explicitly being exploited for better pixel representation learning. Moreover, these approaches learn soft class centers based on coarse mask prediction, which is prone to error accumulation. In this paper, aiming to use class level information more effectively, we propose a universal Class-Aware Regularization (CAR) approach to optimize the intra-class variance and inter-class distance during feature learning, motivated by the fact that humans can recognize an object by itself no matter which other objects it appears with. Three novel loss functions are proposed. The first loss function encourages more compact class representations within each class, the second directly maximizes the distance between different class centers, and the third further pushes the distance between inter-class centers and pixels. Furthermore, the class center in our approach is directly generated from ground truth instead of from the error-prone coarse prediction. Our method can be easily applied to most existing segmentation models during training, including OCR and CPNet, and can largely improve their accuracy at no additional inference overhead. Extensive experiments and ablation studies conducted on multiple benchmark datasets demonstrate that the proposed CAR can boost the accuracy of all baseline models by up to 2.23% mIOU with superior generalization ability. The complete code is available at https://github.com/edwardyehuang/CAR.

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, REASONER (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative representations for the premise and hypothesis. Then, multiple decoders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR bench-marking results show that REASONER surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.

We evaluate our GMMSeg on six base segmentation architectures. Four of them, i.e., DeepLabv3+ [1], OCRNet [2], Swin-UperNet [3], SegFormer [4], are presented in our main paper. And the two additional base architectures, i.e., FCN [5] and Mask2Former [6], are provided in this supplemental material (cf .§S2). We follow the default training settings in the official Mask2Former codebase and MMSegmentation for Mask2Former and other base architectures respectively. In particular, we train FCN, DeepLabv3+ and OCRNet using SGD optimizer with initial learning rate 0.1, weight decay 4e-4 with polynomial learning rate annealing; we train Swin-UperNet and SegFormer using AdamW optimizer with initial learning rate 6e-5, weight decay 1e-2 with polynomial learning rate annealing; we train Mask2Former using AdamW optimizer with initial learning rate 1e-4, weight decay 5e-2 and the learning rate is decayed by a factor of 10 at 0.9 and 0.95 fractions of the total training steps.

Our VAR dataset is curated from three main sources, i.e., YouTube Lifestyle video, movie and TV show, in Fig. 1b and Fig. 1c, we illustrate the detailed distribution of videos and examples by collected sources. As seen, most videos in VAR are from YouTube Lifestyle video, while movie videos tend to have more events, and thus leads to more complicated causal structures and more examples. We then study the distribution of frequently used words in VAR descriptions, which is illustrated as a word cloud in Fig. 1a. More frequent words are shown in larger font size. Finally, the distribution of premise events is shown in Fig. 1d.

Background To compare the changes in quantitative parameters and the size and degree of 1 8 F - fluorodeoxyglucose ([ 18 F]FDG) uptake of malignant tumor lesions between Bayesian penalized-likelihood (BPL) and non-BPL reconstruction algorithms. Methods Positron emission tomography/computed tomography images of 86 malignant tumor lesions were reconstructed using the algorithms of ordered subset expectation maximization (OSEM), OSEM + time of flight (TOF), OSEM + TOF + point spread function (PSF), and BPL. [ 18 F]FDG parameters of maximum standardized uptake value (SUVmax), SUVmean, metabolic tumor volume (MTV), total lesion glycolysis (TLG), and signal-to-background ratio (SBR) of these lesions were measured. Quantitative parameters between the different reconstruction algorithms were compared, and correlations between parameter variation and lesion size or the degree of [ 18 F]FDG uptake were analyzed. Results After BPL reconstruction, SUVmax, SUVmean, and SBR were significantly increased, MTV was significantly decreased. The difference values of %ΔSUVmax, %ΔSUVmean, %ΔSBR, and the absolute value of %ΔMTV between BPL and OSEM + TOF were 40.00%, 38.50%, 33.60%, and 33.20%, respectively, which were significantly higher than those between BPL and OSEM + TOF + PSF. Similar results were observed in the comparison of OSEM and OSEM + TOF + PSF with BPL. The %ΔSUVmax, %ΔSUVmean, and %ΔSBR were all significantly negatively correlated with the size and degree of [ 18 F]FDG uptake in the lesions, whereas significant positive correlations were observed for %ΔMTV and %ΔTLG. Conclusion The BPL reconstruction algorithm significantly increased SUVmax, SUVmean, and SBR and decreased MTV of tumor lesions, especially in small or relatively hypometabolic lesions.

In this paper, we propose a novel framework for tackling the interactive video object segmentation. To deal with the gradually increasing scribble information, our framework applies two independent networks for conducting user interaction and temporal propagation. For the former part, we adopt an inside-outside single-object coarseto-fine structure augmented with a pyramid scene parsing module for aggregating global contextual information (IOI-Net). For the temporal propagation part, to record the informative knowledge from previous interaction rounds, the proposed model (MCFBI-Net) adopts a simple yet effective memory aggregation mechanism based on the Collaborative video object segmentation by Multi-scale Foreground-Background Integration (CFBI+) method, which fully utilizes the rich information from both foreground pixels and background pixels. Besides, we introduce the High Confidence Filter and the Background Random Drop Mechanism in this paper to improve the robustness in discovering challenging objects. Our approach took the 2nd place according to 7 &J@ 60s and the 3rd place with AUC score on interactive track in DAVIS Challenge on Video Object Segmentation 2020.