Fault detection is a critical aspect of system reliability and safety in various fields. Early identification of faults can prevent catastrophic failures, optimize maintenance schedules, and reduce operational costs. Traditional fault detection methods have been fundamental to ensuring system reliability. In recent years, deep learning and machine learning have been employed for fault detection in complex systems. These techniques include hierarchical clustering, the Gaussian mixture model (GMM), and Hotelling’s T-square test. This paper focuses on current state-of-the-art approaches to fault detection. The paper also discusses recent developments in the field of fault detection and its applications.

World models now span latent simulators, video generators, interactive environments, and robot policies, but the field lacks a shared definition of what makes such a model general. This article develops a first-principles account of the general world model (GWM) as a single foundation that integrates three inseparable faculties at scale: Understanding the present world, Imagining possible and action-conditioned futures, and Acting to realize goals while incorporating feedback. From this closed loop, we derive a five-level roadmap of increasing agency: World Generation, Interactive World, Actionable World, Autonomous World Agent, and World Orchestrator. To construct a GWM, we organize around three requirements. First, a five-layer data pyramid combines the breadth of web, curated, and egocentric video with action-recording human demonstrations and real-robot experience. Synthetic data augments all five layers, while imperfect action-aligned trajectories provide evidence about intervention effects, failure, and recovery. Second, as one current architectural instantiation, a Mixture-of-Transformers design specializes parameters for language, video, and action while sharing joint attention, enabling understanding, future prediction, and action to exchange information within one foundation. Third, scalable training and accelerated inference determine whether this foundation can progress from offline generation to real-time interaction and physical control. We further present current systems to illustrate the first three roadmap levels: Vidu implements large-scale world generation; Vidu S1 adds continuous, voice-conditioned interaction; Motus unifies understanding, future-state prediction, and action learning; and Motubrain scales this loop toward deployable cross-embodiment control. These examples support a common development path across digital and physical domains, showing promise to ultimately achieve general embodied intelligence, but do not yet establish the autonomy or orchestration required by L4 and L5. Finally, we discuss directions toward these upper levels, including joint evaluation, causal and physical grounding, persistent memory, online self-improvement, efficient closed-loop deployment, and safe, controllable autonomy.

Knowledge-based visual question answering (KB-VQA), requiring external world knowledge beyond the image for reasoning, is more challenging than traditional visual question answering. Recent works have demonstrated the effectiveness of using a large (vision) language model as an implicit knowledge source to acquire the necessary information. However, the knowledge stored in large models (LMs) is often coarse-grained and inaccurate, causing questions requiring finer-grained information to be answered incorrectly. In this work, we propose a variational expectation-maximization (EM) framework that bootstraps the VQA performance of LMs with its own answer. In contrast to former VQA pipelines, we treat the outside knowledge as a latent variable. In the E-step, we approximate the posterior with two components: First, a rough answer, e.g., a general description of the image, which is usually the strength of LMs, and second, a multi-modal neural retriever to retrieve question-specific knowledge from an external knowledge base. In the M-step, the training objective optimizes the ability of the original LMs to generate rough answers as well as refined answers based on the retrieved information. Extensive experiments show that our proposed framework, BootLM, has a strong retrieval ability and achieves state-of-the-art performance on knowledge-based VQA tasks.

Intelligence is a crucial trait for species to find solutions within a limited number of trial-and-error attempts. Building on this idea, we introduce Survival Game as a framework to evaluate intelligence based on the number of failed attempts in a trial-and-error process. Fewer failures indicate higher intelligence. When the expectation and variance of failure counts are both finite, it signals the ability to consistently find solutions to new challenges, which we define as the Autonomous Level of intelligence. Using Survival Game, we comprehensively evaluate existing AI systems. Our results show that while AI systems achieve the Autonomous Level in simple tasks, they are still far from it in more complex tasks, such as vision, search, recommendation, and language. While scaling current AI technologies might help, this would come at an astronomical cost. Projections suggest that achieving the Autonomous Level for general tasks would require 10^26 parameters. To put this into perspective, loading such a massive model requires so many H100 GPUs that their total value is 10^7 times that of Apple Inc.'s market value. Even with Moore's Law, supporting such a parameter scale would take 70 years. This staggering cost highlights the complexity of human tasks and the inadequacies of current AI technologies. To further investigate this phenomenon, we conduct a theoretical analysis of Survival Game and its experimental results. Our findings suggest that human tasks possess a criticality property. As a result, Autonomous Level requires a deep understanding of the task's underlying mechanisms. Current AI systems, however, do not fully grasp these mechanisms and instead rely on superficial mimicry, making it difficult for them to reach an autonomous level. We believe Survival Game can not only guide the future development of AI but also offer profound insights into human intelligence.

The security issue of large language models (LLMs) has gained significant attention recently, with various defense mechanisms developed to prevent harmful outputs, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the distribution of text embedding model outputs is significantly biased with a large mean. Inspired by this observation, we propose novel efficient methods to search for universal magic words that can attack text embedding models. The universal magic words as suffixes can move the embedding of any text towards the bias direction, therefore manipulate the similarity of any text pair and mislead safeguards. By appending magic words to user prompts and requiring LLMs to end answers with magic words, attackers can jailbreak the safeguard. To eradicate this security risk, we also propose defense mechanisms against such attacks, which can correct the biased distribution of text embeddings in a train-free manner.

With the increasing demand for high-quality telecommunication services, cellular KPI prediction becomes crucial for telecommunication network monitoring and management. In this work, we propose a novel framework for cellular KPI prediction, which considers its distribution discrepancy under different network operation scenarios. In particular, three specific predictors for normal, target alarm, and neighbor alarm scenarios are proposed based on spatiotemporal graph neural networks and unified through transfer learning. Temporal convolution and attention mechanism are embedded to model the impact of anomalies on KPIs and its propagation across neighboring cells according to the cellular network topology. An experiment on a real cellular KPI dataset shows the effectiveness of the proposed method compared to the state-of-the-arts. Note to Practitioners —Cellular network KPI prediction under scenarios of network alarms is crucial to evaluate the impact of alarms on network services and guides cellular network maintenance policies. This problem is similar to a general multivariate time series prediction problem with data multimodality. However, the first challenge in our case is that, under different scenarios, i.e., normal, target alarm, and neighbor alarm, the effective information and spatiotemporal dependencies among KPIs are different. The second challenge is the imbalanced or sparse sample size for specific scenarios, deteriorating the model performance. This paper proposes a cellular KPI prediction framework consisting of three scenario-specific predictors with similar but different modules to process different scenario-specific data. To address the dataset imbalance across scenarios, we adopt a transfer learning strategy to unify the training and prediction of three predictors. The experiment results on a real cellular KPI dataset demonstrate that the proposed framework is more feasible and effective than the state-of-the-art models for multivariate time series prediction. Future research can consider developing maintenance policies such that the cost caused by abnormal KPIs can be minimized.

Deep Neural Networks (DNNs) are known to be susceptible to adversarial attacks. Previous researches mainly fo-cus on improving adversarial robustness in the fully super-vised setting, leaving the challenging domain of zero-shot adversarial robustness an open question. In this work, we investigate this domain by leveraging the recent advances in large vision-language models, such as CLIP, to introduce zero-shot adversarial robustness to DNNs. We pro-pose LAAT, a Language-driven, Anchor-based Adversarial Training strategy. LAAT utilizes the features of a text en-coder for each category as fixed anchors (normalized feature embeddings) for each category, which are then employed for adversarial training. By leveraging the semantic consistency of the text encoders, LAAT aims to enhance the adversarial robustness of the image model on novel cate-gories. However, naively using text encoders leads to poor results. Through analysis, we identified the issue to be the high cosine similarity between text encoders. We then design an expansion algorithm and an alignment cross-entropy loss to alleviate the problem. Our experimental results demonstrated that LAAT significantly improves zero-shot adversarial robustness over state-of-the-art methods. LAAT has the potential to enhance adversarial robustness by large-scale multimodal models, especially when labeled data is unavailable during training. Code is available at https://github.com/LixiaoTHU/LAAT.

Recently, there exists an increased research interest in embodied artificial intelligence (EAI), which involves an agent learning to perform a specific task when dynamically interacting with the surrounding 3D environment. There into, a new challenge is that many unseen objects may appear due to the increased number of object categories in 3D scenes. It makes developing models with strong zero-shot generalization ability to new objects necessary. Existing work tries to achieve this goal by providing embodied agents with massive high-quality human annotations closely related to the task to be learned, while it is too costly in practice. Inspired by recent advances in pre-trained models in 2D visual tasks, we attempt to boost zero-shot generalization for embodied reasoning with vision-language pre-training that can encode common sense as general prior knowledge. To further improve its performance on a specific task, we rectify the pre-trained representation through masked scene graph modeling (MSGM) in a self-supervised manner, where the task-specific knowledge is learned from iterative message passing. Our method can improve a variety of representative embodied reasoning tasks by a large margin (e.g., over 5.0% w.r.t. answer accuracy on MP3D-EQA dataset that consists of many real-world scenes with a large number of new objects during testing), and achieve the new state-of-the-art performance.

Large language models have substantially advanced the state of the art in various AI tasks, such as natural language understanding and text generation, and image processing, and multimodal modeling. In this talk, we will first introduce the development of AI in the past decades, in particular from the angle of China. We will also talk about the opportunities, challenges, and risks of AGI in the future, and its impact on the Web. In the second part of the talk, we will use ChatGLM, an alternative but open sourced model to ChatGPT, as an example to explain our understandings and insights derived during the implementation of the model.

Deep reinforcement learning models are vulnerable to adversarial attacks that can decrease the cumulative expected reward of a victim by manipulating its observations. Despite the efficiency of previous optimization-based methods for generating adversarial noise in supervised learning, such methods might not achieve the lowest cumulative reward since they do not generally explore the environmental dynamics. Herein, a framework is provided to better understand the existing methods by reformulating the problem of adversarial attacks on reinforcement learning in the function space. The reformulation approach adopted herein generates an optimal adversary in the function space of targeted attacks, repelling them via a generic two-stage framework. In the first stage, a deceptive policy is trained by hacking the environment and discovering a set of trajectories routing to the lowest reward or the worst-case performance. Next, the adversary misleads the victim to imitate the deceptive policy by perturbing the observations. Compared to existing approaches, it is theoretically shown that our adversary is strong under an appropriate noise level. Extensive experiments demonstrate the superiority of the proposed method in terms of efficiency and effectiveness, achieving state-of-the-art performance in both Atari and MuJoCo environments.

Neural-symbolic models provide a powerful tool to tackle complex visual reasoning tasks by combining symbolic program execution for reasoning and deep representation learning for visual recognition. A probabilistic formulation of such models with stochastic latent variables can obtain an interpretable and legible reasoning system with less supervision. However, it is still nontrivial to generate reasonable symbolic structures without the guidance of domain knowledge, since it generally involves an optimization problem with both continuous and discrete variables. Despite the challenges, the interpretability of such symbolic structures provides an interface to regularize their generation by domain knowledge. In this article, we propose to incorporate the available domain knowledge into the learning process of probabilistic neural-symbolic (PNS) models via posterior constraints that directly regularize the structure posterior. In this way, our model is able to identify a middle point where the structure generation process mainly learns from data but also selectively borrows information from domain knowledge. We further present inductive reasoning where the posterior constraints can be automatically reweighted to handle noisy annotations. The experimental results show that our method achieves state-of-the-art performance on major abstract reasoning datasets and enjoys good generalization capability and data efficiency.

Recent works have proposed to craft adversarial clothes for evading person detectors, while they are either only effective at limited viewing angles or very conspicuous to humans. We aim to craft adversarial texture for clothes based on 3D modeling, an idea that has been used to craft rigid adversarial objects such as a 3D-printed turtle. Unlike rigid objects, humans and clothes are non-rigid, leading to difficulties in physical realization. In order to craft natural-looking adversarial clothes that can evade person detectors at multiple viewing angles, we propose adversarial camouflage textures (AdvCaT) that resemble one kind of the typical textures of daily clothes, camouflage textures. We leverage the Voronoi diagram and Gumbel-softmax trick to parameterize the camouflage textures and optimize the parameters via 3D modeling. Moreover, we propose an efficient augmentation pipeline on 3D meshes combining topologically plausible projection (TopoProj) and Thin Plate Spline (TPS) to narrow the gap between digital and real-world objects. We printed the developed 3D texture pieces on fabric materials and tailored them into T-shirts and trousers. Experiments show high attack success rates of these clothes against multiple detectors.

Telecommunication systems may have different states, such as normal, warning, etc. Predicting key performance indicators for telecommunication systems in different states is important for monitoring and anomaly detection. However, there are challenges due to the different data evolution mechanisms in different cells and the unbalanced data sizes in different states. To address this, we propose a Meta-Learning Bayesian Hierarchical Model (ML-BHM) that incorporates knowledge sharing between cells and states. We apply this model to a real case study for cell KPI prediction and show that its effectiveness and efficiency.

Deep reinforcement learning (RL) is a powerful approach for solving optimal control problems. However, RL-trained policies often suffer from the action fluctuation problem, where the consecutive actions significantly differ despite only slight state variations. This problem results in mechanical components' wear and tear and poses safety hazards. The action fluctuation is caused by the high Lipschitz constant of actor networks. To address this problem, we propose a neural network named LipsNet. We propose the Multi-dimensional Gradient Normalization (MGN) method, to constrain the Lipschitz constant of networks with multi-dimensional input and output. Benefiting from MGN, LipsNet achieves Lipschitz continuity, allowing smooth actions while preserving control performance by adjusting Lipschitz constant. LipsNet addresses the action fluctuation problem at network level rather than algorithm level, which can serve as actor networks in most RL algorithms, making it more flexible and user-friendly than previous works. Experiments demonstrate that LipsNet has good landscape smoothness and noise robustness, resulting in significantly smoother action compared to the Multilayer Perceptron.

Adversarial attacks can easily fool object recognition systems based on deep neural networks (DNNs). Although many defense methods have been proposed in recent years, most of them can still be adaptively evaded. One reason for the weak adversarial robustness may be that DNNs are only supervised by category labels and do not have part-based inductive bias like the recognition process of humans. Inspired by a well-known theory in cognitive psychology -- recognition-by-components, we propose a novel object recognition model ROCK (Recognizing Object by Components with human prior Knowledge). It first segments parts of objects from images, then scores part segmentation results with predefined human prior knowledge, and finally outputs prediction based on the scores. The first stage of ROCK corresponds to the process of decomposing objects into parts in human vision. The second stage corresponds to the decision process of the human brain. ROCK shows better robustness than classical recognition models across various attack settings. These results encourage researchers to rethink the rationality of currently widely-used DNN-based object recognition models and explore the potential of part-based models, once important but recently ignored, for improving robustness.

Reliable object detection under dark environment is of wide applications but severely challenged by heavy noise washing out informative features and uneven radiance caused by nighttime illuminations. These unique features of dark videos would largely degenerate the performance of existing detectors. To address this issue, specially designed algorithms being able to extract and fuse the weak features buried in the low-quality videos are of vital importance. Bearing these in mind, we propose illumination-aware spatio-temporal feature fusion modules for low-light video object detection and implement a Dark Video Detector under a TRansformer network structure, dubbed as DVD-TR. Firstly, we use a dual-backbone Transformer to extract separate complementary features and fuse them to strengthen the network’s feature extraction capability. Secondly, we incorporate a spatio-temporal sampling mechanism to aggregate features from multiple frames, which can enhance detection accuracy in dark videos. Thirdly, we use a small encoder–decoder network to obtain irradiance distribution which is further incorporated for illumination-aware feature fusion. Extensive experiments on large-scale multi-illuminance dark video benchmark show that DVD-TR outperforms state-of-the-art video detectors by a large margin and validate the effectiveness of the proposed approach.

Thanks to the impressive progress of large-scale vision-language pretraining, recent recognition models can classify arbitrary objects in a zero-shot and open-set manner, with a surprisingly high accuracy. However, translating this success to semantic segmentation is not trivial, because this dense prediction task requires not only accurate semantic understanding but also fine shape delineation and existing vision-language models are trained with image-level language descriptions. To bridge this gap, we pursue shape-aware zero-shot semantic segmentation in this study. Inspired by classical spectral methods in the image segmentation literature, we propose to leverage the eigen vectors of Laplacian matrices constructed with self-supervised pixel-wise features to promote shape-awareness. Despite that this simple and effective technique does not make use of the masks of seen classes at all, we demonstrate that it out-performs a state-of-the-art shape-aware formulation that aligns ground truth and predicted edges during training. We also delve into the performance gains achieved on different datasets using different backbones and draw several interesting and conclusive observations: the benefits of promoting shape-awareness highly relates to mask compactness and language embedding locality. Finally, our method sets new state-of-the-art performance for zero-shot semantic segmentation on both Pascal and COCO, with significant margins. Code and models will be accessed at SAZS.

There have been two competing paradigms in artificial intelligence (AI) development ever since its birth in 1956, i.e., symbolism and connectionism (or sub-symbolism). While symbolism dominated AI research by the end of 1980s, connectionism gained momentum in the 1990s and is gradually displacing symbolism. This paper considers symbolism as the first generation of AI and connectionism as the second generation. However, each of these two paradigms simulates the human mind from only one perspective. AI cannot achieve true human behaviors by relying on only one paradigm. In order to develop novel AI technologies that are safe, reliable, and extensible, it is necessary to establish a new explainable and robust AI theory. To this end, this paper looks toward developing a third generation artificial intelligence by combining the current paradigms.

Deep neural networks are vulnerable to adversarial attacks. We consider adversarial defense in the case of zero-shot image classification setting, which has rarely been explored because both adversarial defense and zero-shot learning are challenging. We propose LAAT, a novel Language-driven, Anchor-based Adversarial Training strategy, to improve the adversarial robustness in a zero-shot setting. LAAT uses a text encoder to obtain fixed anchors (normalized feature embeddings) of each category, then uses these anchors to perform adversarial training. The text encoder has the property that semantically similar categories can be mapped to neighboring anchors in the feature space. By leveraging this property, LAAT can make the image model adversarially robust on novel categories without any extra examples. Experimental results show that our method achieves impressive zero-shot adversarial performance, even surpassing the previous state-of-the-art adversarially robust one-shot methods in most attacking settings. When models are trained with LAAT on large datasets like ImageNet-1K, they can have substantial zero-shot adversarial robustness across several downstream datasets.

Generative Adversarial Nets (GANs) have shown promise in image generation and semi-supervised learning (SSL). However, existing GANs in SSL have two problems: (1) the generator and the discriminator (i.e. the classifier) may not be optimal at the same time; and (2) the generator cannot control the semantics of the generated samples. The problems essentially arise from the two-player formulation, where a single discriminator shares incompatible roles of identifying fake samples and predicting labels and it only estimates the data without considering the labels. To address the problems, we present triple generative adversarial net (Triple-GAN), which consists of three players---a generator, a discriminator and a classifier. The generator and the classifier characterize the conditional distributions between images and labels, and the discriminator solely focuses on identifying fake image-label pairs. We design compatible utilities to ensure that the distributions characterized by the classifier and the generator both converge to the data distribution. Our results on various datasets demonstrate that Triple-GAN as a unified model can simultaneously (1) achieve the state-of-the-art classification results among deep generative models, and (2) disentangle the classes and styles of the input and transfer smoothly in the data space via interpolation in the latent space class-conditionally.