Diffusion Transformer has shown remarkable abilities in generating high-fidelity videos, delivering visually coherent frames and rich details over extended durations. However, existing video generation models still fall short in subject-consistent video generation due to an inherent difficulty in parsing prompts that specify complex spatial relationships, temporal logic, and interactions among multiple subjects. To address this issue, we propose BindWeave, a unified framework that handles a broad range of subject-to-video scenarios from single-subject cases to complex multi-subject scenes with heterogeneous entities. To bind complex prompt semantics to concrete visual subjects, we introduce an MLLM-DiT framework in which a pretrained multimodal large language model performs deep cross-modal reasoning to ground entities and disentangle roles, attributes, and interactions, yielding subject-aware hidden states that condition the diffusion transformer for high-fidelity subject-consistent video generation. Experiments on the OpenS2V benchmark demonstrate that our method achieves superior performance across subject consistency, naturalness, and text relevance in generated videos, outperforming existing open-source and commercial models.

As short videos have risen in popularity, the role of video content in advertising has become increasingly significant. Typically, advertisers record a large amount of raw footage about the product and then create numerous different short-form advertisement videos based on this raw footage. Creating such videos mainly involves editing raw footage and writing advertisement scripts, which requires a certain level of creative ability. It is usually challenging to create many different video contents for the same product, and manual efficiency is often low. In this paper, we present VC-LLM, a framework powered by Large Language Models for the automatic creation of high-quality short-form advertisement videos. Our approach leverages high-resolution spatial input and low-resolution temporal input to represent video clips more effectively, capturing both fine-grained visual details and broader temporal dynamics. In addition, during training, we incorporate supplementary information generated by rewriting the ground truth text, ensuring that all key output information can be directly traced back to the input, thereby reducing model hallucinations. We also designed a benchmark to evaluate the quality of the created videos. Experiments show that VC-LLM based on GPT-4o can produce videos comparable to those created by humans. Furthermore, we collected numerous high-quality short advertisement videos to create a pre-training dataset and manually cleaned a portion of the data to construct a high-quality fine-tuning dataset. Experiments indicate that, on the benchmark, the VC-LLM based on fine-tuned LLM can produce videos with superior narrative logic compared to those created by the VC-LLM based on GPT-4o.

This study introduces a novel selective multi-branch network architecture designed to speed up object localization and classification on low-performance edge devices. The concept builds upon the You Only Look at Interested Cells (YOLIC) method, which was proposed by us earlier. In this approach, we categorize cells of interest (CoIs) into distinct regions of interest (RoIs) based on their locations and urgency. We then employ some expert branch networks for detailed object detection in each of the RoIs. To steer these branches effectively, a selective attention unit is added into the detection process. This unit can locate RoIs that are likely to contain objects under concern and trigger corresponding expert branch networks. The inference can be more efficient because only part of the feature map is used to make decisions. Through extensive experiments on various datasets, the proposed network demonstrates its ability to reduce the inference time while still maintaining competitive performance levels compared to the current detection algorithms.

In the realm of Tiny AI, we introduce “You Only Look at Interested Cells” (YOLIC), an efficient method for object localization and classification on edge devices. Through seamlessly blending the strengths of semantic segmentation and object detection, YOLIC provides improved computational efficiency and precision compared to traditional methods. By adopting Cells of Interest for classification instead of individual pixels, YOLIC encapsulates relevant information, reduces computational load, and enables rough object shape inference. Importantly, the need for bounding box regression is obviated, as YOLIC capitalizes on the predetermined cell configuration that provides information about potential object location, size, and shape. To tackle the issue of single-label classification limitations, a multi-label classification approach is applied to each cell for effectively recognizing overlapping or closely situated objects. This paper presents extensive experiments on multiple datasets to demonstrate that YOLIC achieves detection performance comparable to the state-of-the-art YOLO algorithms while surpassing in speed, exceeding 30fps on a Raspberry Pi 4B CPU.

Road obstacle detection is one of the crucial tasks for the safe and efficient operation of autonomous vehicles. Existing detection methods often try to detect all objects in the image (i.e. a frame in a video stream). However, it is important to note that not all objects are equally dangerous and that the detection system should not react equally to them. To reduce computation costs (e.g. time and power consumption) in a mobile device, this study introduces a cost-sensitive obstacle detection method to leverage the performance of YOLIC (You Only Look at Interested Cells), a method proposed by us previously. The new method allows YOLIC to pay more attention to important areas for driving. In the experiments, we assign high weights to areas close to the vehicle, and the weights vary depending on the detection distance, direction, and relation to driving. Experimental results on two road obstacle datasets demonstrate that the proposed cost-sensitive detection method can effectively reduce the costs in the most dangerous areas of a given image compared with the baseline YOLIC model. Moreover, our fastest model can achieve real-time performance on a Raspberry Pi 4B, making it possible to deploy the proposed method in low-cost vehicles such as scooters and delivery robots.

Misses in artificial intelligence inference due to hardware failures can cause serious accidents in mission-critical systems involving automated guided vehicles, delivery robots, and so on. It is desirable to realize a fault-tolerant AI system that continues to operate normally even when a failure occurs. Conventional fault-tolerant techniques have the problems of increased computational cost and power consumption. In this paper, we proposed an accurate and fault-tolerant object localization and classification based on an ensemble of dual modular redundancy of ternary You Only Look at Interested Cells (YOLIC) models. In our experiments, we evaluate the proposed method with two types of road surface risk detection datasets. We demonstrate that the fault-tolerant model consisting of three ternary (ternary-weight) models achieves comparable accuracy and reduces the computational cost by 89.5% (79.7%) compared to the triple modular redundancy (TMR) of the floating-point model of the same structure.

Person search aims at localizing and recognizing query persons from raw video frames, which is a combination of two sub-tasks, i.e., pedestrian detection and person re-identification. The dominant fashion is termed as the one-step person search that jointly optimizes detection and identification in a unified network, exhibiting higher efficiency. However, there remain major challenges: (i) conflicting objectives of multiple sub-tasks under the shared feature space, (ii) inconsistent memory bank caused by the limited batch size, (iii) underutilized unlabeled identities during the identification learning. To address these issues, we develop an enhanced decoupled and memory-reinforced network (DMRNet++). First, we simplify the standard tightly coupled pipelines and establish a task-decoupled framework (TDF). Second, we build a memory-reinforced mechanism (MRM), with a slow-moving average of the network to better encode the consistency of the memorized features. Third, considering the potential of unlabeled samples, we model the recognition process as semi-supervised learning. An unlabeled-aided contrastive loss (UCL) is developed to boost the identification feature learning by exploiting the aggregation of unlabeled identities. Experimentally, the proposed DMRNet++ obtains the mAP of 94.5% and 52.1% on CUHK-SYSU and PRW datasets, which exceeds most existing methods.

To detect and locate road objects precisely is crucial for the routing and navigation of autonomous vehicles and robots. A deep neural network can be used for this purpose but the inference cost is usually too high for real-time processing with low-cost devices. How to guide the network to attend risky areas plays an essential role in cost reduction. This paper proposes a novel multi-branch network that selectively extracts pre-specified features and detects road risks in corresponding regions of interest through different branch networks. The proposed network consists of one common block to obtain global features and several branches to detect road risks in particular interested regions. Experimental results show that our method can reduce the inference cost significantly while preserving accuracy.

We propose a sparse end-to-end multi-person pose regression framework, termed QueryPose, which can directly predict multi-person keypoint sequences from the input image. The existing end-to-end methods rely on dense representations to preserve the spatial detail and structure for precise keypoint localization. However, the dense paradigm introduces complex and redundant post-processes during inference. In our framework, each human instance is encoded by several learnable spatial-aware part-level queries associated with an instance-level query. First, we propose the Spatial Part Embedding Generation Module (SPEGM) that considers the local spatial attention mechanism to generate several spatial-sensitive part embeddings, which contain spatial details and structural information for enhancing the part-level queries. Second, we introduce the Selective Iteration Module (SIM) to adaptively update the sparse part-level queries via the generated spatial-sensitive part embeddings stage-by-stage. Based on the two proposed modules, the part-level queries are able to fully encode the spatial details and structural information for precise keypoint regression. With the bipartite matching, QueryPose avoids the hand-designed post-processes and surpasses the existing dense end-to-end methods with 73.6 AP on MS COCO mini-val set and 72.7 AP on CrowdPose test set. Code is available at https://github.com/buptxyb666/QueryPose.

Multi-person pose estimation generally follows top-down and bottom-up paradigms. Both of them use an extra stage ($\boldsymbol{e.g.,}$ human detection in top-down paradigm or grouping process in bottom-up paradigm) to build the relationship between the human instance and corresponding keypoints, thus leading to the high computation cost and redundant two-stage pipeline. To address the above issue, we propose to represent the human parts as adaptive points and introduce a fine-grained body representation method. The novel body representation is able to sufficiently encode the diverse pose information and effectively model the relationship between the human instance and corresponding keypoints in a single-forward pass. With the proposed body representation, we further deliver a compact single-stage multi-person pose regression network, termed as AdaptivePose. During inference, our proposed network only needs a single-step decode operation to form the multi-person pose without complex post-processes and refinements. We employ AdaptivePose for both 2D/3D multi-person pose estimation tasks to verify the effectiveness of AdaptivePose. Without any bells and whistles, we achieve the most competitive performance on MS COCO and CrowdPose in terms of accuracy and speed. Furthermore, the outstanding performance on MuCo-3DHP and MuPoTS-3D further demonstrates the effectiveness and generalizability on 3D scenes. Code is available at https://github.com/buptxyb666/AdaptivePose.

Human emotion is perceived not only in facial expressions but also in every kind of body language, including a human’s walking gait. In this paper, we propose a VFL framework for classifying a human’s walking gait into emotions. This framework introduces deep learning methods, which are merely applied to gait data, as the main methods for performing emotion recognition tasks with walking gaits. First, we obtain gait movement data from original walking videos or records and use the gait data that contain only body keypoint positions as input. Then, we expand the data to other kinetic features, rebuild the main skeleton in images, and extract vision features from the images. Based on the data and fusion features, we perform feature fusion and apply our framework to the fused features to complete the task. On various human movement datasets, we obtain an overall accuracy of 92 percent.

Video scene parsing is a long-standing challenging task in computer vision, aiming to assign pre-defined semantic labels to pixels of all frames in a given video. Compared with image semantic segmentation, this task pays more attention on studying how to adopt the temporal information to obtain higher predictive accuracy. In this report, we introduce our solution for the 1st Video Scene Parsing in the Wild Challenge, which achieves a mIoU of 57.44 and obtained the 2nd place (our team name is CharlesBLWX).

In this paper, we propose a multi-branch neural network architecture named Coupled Modular Neural Network (CMNN). A CMNN is a network consisting of $\beta $ closely coupled sub-networks, where $\beta $ is termed as the branching factor in this paper. We call the whole network a super-graph and each sub-network a sub-graph. Each sub-graph is a stand-alone neural network and shares a common block with other sub-graphs. To effectively leverage the super-graph we propose a simple but easy-to-implement Round-Robin-based learning algorithm. Each training iteration contains two phases. In the first phase, we choose a sub-graph in a Round-Robin fashion and train it using knowledge of the super-graph (distillation). In the second phase, we fine-tune the super-graph based on the updated sub-graphs. This algorithm produces a different copy of the super-graph at each iteration which acts as an improved teacher network for the sub-graph; and a different copy of one of the sub-graphs which functions as a new building block for the super-graph. To validate and test CMNN and the proposed algorithm, we conduct experiments on CIFAR-10, CIFAR-100, Tiny ImageNet and a private On-Road-Risk (ORR) datasets. Empirical results on all these four datasets indicate that we not only obtain a strong sub-graph network, the learning framework can also produce strong ensemble performance which substantiates the diversity introduced throughout the learning framework.

Supervised learning is dominant in person search, but it requires elaborate labeling of bounding boxes and identities. Large-scale labeled training data is often difficult to collect, especially for person identities. A natural question is whether a good person search model can be trained without the need of identity supervision. In this paper, we present a weakly supervised setting where only bounding box annotations are available. Based on this new setting, we provide an effective baseline model termed Region Siamese Networks (R-SiamNets). Towards learning useful representations for recognition in the absence of identity labels, we supervise the R-SiamNet with instance-level consistency loss and cluster-level contrastive loss. For instance-level consistency learning, the R-SiamNet is constrained to extract consistent features from each person region with or without out-of-region context. For cluster-level contrastive learning, we enforce the aggregation of closest instances and the separation of dissimilar ones in feature space. Extensive experiments validate the utility of our weakly supervised method. Our model achieves the rank-1 of 87.1% and mAP of 86.0% on CUHK-SYSU benchmark, which surpasses several fully supervised methods, such as OIM [36] and MGTS [4], by a clear margin. More promising performance can be reached by incorporating extra training data. We hope this work could encourage the future research in this field.

We propose an optimization of Modular Selective Network or MS-Net by reducing the number of expert network evaluations. MS-Net is composed of a router and a set of expert networks. In our original proposal, MS-Net is constructed based on a Round-Robin dataset partition with controlled redundancy among the subsets of classes. In this paper, we propose a novel way for reducing the inference cost by performing Inter-Class-Correlation (ICC) analysis through calculating the joint-probability of appearance of top-2 classes in router’s prediction. Next, we construct subset of classes on the most frequently occurring class pairs and train experts on those subsets. We do not enforce redundancy in these subsets, thus during inference, only one expert is leveraged per sample. Our empirical results show that, with the ResNet-20 backbone, the optimized MS-Net reduces parameter utilization by over 70% yet performs with neck and neck score with the original MS-Net for CIFAR-10 and CIFAR-100 dataset.

On-road risk detection and alert system is a crucial and important task in our day to day life. Deep Learning approaches have got much attention in solving this noble task. In this paper, we have performed a comparative study on two recent architectures that handle the on-road risk detection task, which are Block-Wise Detection and Modular Selective Network (MSNet). In the Block-Wise Detection, we have used the VGG19, VGG19-BN, and ResNet family as the backbone network. On the other hand, for MS-Net we have used the ResNet-44 as the router and ResNet-101 as the expert network. In this experiment, we evaluate our model on an "on-road risk detection dataset", which was created by our research group using an RGB-D sensor mounted on a senior car. On this dataset, we can achieve an accuracy of 89.40 % for MS-Net. For the Block-Wise Detection model, we can achieve an accuracy of 90.51% if we use ResNet50 as the backbone network. However, if we choose the network models used in MS-Net, we can double the inference speed. Thus, compared with Block-Wise Detection, we think the overall performance of MS-NET is better, and is potentially more useful for driving assistance of elderly drivers.

We propose a modular Deep Convolutional Neural Network (DCNN) architecture which has the property of block-like design and re-usage of parameters by certain blocks. We leverage networks from the ResNet family as the backbone for our proposed architecture. The proposed network architecture is composed of three primary blocks, i) common block ii) hidden blocks and iii) output block. The common block mainly focuses on learning low-level features which are afterward fed to ρ hidden blocks at a time, where ρ is the branching factor. The hidden blocks are placed in a row-wise manner and replicated ρ times depending on the computational availability. The output block takes features as an input from the common block and all the hidden blocks during both training and testing phases. In addition, each of the hidden blocks together with the common block is a stand-alone neural network, which implies that we have ρ neural networks coupled in a single framework. During the training phase, in every single epoch, each of the stand-alone neural networks (i.e. common block + hidden block) is trained in Round Robin fashion followed by fine-tuning the whole network graph. Inference can be performed using either the whole network graph or just any sub-graph of it. To substantiate the effectiveness of our proposed neural architecture we performed experiment on CIFAR-10 and CIFAR-100 datasets.

On-road risk detection is one of the most important applications of Deep Neural Network (DNN). A good on-road risk detection system can accurately detect dangerous obstacles on the road and provides drivers with early warning information to avoid danger. However, current high-precision detection systems usually use a cumbersome DNN, which require a large number of computing resources. Due to the limitations of mobile devices such as limited computing power and power consumption problem, it is difficult to apply the high-precision detection model to such devices for running real-time applications. Therefore a fast and accurate model is needed to solve the on-road risk detection problem for low-cost mobile devices. In this paper, we propose a real-time on-road risk detection framework for the low computational platform. The proposed framework uses a tiny neural network model that improves accuracy by leveraging the knowledge distillation technique. By using the proposed method, we conduct our experiments in a Raspberry Pi using one Intel Neural Compute Stick-2 (NCS-2). The practical results show that our detection system can successfully detect the position and type of road obstacles in real-time while maintaining performance comparable to more complex DNN.

We propose a modular architecture of Deep Neural Network (DNN) for multi-class classification task. The architecture consists of two parts, a router network and a set of expert networks. In this architecture, for a C-class classification problem, we have exactly C experts. The backbone network for these experts and the router are built with simple and identical DNN architecture. For each class, the modular network has a certain number $$\rho$$ of expert networks specializing in that particular class, where $$\rho$$ is called the redundancy rate in this study. We demonstrate that $$\rho$$ plays a vital role in the performance of the network. Although these experts are light weight and weak learners alone, together they match the performance of more complex DNNs. We train the network in two phase wherein, first the router is trained on the whole set of training data followed by training each expert network enforced by a new stochastic objective function that facilitates alternative training on a small subset of expert data and the whole set of data. This alternative training provides an additional form of regularization and avoids over-fitting the expert network on subset data. During the testing phase, the router dynamically selects a fixed number of experts for further evaluation of the input datum. The modular nature and low parameter requirement of the network makes it very suitable in distributed and low computational environments. Extensive empirical study and theoretical analysis on CIFAR-10, CIFAR-100 and F-MNIST substantiate the effectiveness and efficiency of our proposed modular network.

In this paper, we study real-time object detection based on cell-wise segmentation. Existing object detection methods usually focus on detecting interesting object's positions and sizes and demand expensive computing resources. This process makes it difficult to achieve high-speed and high-precision detection with low-cost devices. We propose a method called You Only Look at Interested Cells or in-short YOLIC to solve the problem by focusing on predefined interested cells (i.e., subregions) in an image. A key challenge here is how to predict the object types contained in all interested cells efficiently, all at once. Instead of using multiple predictors for all interested cells, we use only one deep learner to classify all interested cells. In other words, YOLIC applies the concept of multi-label classification for object detection. YOLIC can use exiting classification models without any structural change. The main point is to define a proper loss function for training. Using on-road risk detection as a test case, we confirmed that YOLIC is significantly faster and accurate than YOLO-v3 in terms of FPS and F1-score.