Accurately predicting the future trajectories of surrounding agents and planning a safe, comfortable trajectory for the automated vehicle (AV) are of utmost importance for autonomous driving systems (ADS). In particular, the rise of machine learning-driven interactive prediction and planning frameworks has revolutionized this field. Existing autonomous driving systems, such as modular stacks and end-to-end frameworks, remain challenges as they commonly model the prediction and planning sequentially, neglecting the dynamic reactions of surrounding agents to the planning behaviors of the AV. Instead, interactive prediction and planning frameworks benefit from bi-directional interactive modeling between the AV and surrounding agents in a joint optimization manner, circumventing the drawbacks associated with existing autonomous driving systems and boosting safety and comfortable transportation. Through gradual modeling paradigms ranging from marginal prediction, conditional prediction, and interactive prediction and planning, this paper delves into fundamental principles and the growth journey of the ultimate target for autonomous driving. Built upon this, we systematically present a comprehensive review of state-of-the-art papers, encompassing the methodology, experimental platforms, and evaluation criteria during the development of interactive prediction and planning. Additionally, this paper details critical challenges concerning scene comprehension, planning safety, robustness, generalization and deployment. Accordingly, we discuss current advancements such as hot topics of large language models (LLMs), world models, distillation strategies, reinforcement learning from human feedback (RLHF), and data balance and long-tail learning, finally proposing future promising directions to inspire continued innovation and exploration.

Integrated prediction and planning of autonomous driving leverages imitation learning (IL) to mimic rational motion behaviors from a wide span of expert demonstrations. However, real-world data collected from human experts are commonly dominated by common cases, causing the learning-based model to suffer from distribution shift under rare, non-trivial cases. To address this problem, we unlock motion simulation data scaling properties and propose a novel motion simulation-enhanced imitation learning framework to significantly improve generalization and adaptivity of integrated prediction and planning models in challenging, highly-interactive driving scenarios. In this work, we first propose a motion simulation-enhanced data generation engine to rollout diverse and interactive multi-agent motion states as pseudo-expert data for subsequent action supervision. Further, we develop a sim-real co-training strategy to optimize integrated prediction and planning models upon both real-world and simulated motion samples. Finally, extensive experiments conducted on large-scale open-pit motion datasets demonstrate that the proposed motion simulation-enhanced imitation learning framework improves planning safety and feasibility in challenging scenarios.

Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been applied to traffic simulation and achieves iterative improvements via supervised fine-tuning (SFT). However, such methods limit active exploration of potentially valuable motion tokens, particularly in suboptimal regions. Entropy patterns provide a promising perspective for enabling exploration driven by motion token uncertainty. Motivated by this insight, we propose a novel tokenized traffic simulation policy, R1Sim, which represents an initial attempt to explore reinforcement learning based on motion token entropy patterns, and systematically analyzes the impact of different motion tokens on simulation outcomes. Specifically, we introduce an entropy-guided adaptive sampling mechanism that focuses on previously overlooked motion tokens with high uncertainty yet high potential. We further optimize motion behaviors using Group Relative Policy Optimization (GRPO), guided by a safety-aware reward design. Overall, these components enable a balanced exploration-exploitation trade-off through diverse high-uncertainty sampling and group-wise comparative estimation, resulting in realistic, safe, and diverse multi-agent behaviors. Extensive experiments on the Waymo Sim Agent benchmark demonstrate that R1Sim achieves competitive performance compared to state-of-the-art methods.

In cluster computing, job scheduling enhances resource utilization and operational efficiency based on user requirements. Scheduling systems utilizing deep reinforcement learning have proven effective in simple scenarios, such as homogeneous clusters and single resource pools. In this paper, we extend deep reinforcement learning to more realistic scenarios: heterogeneous load-aware computing clusters, referred to as DeepHCM. DeepHCM makes scheduling decisions based on the overall state of the cluster, incorporating aggregated information about tasks and nodes. Experimental results on various simulated heterogeneous load-aware computing clusters demonstrate the effectiveness of deep reinforcement learning. Under a similar makespan, DeepHCM exhibits smaller slowdown, indicating better scheduling performance. In a cluster with three heterogeneous nodes, the slowdown was reduced from 10.461 to 8.850. DeepHCM significantly improved scheduling performance in heterogeneous load-aware clusters, particularly when resources were limited and workloads were heavy.

The rise of GPU-based high-performance computing (HPC) has driven the widespread adoption of parallel programming models such as CUDA. Yet, the inherent complexity of parallel programming creates a demand for the automated sequential-to-parallel approaches. However, data scarcity poses a significant challenge for machine learning-based sequential-to-parallel code translation. Although recent back-translation methods show promise, they still fail to ensure functional equivalence in the translated code. In this paper, we propose \textbf{QiMeng-MuPa}, a novel \textbf{Mu}tual-Supervised Learning framework for Sequential-to-\textbf{Pa}rallel code translation, to address the functional equivalence issue. QiMeng-MuPa consists of two models, a Translator and a Tester. Through an iterative loop consisting of Co-verify and Co-evolve steps, the Translator and the Tester mutually generate data for each other and improve collectively. The Tester generates unit tests to verify and filter functionally equivalent translated code, thereby evolving the Translator, while the Translator generates translated code as augmented input to evolve the Tester. Experimental results demonstrate that QiMeng-MuPa significantly enhances the performance of the base models: when applied to Qwen2.5-Coder, it not only improves Pass@1 by up to 28.91\% and boosts Tester performance by 68.90\%, but also outperforms the previous state-of-the-art method CodeRosetta by 1.56 and 6.92 in BLEU and CodeBLEU scores, while achieving performance comparable to DeepSeek-R1 and GPT-4.1. Our code is available at \url{https://github.com/kcxain/mupa}.

In semiconductor manufacturing, the deposition quality of thin film plays a critical role in both production efficiency and device performance. Plasma-enhanced Chemical Vapor Deposition (PECVD) has emerged as a widely adopted technique due to its low process temperature and high deposition rate. A comprehensive understanding of the thin film deposition mechanism of PECVD is essential for optimizing process parameters. This study introduces a Spatiotemporal Attention-enhanced ConvLSTM (STAE-ConvLSTM) model for predicting thin film deposition, specifically modeling the SiOxNy deposition process in PECVD. By integrating advanced spatiotemporal prediction and self-attention techniques, the model effectively captures global spatiotemporal dependencies and the influence of process parameters on film growth. Through extensive testing on diverse datasets, the proposed model demonstrates superior performance in predicting thin film deposition compared to both standard ConvLSTM networks and traditional physical-chemistry-based models. The experimental results highlight the STAE-ConvLSTM model’s ability to handle complex substrate structures with high precision. Additionally, long-sequence prediction experiments confirm the model’s robustness. The proposed model can predict thin film deposition process effectively.

Modeling complicated interactions among the egovehicle, road agents, and map elements has been a crucial part for safety-critical autonomous driving. Previous work on end-to-end autonomous driving relies on the attention mechanism to handle heterogeneous interactions, which fails to capture geometric priors and is also computationally intensive. In this paper, we propose the Interaction Scene Graph (ISG) as a unified method to model the interactions among the ego-vehicle, road agents, and map elements. With the representation of the ISG, the driving agents aggregate essential information from the most influential elements, including the road agents with potential collisions and the map elements to follow. Since a mass of unnecessary interactions are omitted, the more efficient scene-graph-based framework is able to focus on indispensable connections and leads to better performance. We evaluate the proposed method for end-to-end autonomous driving on the nuScenes dataset. Compared with strong baselines, our method significantly outperforms in full-stack driving tasks.

Autonomous vehicles (AVs) require accurately predicting future trajectories of neighboring agents and identifying highly interactive agents from them to make safe downstream planning. However, the immense uncertainty easily results in temporal incoherence between trajectory predictions from adjacent frames, misleading the selection of interactive agents and posing safety risks for planning. We find that the temporal coherence of trajectory predictions can be improved by predetermined, map-adaptive paths. This paper proposes an Integrated Path-guided Prediction and Planning (IP3) framework under the pre-train and fine-tune learning paradigm for temporal coherence prediction and safe planning. First, we adopt a new concept of path-trajectory query pairs to pre-train a dual-query Transformer for performing modality alignment between path and trajectory predictions, estimating accurate distributions over them, and further achieving better temporal coherence trajectory prediction. Second, we fine-tune the planning policy by a collision-aware imitation loss to prevent potential collision threats from highly interactive agents. Besides, we incorporate a path-guided interactive agent selection strategy to enable accurate identification of highly interactive agents and boost safe planning with temporal coherence prediction. Experiments validate IP3 outperforms previous state-of-the-arts in both open-loop and closed-loop planning tests. Especially, IP3 surpasses strong baselines on the challenging Waymax benchmark, greatly decreasing the safety-critical metric collision rate by 68.40 %.

Recently, anchor-based trajectory prediction methods have shown promising performance, which directly selects a final set of anchors as future intents in the spatio-temporal coupled space. However, such methods typically neglect a deeper semantic interpretation of path intents and suffer from inferior performance under the imperfect High-Definition (HD) map. To address this challenge, we propose a novel Planning-inspired Hierarchical (PiH) trajectory prediction framework that selects path and speed intents through a hierarchical lateral and longitudinal decomposition. Especially, a hybrid lateral predictor is presented to select a set of fixed-distance lateral paths from map-based road-following and cluster-based free-move path candidates. Then, the subsequent longitudinal predictor selects plausible goals sampled from a set of lateral paths as speed intents. Finally, a trajectory decoder is given to generate future trajectories conditioned on a categorical distribution over lateral-longitudinal intents. Experiments demonstrate that PiH achieves competitive and more balanced results against state-of-the-art methods on the Argoverse motion forecasting benchmark and has the strongest robustness under the imperfect HD map.

Predicting future trajectories of pairwise traffic agents in highly interactive scenarios, such as cut-in, yielding, and merging, is challenging for autonomous driving. The existing works either treat such a problem as a marginal prediction task or perform single-axis factorized joint prediction, where the former strategy produces individual predictions without considering future interaction, while the latter strategy conducts conditional trajectory-oriented prediction via agentwise interaction or achieves conditional rollout-oriented prediction via timewise interaction. In this article, we propose a novel double-axis factorized joint prediction pipeline, namely, conditional goal-oriented trajectory prediction (CGTP) framework, which models future interaction both along the agent and time axes to achieve goal and trajectory interactive prediction. First, a goals-of-interest network (GoINet) is designed to extract fine-grained features of goal candidates via hierarchical vectorized representation. Furthermore, we propose a conditional goal prediction network (CGPNet) to produce multimodal goal pairs in an agentwise conditional manner, along with a newly designed goal interactive loss to better learn the joint distribution of the intermediate interpretable modes. Explicitly guided by the goal-pair predictions, we propose a goal-oriented trajectory rollout network (GTRNet) to predict scene-compliant trajectory pairs via timewise interactive rollouts. Extensive experimental results confirm that the proposed CGTP outperforms the state-of-the-art (SOTA) prediction models on the Waymo open motion dataset (WOMD), Argoverse motion forecasting dataset, and In-house cut-in dataset. Code is available at https://github.com/LiDinga/CGTP/.

Predicting the future trajectories of surrounding vehicles plays a vital role in ensuring the safety of autonomous driving. It is extremely challenging for the pure imitation method due to the high degree of multimodality and uncertainty in the future. In fact, when driving in most traffic scenarios, vehicles should obey some traffic rules such as “vehicles follow the lane and do not collide with each other”. Inspired by this, this paper proposes a goal-aware prediction (GAP) framework to predict the multimodal trajectories, where goals are chosen in the lanes with hierarchical interactive representation and a multi-task loss. Based on the graph-based vectorized input, a novel hierarchical interactive representation module is first designed to obtain the fine-grained goal features, which progressively models interactions between goal-to-goal, goal-to-lane, and lane-to-agent, corresponding to the individual, local and global levels, respectively. Then, an auxiliary collision loss is developed to take into account learning from demonstration and injecting common sense of collision avoidance, and is served as a part of the multi-task loss to guide the generation of multimodal plausible trajectories. In the end, the proposed method is verified on the Baidu In-house Cut-in dataset, which includes more than 370K interactive scenarios collected in the real road testing. The comparative results demonstrate the superior performance of our proposed GAP model than the mainstream prediction methods.

Temporal Action Localization (TAL) aims to predict both action category and temporal boundary of action instances in untrimmed videos, i.e., start and end time. Existing works usually adopt fully-supervised solutions, however, one of the practical bottlenecks in these solutions is the large amount of labeled training data required. To reduce expensive human label cost, this paper focuses on a rarely investigated yet practical task named semi-supervised TAL and proposes an effective active learning method, named AL-STAL. We leverage four steps for actively selecting video samples with high informativeness and training the localization model, named Train, Query, Annotate, Append. Two scoring functions that consider the uncertainty of localization model are equipped in AL-STAL, thus facilitating the video sample ranking and selection. One takes entropy of predicted label distribution as measure of uncertainty, named Temporal Proposal Entropy (TPE). And the other introduces a new metric based on mutual information between adjacent action proposals, named Temporal Context Inconsistency (TCI). To validate the effectiveness of proposed method, we conduct extensive experiments on three benchmark datasets THUMOS'14, ActivityNet 1.3 and ActivityNet 1.2. Experiment results show that AL-STAL outperforms the existing competitors and achieves satisfying performance compared with fully-supervised learning.

Self-supervised skeleton-based action recognition enjoys a rapid growth along with the development of contrastive learning. The existing methods rely on imposing invariance to augmentations of 3D skeleton within a single data stream, which merely leverages the easy positive pairs and limits the ability to explore the complicated movement patterns. In this paper, we advocate that the defect of single-stream contrast and the lack of necessary feature transformation are responsible for easy positives, and therefore propose a Cross-Stream Contrastive Learning framework for skeleton-based action Representation learning (CSCLR). Specifically, the proposed CSCLR not only utilizes intra-stream contrast pairs, but introduces inter-stream contrast pairs as hard samples to formulate a better representation learning. Besides, to further exploit the potential of positive pairs and increase the robustness of self-supervised representation learning, we propose a Positive Feature Transformation (PFT) strategy which adopts feature-level manipulation to increase the variance of positive pairs. To validate the effectiveness of our method, we conduct extensive experiments on three benchmark datasets NTU-RGB + D 60, NTU-RGB + D 120 and PKU-MMD. Experimental results show that our proposed CSCLR exceeds the state-of-the-art methods on a diverse range of evaluation protocols.

To enable temporal action localization, the computer needs to recognize the locations and classes of action instances in a video. The main challenge to temporal action detection is that the videos are often long and untrimmed, consisting of varying action content. Existing temporal action detection frameworks exhibit a gap between the training and testing phases, which is detrimental to model performance. Specifically, all positive samples are trained identically in the training phase. By contrast, in the testing phase, the positive samples with the best classification and localization scores are selected, while all others are suppressed. To mitigate this issue, we build an auxiliary branch to unify the training and testing procedures. In the construction of the auxiliary branch, we design a dynamic weighting strategy based on curriculum learning, where the weights of training samples are a combination of their classification and localization scores. Motivated by the speculation of curriculum learning, we emphasize the importance of classification and localization scores in different training stages. The classification score accounts for a higher proportion of the combined score in the early stages of the training process. As the epoch increases, the localization score gradually increases in proportion as well. The experimental results demonstrate that our methodology of curriculum-based learning enhances the performance of current action localization techniques. On THUMOS14, our technique outperforms the existing state-of-the-art technique (57.6% vs 55.5%). And the performance on ActivityNet v1.3 (mAP@Avg) reaches 35.4%.

Trajectory prediction is one of the main challenges to autonomous vehicles. Except for the predicted vehicle historical trajectory information, viable solutions for this task must also consider the static geometric context, such as lane centerlines, and the dynamic interaction information among traffic participants, such as the influence of surrounding vehicles or pedestrians on the predicted vehicle. Recently, mainstream methods are mostly based on recurrent neural networks, which have achieved state-of-the-art (SOTA) performance on distance-based prediction metrics. In order to consider the influence of static geometric context and dynamic interaction information better for the vehicle trajectory prediction, we propose a two-stage prediction framework and a combined VAE-GAT model. Stage I estimates the future destination point of the predicted vehicle using a destination point estimation module based on VAE. In stage II, it produces multiple trajectories using a multiple trajectories prediction module based on GAT which consists of an LSTM-based encoder-decoder and a graph attention network. Finally, one specific trajectory is selected from multiple trajectories conditioned on the estimated destination point. The experimental results and performance comparison based on the Argoverse motion forecasting dataset are given to prove the effectiveness of the proposed method.

The target of 2D human pose estimation is to locate the keypoints of body parts from 2D images. State-of-the-art methods for pose estimation usually construct pixel-wise heatmaps from keypoints as labels for learning neural networks, which are usually initialized randomly or using classification models on large dataset, such as ImageNet, for their backbones. According to statistical data, there are strong positional priors for human keypoints, which are highly dependent on their relationship between image patches. To learn positional priors for pretraining pose estimators, we propose Heatmap-Style Jigsaw Puzzles (HSJP) problem as self-supervised pretext task, whose target is to predict the location of each patch from an image composed of shuffled patches. During pretraining, we only use person images in MS-COCO, rather than introducing extra large dataset like ImageNet. A heatmap-style label for patch location is designed and our learning process is in a non-contrastive way. The weights learned by HSJP pretext task are utilised as backbones of 2D human pose estimators, which are then finetuned on MS-COCO human keypoints dataset. With two popular and strong 2D human pose estimators, HRNet and SimpleBaseline, we evaluate mAP score on both MS-COCO validation and test-dev datasets. Our experiments show that downstream pose estimators with our self-supervised pretraining obtain much better performance than those trained from scratch, and are comparable to those using ImageNet classification models as their initial backbones.

Video moment retrieval aims to search the moment most relevant to a given language query. However, most existing methods in this community often require temporal boundary annotations which are expensive and time-consuming to label. Hence weakly supervised methods have been put forward recently by only using coarse video-level label. Despite effectiveness, these methods usually process moment candidates independently, while ignoring a critical issue that the natural temporal dependencies between candidates in different temporal scales. To cope with this issue, we propose a Multiscale 2D Representation Learning method for weakly supervised video moment retrieval. Specifically, we first construct a two-dimensional map for each temporal scale to capture the temporal dependencies between candidates. Two dimensions in this map indicate the start and end time points of these candidates. Then, we select top-K candidates from each scale-varied map with a learnable convolutional neural network. With a newly designed Moments Evaluation Module, we obtain the alignment scores of the selected candidates. At last, the similarity between captions and language query is served as supervision for further training the candidates' selector. Experiments on two benchmark datasets Charades-STA and ActivityNet Captions demonstrate that our approach achieves superior performance to state-of-the-art results.

Domain adaptation in person re-identification (re-ID) has always been challenging, especially for the lack of supervision information on the target domain. Existing methods generally introduced extra supervision by adversarial learning techniques, then added all the augmented data in the training process to optimize the re-ID model. However, the direct utilization of all the generated data not only increases additional computational cost but also ignores the potential correlation between the origin and generated data. In this article, we propose a novel dual-alignment learning framework (DAL) with camera-aware image generation to efficiently and effectively tackle this issue. Specifically, we propose a camera transfer matching module to generate additional training images with different camera styles, and construct the matching pairs with each containing a origin image and one corresponding camera transferred image. To strengthen the correlation of images for each matching pair, we align the pseudo-labels via clustering algorithm to reduce the pseudo-labels distribution discrepancy between the origin and generated images. Besides, to avoid model degeneration affected by some inaccurate pseudo-labels on unlabelled data, we maximize the mutual information to align the image feature representations of matching pair. The DAL allows us to decrease the camera variance and enhance the discrimination ability of re-ID model. Extensive experiments on three large-scale benchmarks demonstrate the superiority of DAL over state-of-the-art methods.

Existing successful person re-identification (Re-ID) models often employ the part-level representation to extract the fine-grained information, but commonly use the loss that is particularly designed for global features, ignoring the relationship between semantic parts. In this paper, we present a novel triplet loss that emphasizes the salient parts and also takes the consideration of alignment. This loss is based on the crossing-bing matching metric that also known as Wasserstein Distance. It measures how much effort it would take to move the embeddings of local features to align two distributions, such that it is able to find an optimal transport matrix to re-weight the distance of different local parts. The distributions in support of local parts is produced via a new attention mechanism, which is calculated by the inner product between high-level global feature and local features, representing the importance of different semantic parts w.r.t. identification. We show that the obtained optimal transport matrix can not only distinguish the relevant and misleading parts, and hence assign different weights to them, but also rectify the original distance according to the learned distributions, resulting in an elegant solution for the mis-alignment issue. Besides, the proposed method is easily implemented in most Re-ID learning system with end-to-end training style, and can obviously improve their performance. Extensive experiments and comparisons with recent Re-ID methods manifest the competitive performance of our method.