We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6

Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models.

Most existing semi-parameter-sharing federated learning (FL) frameworks utilize generative models to achieve partial parameter sharing with the server, which effectively enhances the data privacy of each client. However, these generative models often suffer from model utility degradation due to poor representation robustness. Meanwhile, representation inconsistency between local and global models exacerbates the client drift problem under non-IID scenarios. Furthermore, existing semi-parameter-sharing FL frameworks overlook representation leakage risks associated with generator sharing, while failing to balance privacy and utility. To alleviate these challenges, we propose FedPDM, a semi-parameter-sharing FL framework built upon a privacy-preserving diffusion model (PDM). Specifically, our proposed PDM enables model alignment with features from the privacy extractor without requiring direct exposure of this extractor, effectively mitigating utility degradation caused by poor representation robustness. Moreover, a feature-level penalty term is introduced into the optimization objective of PDM to avoid representation leakage. We further design a two-stage aggregation strategy that addresses representation inconsistency through initialization correction with a Gaussian constraint for knowledge distillation. Finally, we provide the first theoretical convergence analysis for semi-parameter-sharing FL, demonstrating that our framework converges at a rate of O(1/f). Extensive experiments on four datasets show that FedPDM achieves average accuracy improvements of 1.78% to 5.56% compared with various state-of-the-art baselines.

Agricultural news recommendation systems play a vital role in mitigating information overload within the rapidly expanding media landscape. However, existing news recommendation methods typically model user interests as a single or fixed number of feature representations, which presents significant challenges when applied to the agricultural domain. First, users often exhibit diverse interests spanning multiple subfields within agriculture, making it insufficient to capture user preferences with a single unified representation. Second, current methods that rely on fixed-size multi-interest modeling lack the flexibility to adapt to the varying breadth of interests exhibited by agricultural users across different career roles. Third, user interests in the agricultural domain demonstrate different levels of stability, manifesting as the coexistence of long-term fundamental preferences and short-term contextual interests. To address these problems, this paper proposes ELStar, an elastic long- and short-term multi-interest user modeling network for agricultural news recommendation. ELStar initializes users’ long-term interests from a global interest pool and employs a diversity-aware selection-and-aggregation strategy to elastically generate long-term user representations. Additionally, a self-additive attention mechanism is introduced to capture users’ short-term unique interest based on their recent click behaviors. Finally, a target-aware matching network is proposed to generate personalized news recommendations by integrating both long- and short-term multi-interest representations. Extensive experiments on two real-world agricultural news recommendation datasets validate the superiority of our proposed ELStar compared with various state-of-the-art news recommendation baselines.

Temporal Knowledge Graph (TKG) reasoning plays a pivotal role in predicting emerging facts based on historical data. However, existing TKG reasoning methods typically aggregate all historical facts within a given time window indiscriminately, which often introduces outdated or irrelevant information. This information redundancy can significantly hinder the reasoning performance, especially as TKGs continue to grow in scale and complexity. Effectively filtering out irrelevant facts is thus essential for improving inference accuracy and efficiency. To address this critical challenge, we focus on how to refine the TKGs and propose a Temporal knowledge graph reasoning model via Multi-granularity Knowledge Refinement (T-MKR). Specifically, we propose a multi-granularity knowledge refinement approach to prune historical TKGs, which selectively removes irrelevant edges and unnecessary nodes at both the edge and node levels. The resulting refined subgraphs are used for representation learning. To effectively combine information from both refinements, we introduce a subgraph gating integration module. Additionally, we leverage contrastive learning for subgraph alignment to emphasize the relationships between the two refined subgraphs. Extensive experiments on six commonly used datasets demonstrate the superiority of T-MKR compared with many state-of-the-art baselines.

Recent advances in large language models (LLMs) have expanded the context window to beyond 128K tokens, enabling long-document understanding and multi-source reasoning. A key challenge, however, lies in choosing between retrieval-augmented generation (RAG) and long-context (LC) strategies: RAG is efficient but constrained by retrieval quality, while LC supports global reasoning at higher cost and with position sensitivity. Existing methods such as Self-Route adopt failure-driven fallback from RAG to LC, but remain passive, inefficient, and hard to interpret. We propose Pre-Route, a proactive routing framework that performs structured reasoning before answering. Using lightweight metadata (e.g., document type, length, initial snippet), Pre-Route enables task analysis, coverage estimation, and information-need prediction, producing explainable and cost-efficient routing decisions. Our study shows three key findings: (i) LLMs possess latent routing ability that can be reliably elicited with guidelines, allowing single-sample performance to approach that of multi-sample (Best-of-N) results; (ii) linear probes reveal that structured prompts sharpen the separability of the "optimal routing dimension" in representation space; and (iii) distillation transfers this reasoning structure to smaller models for lightweight deployment. Experiments on LaRA (in-domain) and LongBench-v2 (OOD) confirm that Pre-Route outperforms Always-RAG, Always-LC, and Self-Route baselines, achieving superior overall cost-effectiveness.

On-policy distillation (OPD) aligns a student with a teacher on trajectories sampled from the student itself, reducing the train-test state mismatch of offline distillation. The same feedback loop can nevertheless be unstable: each update changes both the policy and the states on which the next update is computed. We introduce WDL-OPD, a mixture-constrained co-training method with two trainable policies. An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL. Both policies receive gradient. We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD^2 and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express. In recorded Qwen3 experiments at 1.7B and 4B scale, WDL-OPD produces the strongest student checkpoint in each of four scale-domain settings. It raises MATH500 accuracy from 0.630 to 0.685 at 4B and from 0.521 to 0.585 at 1.7B. In code generation, seven single-policy OPD configurations exhibit entropy growth or trajectory degradation, while co-training reaches independently re-evaluated development scores of 0.637 and 0.375. Because several comparisons differ in curriculum or initialization, these results support a stabilization hypothesis rather than a universal causal claim. We provide the exact training algorithm, failure evidence, and the controlled comparison matrix needed to test that hypothesis.

Compositional e-commerce queries express multiple requirements that must hold jointly, yet existing rerankers collapse these constraints into aggregate relevance and often promote topical near misses over feasible products. In this paper, we introduce REAlign, a novel requirement-evidence-aligned reranking framework that explicitly connects typed query requirements with visible evidence. REAlign distinguishes satisfied, violated, and unsupported conditions, constructs requirement-targeted contrasts that expose failure modes, and optimizes duplicate-free partial rankings through Requirement-Aware Group-Relative Policy Optimization. Its list utility preserves relevance while incorporating requirement satisfaction, evidence support, material violations, and output validity. Experiments on two fixed-pool e-commerce benchmarks show consistent improvements over strong supervised and policy-optimization baselines under matched training budgets, with fewer violations among top-ranked candidates and larger gains at shallow ranks. Controlled ablations confirm the complementary value of requirement modeling, evidence grounding, and decomposed optimization.

Existing scientific document retrieval (SDR) methods primarily rely on document-centric representations learned from inter-document relationships for document-document (doc-doc) retrieval. However, the rise of LLMs and RAG has shifted SDR toward question-driven retrieval, where documents are retrieved in response to natural-language questions (q-doc). This change has led to systematic mismatches between document-centric models and question-driven retrieval, including (1) input granularity (long documents vs. short questions), (2) semantic focus (scientific discourse structure vs. specific question intent), and (3) training signals (citation-based similarity vs. question-oriented relevance). To this end, we propose UniFAR, a Unified Facet-Aware Retrieval framework to jointly support doc-doc and q-doc SDR within a single architecture. UniFAR reconciles granularity differences through adaptive multi-granularity aggregation, aligns document structure with question intent via learnable facet anchors, and unifies doc-doc and q-doc supervision through joint training. Experimental results show that UniFAR consistently outperforms prior methods across multiple retrieval tasks and base models, confirming its effectiveness and generality.

Generative Recommendation (GR) formulates recommendation as autoregressive generation over discrete semantic identifiers (IDs). Although recent multimodal GR methods improve semantic ID construction with visual and textual information, they typically require item-level paired observations, restricting tokenization to the intersection of modality availability. Moreover, incorporating unpaired observations is nontrivial because small representation shifts may cross quantization boundaries and produce incompatible identifier sequences. To address this challenge, we propose Unpaired Modality-Agnostic Generative Recommendation (UnpairGR), which learns a unified semantic-ID space from paired, image-only, and text-only observations. UnpairGR confines modality-specific processing to lightweight input projections while sharing the subsequent Transformer and residual codebooks across all observation conditions. Paired observations establish a reliability-guided cross-modal consensus, whereas unimodal observations directly refine the same representations and codes. The learned tokenizer is then fixed to provide stationary targets for a single autoregressive recommender, without feature imputation, modality-specific codebooks, or fallback mappings. Extensive experiments on three benchmark datasets demonstrate that UnpairGR consistently improves recommendation performance under both fully observed and incomplete-observation settings.

Generative retrieval offers a new paradigm for e-commerce search by mapping user queries directly to product Semantic Identifiers (SIDs). However, e-commerce queries are often short, noisy, attribute-heavy, and associated with multiple category-consistent products, creating a substantial representation gap between natural-language shopping intent and artificially constructed item SIDs. Explicit Chain-of-Thought (CoT) reasoning can help bridge this gap, but its extra generation cost is difficult to reconcile with the low-latency requirements of online e-commerce systems. To address this challenge, we propose CaLIR (Category-guided Latent Intent Reasoning), a category-guided latent intent reasoning framework for e-commerce generative retrieval. Rather than generating explicit textual rationales, CaLIR learns continuous latent intent states before SID decoding and uses product category hierarchies as a natural scaffold for coarse-to-fine intent reasoning. Specifically, we introduce hierarchical semantic reasoning to align latent states with category-level shopping intent, and query-wise reasoning enhancement to model diverse intent paths under multi-positive queries. CaLIR further combines a query-specific dynamic prefix trie, assembled from pre-indexed category-level tries, with reasoning-aware constrained decoding. Experiments on multilingual e-commerce search datasets show that CaLIR achieves a better balance between retrieval effectiveness and inference efficiency than existing methods, while also demonstrating transferability and robustness across induced hierarchies and different generative backbones.

Anonymizing sensitive information in user text is essential for privacy, yet existing methods often apply uniform treatment across attributes, which can conflict with communicative intent and obscure necessary information. This is particularly problematic when personal attributes are integral to expressive or pragmatic goals. The central challenge lies in determining which attributes to protect, and to what extent, while preserving semantic and pragmatic functions. We propose IntentAnony, a utility-preserving anonymization approach that performs intent-conditioned exposure control. IntentAnony models pragmatic intent and constructs privacy inference evidence chains to capture how distributed cues support attribute inference. Conditioned on intent, it assigns each attribute an exposure budget and selectively suppresses non-intent inference pathways while preserving intent-relevant content, semantic structure, affective nuance, and interactional function. We evaluate IntentAnony using privacy inference success rates, text utility metrics, and human evaluation. The results show an approximately 30

Data allocation plays a critical role in federated large language model (LLM) and small language models (SLMs) reasoning collaboration. Nevertheless, existing data allocation methods fail to address an under-explored challenge in collaboration: bidirectional model learnability gap, where client-side SLMs cannot identify high-reward samples matching their learnability constraints for effective knowledge transfer from LLMs, while LLMs struggle to select samples contributing novel knowledge beyond their existing data. Furthermore, these collaboration frameworks face another key challenge: domain-agnostic reasoning transfer, where existing reasoning transfer methods fail to flexibly adapt to the local domain data, preventing SLMs from effectively acquiring step-by-step reasoning abilities within from general LLM. To address these challenges, we propose LaDa, a federated reasoning distillation framework with model learnability-aware data allocation. It introduces a model learnability-aware data filter that adaptively allocates high-reward samples based on the learnability gap between each SLM and LLM pair, effectively facilitating bidirectional knowledge transfer. We further design a domain adaptive reasoning distillation method that aligns joint probabilities of reasoning paths on filtered high-reward samples through contrastive distillation learning between SLM and LLM, enabling SLM to capture underlying reasoning patterns under local data distribution. LaDa operates as a plug-in module for existing collaboration frameworks, adapting knowledge transfer based on model learnability gaps.

Large Language Models (LLMs) have demonstrated powerful reasoning capabilities through Chain-of-Thought (CoT) in various tasks, yet the inefficiency of token-by-token generation hinders real-world deployment in latency-sensitive recommender systems. Latent reasoning has emerged as an effective paradigm in LLMs, performing multi-step inference in a continuous hidden-state space to achieve stronger reasoning at lower cost. However, this paradigm remains underexplored in mainstream generative recommendation. Achieving this reveals three key challenges: (1) the gap between prior-less Semantic ID (SID) symbols and continuous latent reasoning, as SIDs lack pre-trained semantics, hindering joint optimization; (2) representation drift due to a lack of reasoning chain supervision; and (3) the suboptimality of applying a globally fixed reasoning depth. To address these, we propose LASAR (Latent Adaptive Semantic Aligned Reasoning), an SFT-then-RL framework. First, we bridge this gap via two-stage training: Stage 1 grounds SID semantics before Stage 2 introduces latent reasoning, ensuring efficient convergence. Second, we mitigate representation drift through explicit CoT semantic alignment. Step-wise bidirectional KL divergence constrains the latent reasoning trajectory using hidden-state anchors extracted from CoT text, while a Policy Head predicts per-sample reasoning depth. Third, during the GRPO-based RL phase, terminal-only KL alignment accommodates variable-length reasoning, and REINFORCE optimizes the Policy Head to dynamically allocate steps. This nearly halves the average latent step count while simultaneously improving recommendation quality. Experiments on three real-world datasets show that LASAR achieves the best overall performance across the evaluated settings. It adds limited inference latency and is roughly 20x faster than generating explicit CoT text.

Recent advancements in large reasoning models (LRMs) have greatly improved their capabilities on complex reasoning tasks through Long Chains of Thought (CoTs). However, this approach often results in substantial redundancy, impairing computational efficiency and causing significant delays in real-time applications. Recent studies show that longer reasoning chains are frequently uncorrelated with correctness and can even be detrimental to accuracy. In a further in-depth analysis of this phenomenon, we surprisingly uncover and empirically verify that LRMs implicitly know the appropriate time to stop thinking, while this capability is obscured by current sampling paradigms. Motivated by this, we introduce SAGE (Self-Aware Guided Efficient Reasoning), a novel sampling paradigm that unleashes this efficient reasoning potential. Furthermore, integrating SAGE as mixed sampling into group-based reinforcement learning (SAGE-RL) enables SAGE-RL to effectively incorporate SAGE-discovered efficient reasoning patterns into standard pass@1 inference, markedly enhancing both the reasoning accuracy and efficiency of LRMs across multiple challenging mathematical benchmarks.

Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models. However, existing RLVR methods utilize rollouts in an indiscriminate and short-horizon manner: responses of heterogeneous quality within each prompt are treated uniformly, and historical rollouts are discarded after a single use. This leads to noisy supervision, poor sample efficiency, and suboptimal policy updates. We address these issues by formulating rollout scheduling in RLVR as a contextual bandit problem and proposing a unified neural scheduling framework that adaptively selects high-value rollouts throughout training. Each rollout is treated as an arm whose reward is defined by the induced performance gain between consecutive optimization steps. The resulting scheduler supports both noise-aware intra-group selection and adaptive global reuse of historical rollouts within a single principled framework. We provide theoretical justification by deriving sublinear regret bounds and showing that enlarging the rollout buffer improves the achievable performance upper bound. Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.

The impressive capabilities of Large Language Models (LLMs) enable them to perform recommendation through prompting, facilitating a novel paradigm of universal recommender systems. However, in practice, LLMs often exhibit some inherent stereotypes that should be avoided in recommendations. This necessitates aligning LLMs to meet the fairness requirements of recommendation systems. But the typical alignment methods often require substantial human labors for external supervision, which is further exacerbated when addressing fairness across multiple sensitive attributes. To address this limitation, we propose a novel Mixture of Experts (MoE) contrastive learning approach to enhance fairness of LLM-based recommenders without additional external supervision. Specifically, we first leverage contrastive learning, along with counterfactual data augmentation, to improve fairness by reducing the difference between the hidden states of contrastive sample pairs. Besides, to better handle scenarios involving multiple sensitive attributes, we propose a LoRA-based MoE framework to disentangle attribute relationships for efficient fine-tuning. And to avoid the distortion from the varying sample training difficulty due to the differing involved attributes, we further incorporate a tailored Curriculum Learning strategy, which progressively trains on samples of increasing difficulty based on the sensitive attributes involved. Finally, extensive experiments on two public datasets demonstrate the effectiveness of our proposed method.

Exploration is not an optional behavior in natural intelligence; it is an evolutionary principle underlying the emergence and adaptation of intelligence. Curiosity, play, and deliberate probing emerge as evolved responses to uncertainty, enabling organisms to construct internal models, expand competence, and preserve adaptability in changing environments. We argue that this evolutionary logic is equally indispensable for artificial general intelligence (AGI): exploration is not a heuristic appended to learning, but the mechanism through which generality becomes possible. We develop a unified view of Epistemic Exploration for agentic systems: the capacity of an agent to actively acquire information that reduces uncertainty about the world, seek experiences at the boundary of its current capabilities and convert them into durable capability improvement, and preserve epistemic reachability as the readiness and ability to adapt when the world changes. This view yields three criteria: Information Gain, Value Improvement, and Epistemic Reachability. We then introduce an exploration-centered five-level trajectory toward AGI, in which each level is characterised by a distinct exploration capacity and exploration serves as the transition mechanism among levels: • Responder: minimal or no explicit epistemic exploration; the system mainly relies on learned input-output mappings and local token-level variation. • Reasoner: reasoning-space exploration enables hypothesis search, deliberate reasoning trajectories, branching, backtracking, and self-verification beyond reactive response generation. • Agent: interaction-space exploration extends internal deliberation into embodied perception, tool use, memory, and closed-loop action under partial observability. • Prospector: imagination-space exploration uses learned world models to simulate counterfactual futures, reduce the cost and risk of real interaction, and support long-horizon policy improvement. • Ecosystem: coordination-space exploration enables collectives of heterogeneous agents to co-evolve roles, shared representations, communications, and collaborative strategies beyond the limits of any single agent. Finally, we conclude with evaluation principles and open challenges for building exploration-centric agents that continually reduce uncertainty, improve their own capabilities, and maintain readiness to adapt beyond predefined tasks.

Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.