Hierarchical Multi-Domain Collaboration Based on DRL: an Intelligent and Generalized Architecture for Multi-Satellite Collaborative Communications

Multi-satellite collaborative communications (MSCC) is emerging as a cornerstone of 6G satellite networks to ensure ubiquitous coverage and high quality of service (QoS). However, the non-stationary channel conditions, limited link resource constraints, highly dynamic topology, and rapidly increasing problem scale pose significant challenges to traditional optimization methods. Deep reinforcement learning (DRL) offers a promising solution, yet current algorithms are tailored to specific problems and lack a holistic perspective. To bridge this gap, this paper presents a comprehensive survey and proposes an intelligent, generalized architecture for multi-satellite collaboration. First, we synthesize a set of algorithmic design principles along the dimensions of problem type, observability, and system scale, providing a systematic guide for algorithm selection. Subsequently, guided by these principles, we establish a cloud-edge-terminal hierarchical federated DRL framework. This architecture leverages geostationary earth orbit (GEO) satellites as intermediate aggregators and global context providers, bridging the gap between the ground cloud’s massive computing power and low earth orbit (LEO) edge’s real-time inference needs. This design effectively mitigates partial observability and feeder link congestion, facilitating the resolution of communication optimization issues spanning from the physical layer to the network layer. We propose a hierarchical multi-satellite collaborative framework where LEO agents, guided by GEO-aggregated global context, employ the multi-agent proximal policy optimization (MAPPO) based algorithm to dynamically optimize semantic modality selection and access control. Simulation results validate that this approach effectively manages multi-level image and text semantic transmission, enhancing system adaptability. Finally, we outline future directions, including communication-computing synergy and embodied intelligence, to guide the evolution of autonomous satellite networks.