Adaptive Federated Deep Reinforcement Learning for Proactive Content Caching in Edge Computing

With the aggravation of data explosion and backhaul loads on 5 G edge network, it is difficult for traditional centralized cloud to meet the low latency requirements for content access. The federated learning (FL)-based proactive content caching (FPC) can alleviate the matter by placing content in local cache to achieve fast and repetitive data access while protecting the users' privacy. However, due to the non-independent and identically distributed (Non-IID) data across the clients and limited edge resources, it is unrealistic for FL to aggregate all participated devices in parallel for model update and adopt the fixed iteration frequency in local training process. To address this issue, we propose a distributed resources-efficient FPC policy to improve the content caching efficiency and reduce the resources consumption. Through theoretical analysis, we first formulate the FPC problem into a stacked autoencoders (SAE) model loss minimization problem while satisfying resources constraint. We then propose an adaptive FPC (AFPC) algorithm combined deep reinforcement learning (DRL) consisting of two mechanisms of client selection and local iterations number decision. Next, we show that when training data are Non-IID, aggregating the model parameters of all participated devices may be not an optimal strategy to improve the FL-based content caching efficiency, and it is more meaningful to adopt adaptive local iteration frequency when resources are limited. Finally, experimental results in three real datasets demonstrate that AFPC can effectively improve cache efficiency up to 38.4% and 6.84%, and save resources up to 47.4% and 35.6%, respectively, compared with traditional multi-armed bandit (MAB)-based and FL-based algorithms.