Multi-Agent Deep Reinforcement Learning for Beam Codebook Design in RIS-aided Systems

Reconfigurable intelligent surfaces (RISs) play a vital role in future wireless systems with the capability of enhancing propagation environments by intelligently reflecting the signals toward the target receivers. However, optimal tuning of the phase shifters at the RIS is challenging due to the passive nature of reflective elements and the high complexity of acquiring channel state information (CSI). Furthermore, the joint active beamforming and RIS reflection beam design is a tedious task due to the high computational complexity and the dynamic nature of the wireless environment. Today’s cellular networks establish data transmission by relying on pre-defined generic beamforming codebooks, which are neither site-specific nor adaptive to the changes in the wireless environment. Moreover, identifying the best beam is typically performed using an exhaustive search approach that prohibits the use of large codebook sizes due to the resulting high beam training overhead. Depending merely on the binary received signal strength, this work develops a multi-agent deep reinforcement learning (MA-DRL) framework that jointly designs the active and the passive reflection beam codebooks for the BS and the RIS, reflectively. To accelerate learning convergence and reduce the search space, the proposed model divides the RIS into multiple partitions and associates beam patterns to the surrounding environments with low computational complexity. Moreover, a hierarchical beam training solution is proposed to further reduce the beam training overhead of the single-beam training approach. Simulation results show that the proposed MA-DRL approach can provide a 97% beam training overhead reduction over the discrete Fourier transform (DFT) codebook.