ROD-based hybrid TH precoding and combining for mmWave large-scale MIMO systems

Hybrid precoding is one of key techniques for millimeter wave (mmWave) large-scale multiple-input multiple-output (MIMO) systems. This paper considers a nonlinear hybrid precoding architecture which consists of a nonlinear unit, a reductive digital precoder and a constant modulus radio frequency (RF) precoder, and presents a novel hybrid Tomlinson-Harashima (TH) precoding and combining algorithm. Firstly, due to the intractability of the sum rates maximization problem for such a nonlinear hybrid precoding architecture, a tractable three-stage optimization problem is constructed through the lower bound of the sum rates, which allows the digital precoding matrix, the RF precoding matrix and the RF combining matrix to be optimized sequentially and independently. Then, in order to solve the three-stage optimization problem effectively, a novel row orthogonal decomposition (ROD) is defined. Based on the ROD, it is interesting that the necessary and sufficient condition of the optimal digital precoding matrix can be obtained, and a near-optimal RF precoding matrix can be derived. Finally, the optimization of the RF combining matrix is reformulated as a unimodular quadratic programming and solved by a generalized power method. Theoretical analyses and simulations indicate that the proposed ROD-based hybrid TH precoding and combining algorithm can offer a higher sum rates and a lower bit error rate with a comparable complexity in comparison to the previous works.