Fast CU size decision and mode decision algorithm for intra prediction in HEVC

In the intra prediction process, High Efficiency Video Coding (HEVC) provides a quadtree-based coding unit (CU) block partitioning structure and up to 35 kinds of prediction modes to improve the coding performance. These technologies improve the coding efficiency significantly while the coding complexity is simultaneously increased rapidly as well. In this paper, a novel fast CU size decision and mode decision algorithm is proposed for the intra prediction of HEVC. The overall algorithm consists of two processes, the fast CU size decision and fast mode decision. In the fast CU size decision process, we adopt an adaptive discretization total variation (DTV) threshold-based CU size determination algorithm to skip some specific depth levels. In the fast mode decision process, an orientation gradient-based mode decision is proposed to reduce the candidate modes involved in the rough mode decision (RMD) and the rate distortion optimization (RDO) process. The experimental results on the HEVC reference software HM demonstrate that the proposed algorithm can significantly reduce the coding time with negligible performance loss.