In order to improve the coding efficiency of HEVC in real-time video, this study proposes an improved HEVC efficiency algorithm based on fast coding unit partition and adaptive mode decision. By using the separation depth information of the adjacent coding units on the top and left, it is determined whether the can coding unit could be separated or terminated early. When the condition is not satisfied, the coding unit can end its quad-tree separation by using the Rate-distortion cost threshold determined by training frame online learning. At the same time, the minimum Rate-distortion cost model and the second smallest Rate-distortion cost model were adopted to reduce the number of candidate sets and determine candidate set CS5. Then, CS5 is optimized to reduce the number of candidate Rate-distortion optimization. Finally, the Rate-Distortion optimization calculation was carried out to select the optimal mode to reduce the number of prediction modes. Experimental results show that compared with HM standard test results, the proposed algorithm can effectively reduce the video coding time of 45.93% under the premise of ensuring the coding quality, with only 0.597 8% of BDBR rising.