The H.265 video coding standard promotes the realization of 4K/8K ultrahigh definition (UHD) video applications.To further improve the coding efficiency, H.265 allows motion estimation (ME) performing on multiple reference frame (MRF). Although the MRF can enhance the performance and allow the encoder to search a better reference frame from several previous pictures, the computational complexity of the MRF-based ME (MRF-ME) module dramatically increases. Toimprove the coding performance of H.265 according to the high spatiotemporal correlation existing in the MRF, we firstly proposeneighboring-block-based reference frame decision algorithm(NRFDA) and priority-based reference frame selection algorithm(PRFSA) to reduce the computational complexity of ME-MRF module. The NRFDA utilizes the selected reference frames information among encoded neighboring blocksand the variance of the current block to predict the best reference frame. Therefore, PRFSA define the priority for each reference frame so that ME can perform on the reference frames along the descending order of priority according to the rate distortion cost (RDcost)rank.Finally, we integrate NRFDA and PRFSA into a fast reference frame decision algorithm (FRFDA) to further speed up the ME-MRF module. Simulationresults show that the proposed FRFDA can achieve an average time improving ratio (TIR) about 68.23% when compared to H.265 (HM16.7) under MRF=4. It is clear that the proposed algorithm can efficiently increase the encoding speed of H.265 with insignificant loss of image quality