The three main reasons why the new H.264 (MPEG-4 AVC) video coding standard has a significant performance better than the other standards are the adoption of variable block sizes, multiple reference frames, and the consideration of rate distortion optimization within the codec. However, these features incur a considerable increase in encoder complexity. As for the multiple reference frames motion estimation, the increased computation is in proportion to the number of searched reference frames. In this paper, a fast multi-frame selection method is proposed for H.264 video coding. The proposed scheme can efficiently determine the best reference frame from the allowed five reference frames. Simulation results show that the speed of the proposed method is over two times faster than that of the original scheme adopted in JVT reference software JM73 while keeping the similar video quality and bit-rate.