The Collaborative Filtering (CF) algorithms work fairly well in personalized recommendation except in sparse data environment. To deal with the sparsity problem, researchers either take into account auxiliary information extracted from additional data resources, or set the missing ratings with default values, e.g., video popularity. Nevertheless, the former often costs high and incurs difficulty in knowledge transference whereas the latter degrades the accuracy and coverage of recommendation results. To our best knowledge, few literatures take advantage of users' preference on video popularity to tackle this problem. In this paper, we intend to enhance the performance of recommendation algorithm via the inference of the users' popularity preferences (PPs), especially in a sparse data environment. We propose a scheme to aggregate users' PPs and a Collaborative Filtering based algorithm to make the inference of PP feasible and effective from a small number of watching records. We modify a k-Nearest-Neighbor recommendation algorithm and a Matrix Factorization algorithm via introducing the inferred PP. Experiments on a large-scale commercial dataset show that the modified algorithm outperforms the original CF algorithms on both the recommendation accuracy and coverage. The significance of improvement is significant especially with the data sparsity.