The complicated indoor environment such as obstacles causes the non-line of sight (NLOS) environment. In this paper, we propose a voting matrix based residual weighting (VM-Rwgh) algorithm to mitigate NLOS errors in indoor localization system. The voting matrix is employed to provide initial localization results. The residual weighting is used to improve the localization accuracy. The VM-Rwgh algorithm can overcome the effects of NLOS errors, even when more than half of the measurements contain NLOS errors. Simulation results show that the VM-Rwgh algorithm provides higher location accuracy with relatively lower computational complexity in comparison with other methods.