Adverse weather, such as rain or snow, can cause difficulties in the processing of video streams. Because the appearance of raindrops can affect the performance of human tracking and reduce the efficiency of video compression, the detection and removal of rain is a challenging problem in outdoor surveillance systems. In this paper, we propose a new algorithm for rain detection and removal based on both spatial and wavelet domain features. Our system involves fewer frames during detection and removal, and is robust to moving objects in the rain. Experimental results demonstrate that the proposed algorithm outperforms existing approaches in terms of subjective and objective quality.