Recent deep-learning-based video compression models have demonstrated superior performance over traditional codecs. However, few studies have focused on deep learning rate control. In this paper, end-to-end rate control is proposed for deep contextual video compression (DCVC). With the designed two-branch residual-based network, the optimal bit rate ratio is predicted according to the feature correlation of the adjacent frames. Then, the bit rate can be reasonably allocated for every frame by satisfying the temporal feature. To minimize the rate distortion (RD) cost, the optimal λ of the current frame can be obtained from a two-branch regression-based network using the temporal encoded information. The experimental results show that the achievable BD-rate (PSNR) and BD-rate (SSIM) of the proposed algorithm are -0.84% and -0.35%, respectively, with 2.25% rate control accuracy.