Rate-distortion optimization (RDO) is widely used in video coding, which plays a critical role in enhancing the coding efficiency substantially. Currently, the RDO process is performed in a way that coding efficiency of each coding unit is maximized independently without considering the dependency among coding units. As we know, in the current hybrid video coding structure, spatial/temporal prediction techniques are extensively used which introduces strong dependency among coding units. In this paper, we investigate RDO with inter-frame dependency, where the impact of coding performance of the current coding unit on that of the following frames is considered. Accordingly, a rate-distortion optimization scheme taking the inter-frame dependency into account is proposed by adapting the Lagrangian multiplier. Experimental results show that the proposed scheme can achieve about 3.22% and 3.19% BD-rate saving in average over the state-of-the-art HEVC reference software HM15.0 in the low-delay P and low-delay B coding structures, respectively, with no extra encoding time. The proposed scheme can obtain a significantly higher coding gain than the multiple QP (3) optimization technique which would greatly increase the encoding time by a factor of about 6. Coupled with multiple QP optimization, the proposed scheme can further achieve about 5.96% and 5.57% BD-rate saving in average over the HEVC and about 4.03% and 4.07% over the HEVC with multiple QP optimization, under the specified common test conditions for low-delay P and low-delay B coding structures, respectively.