Memory-Augmented Auto-Regressive Network for Frame Recurrent Inter Prediction
Fig.1 Architecture of the Memory-Augmented Auto-Regressive Network (MAAR-Net). The network uses an auto-encoder as the feature extraction module based on previous frames. The auto-regressive coefficients are generated by short-term temporal redundancy and long-term temporal dynamics. The quality attention is injected for better prediction. A ConvLSTM-based memory update module is proposed to guide the coefficient generation to make full use of the information of all previous frames during the whole coding process.
Abstract
Inter prediction is quite important for the modern codecs to remove temporal redundancy. In this paper, we make endeavors in generating artificial reference frames with previous reconstructed frames for inter prediction, to offer a better choice when the traditional block-wise motion estimation fails to find a good reference block. Long-term temporal dynamics are tracked during the whole coding process to generate more accurate and realistic artificial reference frames. Specifically, we propose a Memory-Augmented Auto-Regressive Network (MAAR-Net) for frame prediction in video coding. MAAR-Net regresses the current frame with two nearest frames via an auto-regressive (AR) model to better capture the main spatial and temporal structures. The AR regression coefficients are generated based on adjacent frame information as well as the long-term motion dynamics accumulated and propagated by a convolutional Long Short-Term Memory (LSTM). To generate the target frame with higher quality, a quality attention mechanism is introduced for the temporal regularization between different reconstructed frames. With the well-designed network, our method surpasses HEVC on average 4:0% BD-rate saving and up to 10:6% BD-rate saving for the luma component under the low-delay configuration.
Resourses
Citation
@article{hyz2020iscas, title={Memory-Augmented Auto-Regressive Network for Frame Recurrent Inter Prediction}, author={Hu, Yuzhang and Xia, Sifeng and Yang, Wenhan and Liu, Jiaying}, booktitle={IEEE International Symposium on Circuits and Systems (ISCAS)}, year={2020}, publisher={IEEE} }
Feature Compression Results
Table 1. Results of rate reduction of the proposed method.
Fig 2. Visual comparison of frames: (a) The original frame; (b) The reconstructed frame of the HEVC anchor; (c) The reconstructed frame of our method with the artificial reference frame.