一种多尺度三维卷积的视频超分辨率方法

doi:10.19665/j.issn1001-2400.2021.05.002

摘要/Abstract

摘要：

视频超分辨率技术可由低分辨率视频获得高分辨率视频,有效提升视频的显示效果。与单幅图像超分辨率不同,如何利用相邻视频帧之间的信息在视频超分辨率中则显得十分重要。为改善视频超分辨率重建的性能,充分利用视频帧的时间-空间相关性,提出一种基于多尺度三维卷积的视频超分辨率模型。该模型输入连续的多帧视频图像,输出中间帧的超分辨率重建结果,包括多尺度特征提取、特征融合以及高分辨率重建3个模块。首先,使用多尺度的三维卷积进行初步特征提取;然后,使用三维卷积残差结构进行特征融合,并将特征图进行通道分离,在融合不同尺度的特征时,有效地减少了网络的参数量;最后,使用多个残差密集连接块和亚像素卷积进行高分辨率重建,并结合全局残差连接得到重建的高分辨率视频图像。Vid4数据集上3倍和4倍超分辨率放大的实验结果表明,与其他已有方法相比,该方法可有效提升峰值信噪比和结构相似性性能,取得较好的视觉效果。

关键词: 视频超分辨率, 三维卷积, 残差网络, 时间-空间相关性

Abstract:

Video super-resolution aims to restore high-resolution videos from low-resolution videos,which can effectively improve the display effect of videos.What is different from single image super-resolution is that how to exploit the information between contiguous video frames is important for video super-resolution.In order to improve the performance of video super-resolution and make full use of the spatio-temporal information on video frames,a video super-resolution model based on multi-scale 3D convolution is proposed,which takes continuous video frames as the input and outputs the reconstruction super-resolution result of the intermediate frame.This model consists of three modules:multi-scale feature extraction,feature fusion and high-resolution reconstruction.First,multi-scale 3D convolution is used for preliminary feature extraction.Then,3D convolution residual structure is adopted in feature fusion,and the feature maps are split,which can not only fuse the features of different scales,but also effectively reduce the number of network parameters.Finally,residual dense blocks and sub-pixel convolution are used for high-resolution reconstruction,and the reconstructed video frame is obtained by combining with the global residual connection.Experimental results of 3× and 4× super-resolution in Vid4 dataset show that compared with other methods,the proposed method can enhance the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) performance effectively with a better visual effect.

Key words: video super-resolution, 3D convolution, residual network, spatio-temporal correlation

中图分类号:

TP391.4

詹克羽,孙岳,李颖. 一种多尺度三维卷积的视频超分辨率方法[J]. 西安电子科技大学学报, 2021, 48(5): 8-14.

ZHAN Keyu,SUN Yue,LI Ying. Video super-resolution based on multi-scale 3D convolution[J]. Journal of Xidian University, 2021, 48(5): 8-14.

图/表 8

图1

图2

图3

图4

表1

表2

图5

图6

参考文献 17

[1]	HARRIS J L. Diffraction and Resolving Power[J]. Journal of the Optical Society of America, 1964, 54(7):931-936. doi: 10.1364/JOSA.54.000931
[2]	MEIJERING E, UNSER M. A Note on Cubic Convolution Interpolation[J]. IEEE Transactions on Image Processing, 2003, 12(4):477-479. doi: 10.1109/TIP.2003.811493
[3]	CHANG K, ZHANG X, DING P L K, et al. Data-Adaptive Low-Rank Modeling and External Gradient Prior for Single Image Super-Resolution[J]. Signal Processing, 2019, 161:36-49. doi: 10.1016/j.sigpro.2019.03.011
[4]	YANG J, WRIGHT J, HUANG T, et al. Image Super-Resolution as Sparse Representation of Raw Image Patches[C]// Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2008:1-8.
[5]	DONG C, LOY CC, HE K, et al. Image Super-Resolution Using Deep Convolutional Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2):295-307. doi: 10.1109/TPAMI.2015.2439281
[6]	KIM J, LEE J K, LEE K M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:1646-1654.
[7]	SHI W, CABALLERO J, HUSZAR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Piscatavay:IEEE, 2016:1874-1883.
[8]	WANG X, GU Y, GAO X, et al. Dual Residual Attention Module Network for Single Image Super Resolution[J]. Neurocomputing, 2019, 364:269-279. doi: 10.1016/j.neucom.2019.06.078
[9]	刘树东, 王晓敏, 张艳. 一种对称残差CNN的图像超分辨率重建方法[J]. 西安电子科技大学学报, 2019, 46(5):15-23.
	LIU Shudong, WANG Xiaomin, ZHANG Yan. Symmetric Residual Convolution Neural Networks for the Image Super-Resolution Reconstruction[J]. Journal of Xidian University, 2019, 46(5):15-23.
[10]	KAPPELER A, YOO S, DAI Q, et al. Video Super-Resolution with Convolutional Neural Networks[J]. IEEE Transactions on Computational Imaging, 2016, 2(2):109-122. doi: 10.1109/TCI.2016.2532323
[11]	CABALLERO J, LEDIG C, AITKEN A, et al. Real-Time Video Super-Resolution with Spatiotemporal Networks and Motion Compensation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:4778-4787.
[12]	WANG L, GUO Y, LIN Z, et al. Learning for Video Super-Resolution Through HR Optical Flow Estimation[C]// Proceedings of the Asian Conference on Computer Vision.Heidelberg:Springer, 2018:514-529.
[13]	TIAN Y, ZHANG Y, FU Y, et al. TDAN:Temporally-Deformable Alignment Network for Video Super-Resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2020:3360-3369.
[14]	KIM S Y, LIM J, NA T, et al. Video Super-Resolution Based on 3d-Cnns with Consideration of Sscene Change[C]// Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP).Piscataway:IEEE, 2019:2831-2835.
[15]	ZHANG Y, TIAN Y, KONG Y, et al. Residual Dense Network for Image Super-Resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:2472-2481.
[16]	XUE T, CHEN B, WU J, et al. Video Enhancement with Task-Oriented Flow[J]. International Journal of Computer Vision, 2019, 127(8):1106-1125. doi: 10.1007/s11263-018-01144-2
[17]	LIU C, SUN D. A Bayesian Approach to Adaptive Video Super Resolution[C]// Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2011:209-216.

模型	峰值信噪比(PSNR)	结构相似性(SSIM)	参数量
模型1	26.48	0.792 6	2.14×10⁶
模型2	26.50	0.793 4	1.75×10⁶

数据集	放大倍数	Bicubic	VSRnet	VESPCN	3DSRnet	SOF-VSR	笔者模型
Calendar	3	21.73/0.672 0	22.85/0.753 6	23.52/0.787 9	24.17/0.821 6	24.51/0.836 8	25.14/0.855 2
Calendar	4	20.54/0.570 3	21.13/0.638 1	21.98/0.692 8	22.37/0.727 3	22.63/0.746 3	23.26/0.776 4
City	3	26.31/0.712 1	26.96/0.768 2	27.54/0.802 0	28.26/0.836 8	28.40/0.845 4	28.83/0.866 2
City	4	25.04/0.597 3	25.78/0.656 5	26.16/0.694 4	26.77/0.738 2	26.90/0.749 8	27.31/0.775 4
Walk	3	25.98/0.793 3	30.21/0.904 3	30.97/0.914 4	30.72/0.912 0	31.72/0.925 0	32.33/0.933 6
Walk	4	25.98/0.793 3	27.55/0.833 8	28.25/0.856 1	28.28/0.853 4	29.04/0.871 6	29.63/0.8847
Foliage	3	25.20/0.697 8	26.47/0.778 8	26.89/0.792 3	27.49/0.826 6	27.78/0.836 0	28.07/0.847 1
Foliage	4	23.56/0.565 8	24.35/0.645 2	24.91/0.672 5	25.26/0.708 6	25.48/0.717 8	25.80/0.737 1
平均值	3	24.80/0.718 8	26.62/0.801 2	27.23/0.824 1	27.66/0.849 2	28.10/0.860 8	28.59/0.875 5
平均值	4	23.78/0.631 6	24.70/0.693 4	25.32/0.728 9	25.67/0.756 8	26.01/0.771 3	26.50/0.793 4
参数量			0.27×10⁶	0.88×10⁶	0.47×10⁶	1.64×10⁶	1.75×10⁶