一种对称残差CNN的图像超分辨率重建方法

doi:10.19665/j.issn1001-2400.2019.05.003

摘要/Abstract

摘要：

基于卷积神经网络的图像超分辨率重建方法具有很高的重建性能。但该类方法存在网络参数多、训练难度大,梯度消失和网络退化等问题。针对这些问题,提出一种基于对称残差卷积神经网络的图像超分辨率重建方法。通过将对称融入到残差块中,采用对称连接实现局部特征融合,提取尽可能多的有价值特征;残差块外采用跳跃连接实现全局特征融合,以提高图像的重建质量。该方法使用峰值信噪比和结构相似度作为评价指标,在Set5、Set14和BSD100标准数据集上进行2倍、3倍和4倍因子重建后的结果大部分优于比较方法,平均峰值信噪比和结构相似度值较比较方法均有提高。实验结果表明,该方法重建的图像纹理更清晰,细节更丰富,具有较好的主观视觉效果。

关键词: 超分辨率重建, 卷积神经网络, 深度学习, 对称残差网络

Abstract:

The image super-resolution reconstruction methods based on the convolutional neural network have high reconstruction performance, but they have many network parameters and are difficult to train. Also, they are prone to problems such as gradient disappearance and network degradation. To solve these problems, a super-resolution image reconstruction method based on the symmetric residual convolution neural network is proposed. This method integrates symmetry into residual blocks. It realizes local feature fusion by adopting the symmetric connection and extracts as many valuable features as possible. To improve the quality of image reconstruction, the global feature fusion is realized by skip connection. In this method, the peak signal to noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation indexes. The results by the proposed reconstruction method on Set5, Set14, and BSD100 are superior to those by most of other methods in comparison. The average PSNR and SSIM values are improved compared with those methods. Experimental results show that the image reconstructed by this method has clearer textures, richer details and a better subjective visual effect.

Key words: super-resolution reconstruction, convolution neural network, deep learning, symmetric residual network

中图分类号:

TP391

刘树东,王晓敏,张艳. 一种对称残差CNN的图像超分辨率重建方法[J]. 西安电子科技大学学报, 2019, 46(5): 15-23.

LIU Shudong,WANG Xiaomin,ZHANG Yan. Symmetric residual convolution neural networks for the image super-resolution reconstruction[J]. Journal of Xidian University, 2019, 46(5): 15-23.

图/表 10

图1

图2

图3

表1

图4

图5

图6

图7

表2

在Set5、Set14和BSD100数据集上不同超分辨率重建方法的平均PSNR/SSIM"

数据集	采样因子	Bicubic (PSNR /SSIM)	A+ (PSNR /SSIM)	SRCNN (PSNR /SSIM)	VDSR (PSNR /SSIM)	DRCN (PSNR /SSIM)	LapSRN (PSNR /SSIM)	SymRCN (PSNR /SSIM)
Set5	$×$ 2	33.66/0.929 9	36.54/0.954 4	36.66/0.954 2	37.53/0.958 7	37.63/0.958 8	37.52/0.959 1	37.83/0.9598
	$×$ 3	30.39/0.868 2	32.58/0.908 8	32.75/0.909 0	33.66/0.921 3	33.82/0.922 6	33.81/0.922 0	34.00/0.924 6
	$×$ 4	28.42/0.810 4	30.28/0.860 3	30.48/0.862 8	31.35/0.883 8	31.53/0.885 4	31.54/0.885 2	31.53/0.885 5
Set14	$×$ 2	30.24/0.868 8	32.28/0.905 6	32.42/0.906 3	33.03/0.912 4	33.04/0.911 8	32.99/0.912 4	33.22/0.913 7
	$×$ 3	27.55/0.774 2	29.13/0.818 8	29.28/0.820 9	29.77/0.831 4	29.76/0.831 1	29.79/0.832 5	29.88/0.833 8
	$×$ 4	26.00/0.702 7	27.32/0.749 1	27.49/0.750 3	28.01/0.767 4	28.02/0.767 0	28.09/0.770 0	28.05/0.768 8
BSD100	$×$ 2	29.56/0.843 1	31.21/0.886 3	31.36/0.887 9	31.90/0.896 0	31.85/0.894 2	31.80/0.895 2	32.01/0.897 4
	$×$ 3	27.21/0.738 5	28.29/0.783 5	28.41/0.786 3	28.82/0.797 6	28.80/0.796 3	28.82/0.798 0	28.87/0.799 6
	$×$ 4	25.96/0.667 5	26.82/0.708 7	26.90/0.710 1	27.29/0.725 1	27.23/0.723 3	27.32/0.727 5	27.24/0.724 7
平均值		28.78/0.800 4	30.50/0.841 7	30.64/0.843 1	31.26/0.854 9	31.30/0.854 5	31.30/0.855 8	31.40/0.867 5

表2

表3

在Set5、Set14和BSD100数据集上不同超分辨率重建方法的平均测试时间s"

数据集	采样因子	A+	SRCNN	VDSR	DRCN	LapSRN	SymRCN
Set5	$×$ 2	0.623	3.389	0.054	0.735	0.032	0.078
	$×$ 3	0.400	3.405	0.062	0.748	0.049	0.040
	$×$ 4	0.281	3.496	0.054	0.735	0.040	0.027
Set14	$×$ 2	1.370	5.134	0.113	1.579	0.035	0.142
	$×$ 3	0.790	5.095	0.122	1.569	0.061	0.068
	$×$ 4	0.630	5.162	0.112	1.526	0.040	0.044
BSD100	$×$ 2	0.921	3.992	0.071	0.983	0.018	0.092
	$×$ 3	0.563	4.140	0.071	0.996	0.037	0.044
	$×$ 4	0.411	3.998	0.071	0.984	0.023	0.028
平均值		0.665	4.201	0.081	1.095	0.037	0.062

表3

参考文献 36

[1]	胡长胜, 詹曙, 吴从中 . 基于深度特征学习的图像超分辨率重建[J]. 自动化学报, 2017,43(5):814-821.
	HU Changsheng, ZHAN Shu, WU Congzhong . Image Super-resolution Based on Deep Learning Features[J]. Acta Automatica Sinica, 2017,43(5):814-821.
[2]	李欣, 韦宏卫, 张洪群 . 结合深度学习的单幅遥感图像超分辨率重建[J]. 中国图象图形学报, 2018,23(2):209-218.
	LI Xin, WEI Hongwei, ZHANG Hongqun . Super-resolution Reconstruction of Single Remote Sensing Image Combined with Deep Learning[J]. Journal of Image and Graphics, 2018,23(2):209-218.
[3]	REN H Y, EI-KHAMY M, LEE J . CT-SRCNN: Cascade Trained and Trimmed Deep Convolutional Neural Networks for Image Super Resolution[C]// Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision. Piscataway: IEEE, 2018: 1423-1431.
[4]	ZOU W W W, YUEN P C . Very Low Resolution Face Recognition Problem[J]. IEEE Transactions on Image Processing, 2012,21(1):327-340.
[5]	KARRAS T, AILA T, LAINE S , et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation[CA/OL]. [2018-12-20]..
[6]	HARRIS J L . Diffraction and Resolving Power[J]. Journal of the Optical Society of America, 1964,54(7):931-936.
[7]	ZHU S, ZENG B, LIU G , et al. Image Interpolation Based on Non-local Geometric Similarities[C]// Proceedings of the 2015 IEEE International Conference on Multimedia and Expo. Washington: IEEE Computer Society, 2015: 7177417.
[8]	ZHANG K, GAO X, TAO D , et al. Single Image Super-resolution with Non-local Means and Steering Kernel Regression[J]. IEEE Transactions on Image Processing, 2012,21(11):4544-4556.
[9]	DONG C, LOY C C, HE K , et al. Image Super-resolution Using Deep Convolutional Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016,38(2):295-307.
[10]	王世平, 毕笃彦, 刘坤 , 等. 一种多映射卷积神经网络的超分辨率重建算法[J]. 西安电子科技大学学报, 2018,45(4):155-160.
	WANG Shiping, BI Duyan, LIU Kun , et al. Multi-mapping Convolution Neural Network for the Image Super-resolution Algorithm[J]. Journal of Xidian University, 2018,45(4):155-160.
[11]	YANG J C, WRIGHT J, HUANG T S , et al. Image Super-resolution as Sparse Representation of Raw Image Patches[C]// Proceedings of the 2008 26th IEEE Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2008: 4587647.
[12]	YANG J C, WRIGHT J, HUANG T S , et al. Image Super-resolution via Sparse Representation[J]. IEEE Transactions on Image Processing, 2010,19(11):2861-2873.
[13]	TIMOFTE R, DE V, GOOL L V . Anchored Neighborhood Regression for Fast Example-based Super-resolution[C]// Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2013: 1920-1927.
[14]	TIMOFTE R, DE S V, VAN G L . A+: Adjusted Anchored Neighborhood Regression for Fast Super-resolution[C]// Lecture Notes in Computer Science: 9006. Heidelberg: Springer Verlag, 2015: 111-126.
[15]	ZHANG H, PATEL V M . Densely Connected Pyramid Dehazing Network[C]// Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2018: 3194-3203.
[16]	ZHANG H, PATEL V M . Density-aware Single Image De-raining Using a Multi-stream Dense Network[C]// Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2018: 695-704.
[17]	张金刚, 方圆, 袁豪 , 等. 一种识别表情序列的卷积神经网络[J]. 西安电子科技大学学报, 2018,45(1):150-155.
	ZHANG Jingang, FANG Yuan, YUAN Hao , et al. Multiple Convolutional Neural Networks for Facial Expression Sequence Recognition[J]. Journal of Xidian University, 2018,45(1):150-155.
[18]	许强, 李伟, 占荣辉 , 等. 一种改进的卷积神经网络SAR目标识别算法[J]. 西安电子科技大学学报, 2018,45(5):177-183.
	XU Qiang, LI Wei, ZHAN Ronghui , et al. Improved Algorithm for SAR Target Recognition Based on the Convolutional Neural Network[J]. Journal of Xidian University, 2018,45(5):177-183.
[19]	KIM J, LEE J K, LEE K M . Accurate Image Super-resolution Using Very Deep Convolutional Networks[C]// Proceedings of the 2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2016: 1646-1654.
[20]	KIM J, LEE J K, LEE K M . Deeply-recursive Convolutional Network for Image Super-resolution[C]// Proceedings of the 2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2016: 1637-1645.
[21]	LAI W S, HUANG J B, AHUJA N , et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-resolution[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 5835-5843.
[22]	TAI Y, YANG J, LIU X . Image Super-resolution via Deep Recursive Residual Network[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2790-2798.
[23]	MAO X J, SHEN C H, YANG Y B . Image Restoration Using Very Deep Convolutional Encoder-decoder Networks with Symmetric Skip Connections[C]// Advances in Neural Information Processing Systems 29 - Proceedings of the 2016 Conference. Vancouver: Neural Information Processing Systems Foundation, 2016: 2810-2818.
[24]	HE K, ZHANG X, REN S , et al. Deep Residual Learning for Image Recognition[C]// Proceedings of the 2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2016: 770-778.
[25]	陈建廷, 向阳 . 深度神经网络训练中梯度不稳定现象研究综述[J]. 软件学报, 2018,29(7):2071-2091.
	CHEN Jianting, XIANG Yang . Survey of Unstable Gradients in Deep Neural Network Training[J]. Journal of Software, 2018,29(7):2071-2091.
[26]	TAI Y, YANG J, LIU X . Image Super-resolution via Deep Recursive Residual Network[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2790-2798.
[27]	LIM B, SON S, KIM H , et al. Enhanced Deep Residual Networks for Single Image Super-resolution[C]// Proceedings of the 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Washington: IEEE Computer Society, 2017: 1132-1140.
[28]	AHN N, KANG B, SOHN K A. Fast, Accurate , Lightweight Super-resolution with Cascading Residual Network[C]// Lecture Notes in Computer Science: 11214. Heidelberg: Springer Verlag, 2018: 256-272.
[29]	ZHANG Y, LI K P, LI K , et al. Image Super-resolution Using Very Deep Residual Channel Attention Networks[C]// Lecture Notes in Computer Science: 11211. Heidelberg: Springer Verlag, 2018: 294-310.
[30]	LI J C, FANG F M, MEI K F , et al. Multi-scale Residual Network for Image Super-resolution[C]// Lecture Notes in Computer Science: 11212. Heidelberg: Springer Verlag, 2018: 527-542.
[31]	孙跃文, 李立涛, 丛鹏 , 等. 基于深度学习的辐射图像超分辨率重建方法[J]. 原子能科学技术, 2017,51(5):890-895.
	SUN Yuewen, LI Litao, CONG Peng , et al. Super-resolution Method for Radiation Image Based on Deep Learning[J]. Atomic Energy Science and Technology, 2017,51(5):890-895.
[32]	MARTIN D, FOWLKES C, TAL D , et al. A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics[C]// Proceedings of the 2001 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2001: 416-423.
[33]	JIA Y, SHELHAMER E, DONAHUE J , et al. Caffe: Convolutional Architecture for Fast Feature Embedding[C]// Proceedings of the 2014 ACM Conference on Multimedia. New York: ACM, 2014: 675-678.
[34]	HUANG J B, SINGH A, AHUJA N . Single Image Super-resolution from Transformed Self-exemplars[C]// Proceedings of the 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2015: 5197-5206.
[35]	李浪宇, 苏卓, 石晓红 , 等. 图像超分辨率重建中的细节互补卷积模型[J]. 中国图象图形学报, 2018,23(4):572-582.
	LI Langyu, SU Zhuo, SHI Xiaohong , et al. Mutual-detail Convolution Model for Image Super-resolution Reconstruction[J]. Journal of Image and Graphics, 2018,23(4):572-582.
[36]	HE K, ZHANG X, REN S , et al. Delving Deep into Rectifiers: Surpassing Human-level Performance on Imagenet Classification[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2015: 1026-1034.