(JBE Vol. 23, No. 4, July 2018) (Special Paper) 23 4, 2018 7 (JBE Vol. 23, No. 4, July 2018) https://doi.org/10.5909/jbe.2018.23.4.484 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) a), a) Uniform Motion Deblurring using Shock Filter and Convolutional Neural Network Minso Jeong a) and Jechang Jeong a) Cho. (Blur) (Shock filter) (Convolutional Neural Network: CNN) (Uniform motion)... Abstract The uniform motion blur removing algorithm of Cho et al. has the problem that the edge region of the image cannot be restored clearly. We propose the effective algorithm to overcome this problem by using shock filter that reconstructs a blurred step signal into a sharp edge, and convolutional neural network (CNN) that learns by extracting features from the image. Then uniform motion blur kernel is estimated from the latent sharp image to remove blur in the image. The proposed algorithm improved the disadvantages of the conventional algorithm by reconstructing the latent sharp image using shock filter and CNN. Through the experimental results, it was confirmed that the proposed algorithm shows excellent reconstruction performance in objective and subjective image quality than the conventional algorithm. Keyword : Deblurring, Convolutional Neural Network (CNN), Shock filter, Uniform Motion blur, Blind deconvolution a) (Department of Electronics and Computer Engineering, Hanyang University) Corresponding Author : (Jechang Jeong) E-mail: jjeong@hanyang.ac.kr Tel: +82-2-2220-4372 ORCID: http://orcid.org/0000-0002-3759-3116 IPIU 2018..[2014-0-00670, ICT SW ] This work was supported by the ICT R&D program of MSIP/IITP.[ 2014-0-00670, Software Platform for ICT Equipment] Manuscript received May 4, 2018; Revised July 9, 2018; Accepted July 9, 2018. Copyright 2018 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND (http://creativecommons.org/licenses/by-nc-nd/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.
1: (Minso Jeong et al.: Uniform Motion Deblurring using Shock Filter and Convolutional Neural Network). (Blur) (Degradation). (Gaussian blur) [1], (Defocus blur) [1], (Motion blur) [2].,,, (Noise) (Detail). ( (1)) (Convolution)..,.. (Nonuniform) [4] (Uniform) [5]. (Pixel), [6]., (Distribution) (Standard deviation). (Out of focus) (Aberration)..,. (Out-focusing),... (Deblurring). (Non-blind deconvolution) [3] (Blind deconvolution) [2]. (Blur kernel).,,. 2,. 1., [6].. [2].,,,,.....
(JBE Vol. 23, No. 4, July 2018) Fergus [2] Shan [7] Fergus. Cho Lee [5]. Xu Jia [8]. Krishnan [9]. Cho [10]. [10]. (Shock filter) (Convolutional Neural Network: CNN).. 2, 3. 4. 5... Cho. 1. Fig 1. Flow chart of conventional algorithm. 3.. 2.. 1 Cho [10]... (Bilateral filter) [11] (Ringing artifact) 2. Fig 2. Flow chart of proposed algorithm
1: (Minso Jeong et al.: Uniform Motion Deblurring using Shock Filter and Convolutional Neural Network). EPLL. 1. (Strong edge). 3 (Step signal). [12].. 4, 8. ReLU (Rectified Linear Unit) [13]. (Kernel) 128, 64 1. Loss ( (5)). optimizer adam-optimizer (Adaptive moment estimation optimizer) (Learning rate) 0.00001. (Laplacian) (Gradient). L2-norm. 3. :, Fig 3. Shock filtering: Blurred signal Shock filtering of blurred signal 2.,,. (Batch), L2-norm. Pascal VOC 2012 [14]. 40 8. 5 Pascal VOC 2012 1 8.. 40,000 64 64 (Patch). 6. 4. Fig 4. Neural network architecture of proposed algorithm
(JBE Vol. 23, No. 4, July 2018) 5. Pascal VOC 2012 :, 8 Fig 5. Example of the Pascal VOC 2012 datasets: Groundtruth image Blurred image by 8 kinds kernels. (6) [15]. 6. :, Fig 6. Dataset used for learning: Input data label data 3. arg min, (Derivative filters),. L2-norm. 51 51. 1 [15]. 4.
1: (Minso Jeong et al.: Uniform Motion Deblurring using Shock Filter and Convolutional Neural Network). EPLL (Expected Patch Log Likelihood) [16] ( 2). EPLL. x ( (7)). x log P x P (Overlapping patches) (Matrix), log P x (Likelihood) [16]. Ratio) SSIM (Structural SIMilarity) [18]. PSNR. (8). log, (Mean Square error). PSNR db.. 3.30 GHz CPU (Intel Core i5-2500), 11 GHz GPU (GTX-1080 Ti) Window 10 64 bit PC. (Tensorflow) Matlab R2017a. 30,000. Sun [17]. Lighthouse, Building, Field, Bedroom, Cockpit, Desert, Temple Warehouse 8. 8. Ground Truth. 1. PSNR (Peak Signal to Noise,,,,. 1 PSNR. 8., Bedroom 1. PSNR (db) Table 1. Comparison of PSNR (db) for proposed and conventional algorithm Sequence Width Height Input image Conventional algorithm Proposed algorithm Lighthouse 924 668 31.02 31.77 32.77 Building 924 616 22.68 24.08 25.46 Field 924 668 30.89 31.34 31.62 Bedroom 924 583 28.38 29.13 32.58 Cockpit 924 583 23.30 23.34 25.10 Desert 924 583 28.33 28.60 30.06 Temple 924 632 26.67 26.67 29.03 Warehouse 924 581 22.28 23.67 25.13
(JBE Vol. 23, No. 4, July 2018) 3.45 db ( 1).. SSIM,, (10).,,,. (11). 0.05 ( 2).. 2. 7. 7 Lighthouse 7 Lighthouse.. and (Dynamic range), 0.01 0.03. 2 SSIM. 8., Warehouse 2. SSIM Table 2. Comparison of SSIM for proposed and conventional algorithm Sequence Width Height Input image Conventional algorithm Proposed algorithm Lighthouse 924 668 0.83 0.91 0.93 Building 924 616 0.68 0.77 0.80 Field 924 668 0.74 0.79 0.82 Bedroom 924 583 0.82 0.89 0.93 Cockpit 924 583 0.62 0.69 0.73 Desert 924 583 0.77 0.84 0.86 Temple 924 632 0.71 0.75 0.79 Warehouse 924 581 0.58 0.69 0.74 7. : Lighthouse., Fig 7. Comparison of locally extension image: Lighthouse. Result image without shock filter Result image using shock filter 8-11 Lighthouse, Building, Field Bedroom 4 (Region of Interest: ROI). 12-15 Cockpit, Desert, Temple Warehouse 4. 8-15, 8-15, 8-15(c), 8-15(d).
1: (Minso Jeong et al.: Uniform Motion Deblurring using Shock Filter and Convolutional Neural Network).. (c) (d) 8. : Lighthouse.,, (c) Cho et al. [10], (d) Fig 8. Comparison of locally extension image: Lighthouse. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm (c) (d) 9. : Building.,, (c) Cho et al. [10], (d) Fig 9. Comparison of locally extension image: Building. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm (c) (d) 10. : Field.,, (c) Cho et al. [10], (d) Fig 10. Comparison of locally extension image: Field. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm (c) (d) 11. : Bedroom.,, (c) Cho et al. [10], (d) Fig 11. Comparison of locally extension image: Bedroom. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm
9 4 2 방송공학회논문지 제23권 제4호, 2018년 7월 (JBE Vol. 23, No. 4, July 2018) (c) (d) 그림 12. 주관적 화질 비교: Cockpit. 원본 영상, 블러된 영상, (c) Cho et al. [10] 방식, (d) 제안하는 알고리듬 Fig 12. Subjective comparison: Cockpit. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm (c) (d) 그림 13. 주관적 화질 비교: Desert. 원본 영상, 블러된 영상, (c) Cho et al. [10] 방식, (d) 제안하는 알고리듬 Fig 13. Subjective comparison: Desert. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm (c) (d) 그림 14. 주관적 화질 비교: Temple. 원본 영상, 블러된 영상, (c) Cho et al. [10] 방식, (d) 제안하는 알고리듬 Fig 14. Subjective comparison: Temple. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm (c) (d) 그림 15. 주관적 화질 비교: Warehouse. 원본 영상, 블러된 영상, (c) Cho et al. [10] 방식, (d) 제안하는 알고리듬 Fig 15. Subjective comparison: Warehouse. Groundtruth image blurred image (c) Cho et al. [10] (d) Proposed algorithm Ⅴ. 결 론 높은 주파수 성분이 많은 영상에서 제안된 알고 리듬이 기존의 알고리듬보다 수치가 높게 측정되어 우수한 블러 제거 성능을 나타내었다. 주관적 평가에서는 제안된 선과 같은 본 논문에서는 쇼크 필터와 합성곱 신경망을 이용하여 균일 모션 블러 현상을 제거하는 알고리듬을 제안하였다. 알고리듬은 8종류의 실험 영상에서 기존의 알고리듬보다 제안된 알고리듬은 PSNR과 SSIM 수치를 비교한 객관적 우수한 외곽선 추출 성능을 이용함으로써 우수한 블러 제 평가와 주관적 평가를 통해 기존의 알고리듬보다 우수한 거 성능을 나타냄을 확인하였다. 하지만 제안된 알고리듬 블러 제거 성능을 나타내 다. 객관적 평가에 따 면 외곽 은 원본 영상과 비교하였을 때 영상 내 외곽선 성분을 었 르 완벽
1: (Minso Jeong et al.: Uniform Motion Deblurring using Shock Filter and Convolutional Neural Network).. (References) [1] R. L. Lagendijk and J. Biemond, Basic Methods For Image Restoration And Identification, Academic Press, London, United Kingdom, pp.125-139, 2000. [2] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, Removing Camera Shake from a Single Photograph, ACM Transactions on Graphics (TOG), Vol.25, No.3, pp.787-794, July 2006. [3] S. A. Bigdeli and M. Zwicker, Image Restoration using Autoencoding Priors, arxiv preprint arxiv:1703.09964 [cs.cv], March 2017. [4] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, Nonuniform deblurring for shaken images, Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, United States of America, pp.491-498, 2010. [5] S. Cho and S. Lee, Fast Motion Deblurring, ACM Transactions on Graphics (TOG), Vol.28, No.5, pp.145-152, December 2009. [6] S. Cho, H. Cho, Y.-W. Tai, and S. Lee, Registration based non-uniform motion deblurring, Computer Graphics Forum, Vol.31, No.7, pp.2183-2192, September 2012. [7] Q. Shan, J. Jia, and A. Agarwala, High-quality motion deblurring from a single image, ACM Transactions on Graphics (TOG), Vol.27, No.3, pp.73:173:10, August 2008. [8] L. Xu and J. Jia, Two-phase kernel estimation for robust motion deblurring, Proceeding of the European Conference on Computer Vision (ECCV), Crete, Greece, pp.157-170, 2010. [9] D. Krishnan, T. Tay, and R. Fergus, Blind deconvolution using a normalized sparsity measure, Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, United States of America, pp.233-240, 2011. [10] T. Cho, S. Paris, B. K. P. Horn, and W. T. Freeman, Blur Kernel Estimation using the Radon Transform, Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, United States of America, pp.241-248, 2011. [11] C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, Proceeding of the IEEE International Conference on Computer Vision (ICCV), Bombay, India, pp.839-846, 1998. [12] S. Osher and L. I. Rudin, Feature-oriented image enhancement using shock filters, Society for Industrial and Applied Mathematics (SIAM) Journal on Numerical Analysis, Vol.27, No.4, pp.910-940, August 1990. [13] A. Krizhevsky, I. Sutskever, and G. Hinton, Imagenet classification with deep convolutional neural networks, Proceeding of the International Conference on Neural Information Processing Systems (NIPS), Nevada, United States of America, Vol.1, pp.1097-1105, 2012. [14] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, The pascal visual object classes challenge: A retrospective, International Journal of Computer Vision (IJCV), Vol.111, No.1, pp.98-136, January 2015. [15] A. Chakrabarti, A Neural Approach to Blind Motion Deblurring, Proceeding of the European Conference on Computer Vision (ECCV), Amsterdam, Netherlands, pp.221-235, 2016. [16] D. Zoran and Y. Weiss, From learning models of natural image patches to whole image restoration, Proceeding of the IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, pp.479-486, 2011. [17] L. Sun, S. Cho, J. Wang, and J. Hays, Edge-based blur kernel estimation using patch priors, Proceeding of the IEEE Conference on Computational Photography (ICCP), Cambridge, United States of America, pp.1-8, 2013. [18] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, Vol.13, No.4, pp.600-612, April 2004. - 2017 2 : - 2017 3 ~ : - ORCID : https://orcid.org/0000-0001-6934-3288 - :,
(JBE Vol. 23, No. 4, July 2018) - 1980 2 : - 1982 2 : KAIST - 1990 : - 1980 ~ 1986 : KBS ( ) - 1990 ~ 1991 : ( ) - 1995 ~ : ( ) - 1990 12 : - 1998 11 : - 2007 : IEEE Chester Sall Award - 2008 : ETRI Journal Paper Award - 2011 5 : 46 - ORCID : http://orcid.org/0000-0002-3759-3116 - :,, 3DTV