(Special Paper) 20 5, 2015 9 (JBE Vol. 20, No. 5, September 2015) http://dx.doi.org/10.5909/jbe.2015.20.5.676 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) 4 Light Field Dictionary Learning a), a) Dictionary Learning based Superresolution on 4D Light Field Images Seung-Jae Lee a) and In Kyu Park a) Light field 4 light field 2 (spatial domain) 2 (angular domain). 4 light field 2 CMOS. 4 light field, 4 light field (dictionary learning-based) (superresolution). 4 light field 4 (patch),. 2. light field Lytro. Abstract A 4D light field image is represented in traditional 2D spatial domain and additional 2D angular domain. The 4D light field has a resolution limitation both in spatial and angular domains since 4D signals are captured by 2D CMOS sensor with limited resolution. In this paper, we propose a dictionary learning-based superresolution algorithm in 4D light field domain to overcome the resolution limitation. The proposed algorithm performs dictionary learning using a large number of extracted 4D light field patches. Then, a high resolution light field image is reconstructed from a low resolution input using the learned dictionary. In this paper, we reconstruct a 4D light field image to have double resolution both in spatial and angular domains. Experimental result shows that the proposed method outperforms the traditional method for the test images captured by a commercial light field camera, i.e. Lytro. Keyword : Superresolution, dictionary learning, light field, spatial domain, angular domain, Lytro a) (Department of Information and Communication Engineering, Inha University) Corresponding Author : (In Kyu Park) E-mail: pik@inha.ac.kr Tel: +82-32-860-9190 ORCID: http://orcid.org/0000-0003-4774-7841 2013 ( ) (NRF-2013R1A2A2A01069181). Manuscript received July 10, 2015; Revised September 22, 2015; Accepted September 22, 2015.
1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images). 4 light field 2,, 3. Light field 2. light field., 4 2 (trade-off). light field [1][2].,., 3.. 2. Farsiu [3], Freeman [4], Yang Yang [5]., light field. Yang Yang [5]. 2 light field. light field. 4,.,.. 2 4 light field,,. 3,. 4.. Light field 4 light field, 4. 4 light field., light field 4, light field,. 4 light field light field
.. 2. light field.,. 1. Light Field light field 1(a). 1(a).. Dansereau [6] (sub-aperture image). 4 1(b) 4 light field.,,,. 2. 4 Light field.. 2 light field light field. 4 light field 4. 2 4 4, (a) raw data (b) 4D light field image 1. light field (Lytro) Fig 1. Light field sub-aperture images generated from the raw data captured by commercial light field camera (Lytro)
1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images) 2. 4 Fig 2. Overview of 4D training patch reconstruction. 2.. 3. light field 4.. K- (K-means clustering). K-. 3. 3. Fig 3. Overview of the dictionary learning procedure
4 light field (4D light field image), light field (LR light field image) 4 (HR patch) (LR patch). K- K (cluster center). L ( ), H ( ).. (1) (regression coefficient) C ( ). C argmin H CL c (dictionary).,. 4. Light Field light field,. light field,. Freedman Fattal [7] Glasner [8],. Freedman Fattal [7], Glasner [8]. 4. light field Fig 4. Region division of an input light field image to match the image resolution of the dictionary and the input image
1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images)..,. 4.1 Light Field - light field (, ),. Light field 4 4 16.. 4.2 16. 4,., (2) C l h. h C l (2). 4.3 Light Field 16 (a) (b) (c) 5.. (a), (b) EPI bicubic, (c) Fig 5. Results of resolution enhancement at the boundary region. (a) Initial results of the proposed algorithm, (b) Result of bicubic interpolation on EPI, (c) Final result of the proposed method
light field., 5(a). 2,. 4.. 6 light field (plane) EPI (epipolar plane image) EPI bicubic EPI. EPI light field EPI EPI light field 5(b) 4 light field. light field 5(c). 4 light field.. Intel i7-3770k 3.5GHz CPU 16G RAM, light field Lytro. 4 light field... 6.. Light field EPI bicubic Fig 6. Postprocessing algorithm to handle the image boundary. Bicubic interploation is applied after converting the light field image to EPI image
1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images) 7. Lytro Fig 7. Training dataset captured by Lytro camera (a) Bicubic (b) 8. Fig 8. Qualitative comparison in the spatial domain
.1. Table 1. Quantitative comparison in the spatial domain Algorithm tiger monkey beer can eiffel rhinoceros PSNR (db) SSIM Bicubic Interpolation Proposed Algorithm Bicubic Interpolation Proposed Algorithm 30.02 29.52 27.63 30.23 28.62 30.16 30.94 29.67 31.05 30.93 0.83 0.88 0.87 0.83 0.86 0.87 0.90 0.90 0.86 0.87. 4 light field,,,,. 7 40 4 light field 200,000. K- K, K 512.,. 2 (a) (1,2) (b) (3,12) (c) (10,10) (d) (14,6) (e) (16,16) 10. Fig 10. Results of superresolution at various locations in the angular domain
1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images). EPI. 2. Lytro. Lytro ( ) ( )., Lytro ( ) ( ). 8 bicubic. 1 8 PSNR(peak signal to noise ratio) SSIM(structural similarity), bicubic PSNR 2dB SSIM 0.4., light field ( ) bicubic Freeman [4]. 9 bicubic Freeman [4],,. 10. ((1,2), (3,12), (10,10), (14,6), (16,16)).. 4 light field 4 light field.,, 2 4 light field. 4,,,. 4 light field. (References) [1] Lytro, https://www.lytro.com/ [2] Raytrix, https://www.raytrix.de/ [3] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super resolution," IEEE Trans. on Image Processing, vol. 13, no. 10, pp. 1327-1344, October 2004. [4] W. T. Freeman, T. R. Jones, and E. C. Pasztor, "Example-based superresolution," IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56-65, March 2002. [5] C.-Y. Yang and M.-H. Yang, "Fast direct super-resolution by simple functions," Proc. IEEE International Conference on Computer Vision, pp. 561-568, December 2013. [6] D. G. Dansereau, O. Pizarro, and S. B. Williams, "Decoding, calibration and rectification for lenselet-based plenoptic cameras," Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 1027-1034, June 2013. [7] G. Freedman and R. Fattal, "Image and video upscaling from local self-examples," ACM Trans.on Graphics, vol. 30, no. 12, pp. 1-12, April 2011. [8] D. Glasner, S. Bagon, and M. Irani, "Super-resolution from a single image," Proc. IEEE International Conference on Computational
Photography, pp. 349-356, September 2009. [9] T. E. Bishop and P. Favaro, "The light field camera: Extended depth of field, aliasing, and superresolution," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 34, no. 5, pp. 972-986, May 2012. [10] T. E. Bishop, S. Zanetti, and P. Favaro, "Light field superresolution," Proc. IEEE International Conference on Computational Photography, pp.1-9, April 2009. [11] V. Boominathan, K. Mitra, and A. Veeraraghavan, "Improving resolution and depth-of-field of light field cameras using a hybrid imaging system," Proc. IEEE International Conference on Computational Photography, pp.1-10, May 2014. [12] D. Cho, M. Lee, S. Kim, and Y.-W. Tai, "Modeling the calibration pipeline of the Lytro camera for high quality light-field image reconstruction," Proc. IEEE International Conference on Computer Vision, pp.3280-3287, December 2013. [13] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, "Learning low-level vision," International Journal of Computer Vision, vol. 40, no. 1, pp. 25-47, October 2000. [14] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," Proc. SIGGRAPH '96, pp. 43-54, August 1996. [15] X. Huang, O. Cossairt, "Dictionary Learning Based Color Demosaicing for Plenoptic Cameras," Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 455-460, June 2014. [16] M. Levoy and P. Hanrahan, "Light field rendering," Proc. SIGGRAPH '96, pp. 31-42, August 1996. [17] Z. Li, Image patch modeling in a light field, EECS Department, University of California, Berkeley, Doctorate thesis, May 2014. [18] K. Mitra and A. Veeraraghavan, "Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior," Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.22-28, June 2012. [19] S. Wanner and B. Goldluecke, "Spatial and angular variational superresolution of 4d light fields," Proc. European Conference on Computer Vision, vol. 7576, pp. 608-621, October 2012. [20] C.-Y. Yang, C. Ma, and M.-H. Yang, "Single-image super-resolution: A benchmark," Proc. European Conference on Computer Vision, vol. 8692, pp. 372-386, September 2014. - 2013 2 : - 2015 8 : - 2015 7 ~ : ( ) - ORCID : http://orcid.org/0000-0001-5309-8898 - : (, Light field ), GPGPU - 1995 2 : - 1997 2 : - 2001 8 : - 2001 9 ~ 2004 3 : - 2007 1 ~ 2008 2 : Mitsubishi Electric Research Laboratories (MERL) - 2014 9 ~ 2015 8 : MIT Media Lab, Visiting Associate Professor - 2004 3 ~ : - ORCID : http://orcid.org/0000-0003-4774-7841 - : ( 3, 3, computational photography), GPGPU