(JBE Vol. 20, No. 5, September 2015) (Special Paper) 20 5, (JBE Vol. 20, No. 5, September 2015) ISS

Transcription

1 (Special Paper) 20 5, (JBE Vol. 20, No. 5, September 2015) ISSN (Online) ISSN (Print) 4 Light Field Dictionary Learning a), a) Dictionary Learning based Superresolution on 4D Light Field Images Seung-Jae Lee a) and In Kyu Park a) Light field 4 light field 2 (spatial domain) 2 (angular domain). 4 light field 2 CMOS. 4 light field, 4 light field (dictionary learning-based) (superresolution). 4 light field 4 (patch),. 2. light field Lytro. Abstract A 4D light field image is represented in traditional 2D spatial domain and additional 2D angular domain. The 4D light field has a resolution limitation both in spatial and angular domains since 4D signals are captured by 2D CMOS sensor with limited resolution. In this paper, we propose a dictionary learning-based superresolution algorithm in 4D light field domain to overcome the resolution limitation. The proposed algorithm performs dictionary learning using a large number of extracted 4D light field patches. Then, a high resolution light field image is reconstructed from a low resolution input using the learned dictionary. In this paper, we reconstruct a 4D light field image to have double resolution both in spatial and angular domains. Experimental result shows that the proposed method outperforms the traditional method for the test images captured by a commercial light field camera, i.e. Lytro. Keyword : Superresolution, dictionary learning, light field, spatial domain, angular domain, Lytro a) (Department of Information and Communication Engineering, Inha University) Corresponding Author : (In Kyu Park) pik@inha.ac.kr Tel: ORCID: ( ) (NRF-2013R1A2A2A ). Manuscript received July 10, 2015; Revised September 22, 2015; Accepted September 22, 2015.

2 1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images). 4 light field 2,, 3. Light field 2. light field., 4 2 (trade-off). light field [1][2].,., Farsiu [3], Freeman [4], Yang Yang [5]., light field. Yang Yang [5]. 2 light field. light field. 4,., light field,,. 3,. 4.. Light field 4 light field, 4. 4 light field., light field 4, light field,. 4 light field light field

3 .. 2. light field.,. 1. Light Field light field 1(a). 1(a).. Dansereau [6] (sub-aperture image). 4 1(b) 4 light field.,,, Light field.. 2 light field light field. 4 light field , (a) raw data (b) 4D light field image 1. light field (Lytro) Fig 1. Light field sub-aperture images generated from the raw data captured by commercial light field camera (Lytro)

4 1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images) 2. 4 Fig 2. Overview of 4D training patch reconstruction light field 4.. K- (K-means clustering). K Fig 3. Overview of the dictionary learning procedure

5 4 light field (4D light field image), light field (LR light field image) 4 (HR patch) (LR patch). K- K (cluster center). L ( ), H ( ).. (1) (regression coefficient) C ( ). C argmin H CL c (dictionary).,. 4. Light Field light field,. light field,. Freedman Fattal [7] Glasner [8],. Freedman Fattal [7], Glasner [8]. 4. light field Fig 4. Region division of an input light field image to match the image resolution of the dictionary and the input image

6 1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images)..,. 4.1 Light Field - light field (, ),. Light field ,., (2) C l h. h C l (2). 4.3 Light Field 16 (a) (b) (c) 5.. (a), (b) EPI bicubic, (c) Fig 5. Results of resolution enhancement at the boundary region. (a) Initial results of the proposed algorithm, (b) Result of bicubic interpolation on EPI, (c) Final result of the proposed method

7 light field., 5(a). 2, light field (plane) EPI (epipolar plane image) EPI bicubic EPI. EPI light field EPI EPI light field 5(b) 4 light field. light field 5(c). 4 light field.. Intel i7-3770k 3.5GHz CPU 16G RAM, light field Lytro. 4 light field Light field EPI bicubic Fig 6. Postprocessing algorithm to handle the image boundary. Bicubic interploation is applied after converting the light field image to EPI image

8 1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images) 7. Lytro Fig 7. Training dataset captured by Lytro camera (a) Bicubic (b) 8. Fig 8. Qualitative comparison in the spatial domain

9 .1. Table 1. Quantitative comparison in the spatial domain Algorithm tiger monkey beer can eiffel rhinoceros PSNR (db) SSIM Bicubic Interpolation Proposed Algorithm Bicubic Interpolation Proposed Algorithm light field,,,, light field 200,000. K- K, K 512.,. 2 (a) (1,2) (b) (3,12) (c) (10,10) (d) (14,6) (e) (16,16) 10. Fig 10. Results of superresolution at various locations in the angular domain

10 1 : 4 Light Field Dictionary Learning (Seung-Jae Lee et al.: Dictionary Learning based Superresolution on 4D Light Field Images). EPI. 2. Lytro. Lytro ( ) ( )., Lytro ( ) ( ). 8 bicubic. 1 8 PSNR(peak signal to noise ratio) SSIM(structural similarity), bicubic PSNR 2dB SSIM 0.4., light field ( ) bicubic Freeman [4]. 9 bicubic Freeman [4],,. 10. ((1,2), (3,12), (10,10), (14,6), (16,16)).. 4 light field 4 light field.,, 2 4 light field. 4,,,. 4 light field. (References) [1] Lytro, [2] Raytrix, [3] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super resolution," IEEE Trans. on Image Processing, vol. 13, no. 10, pp , October [4] W. T. Freeman, T. R. Jones, and E. C. Pasztor, "Example-based superresolution," IEEE Computer Graphics and Applications, vol. 22, no. 2, pp , March [5] C.-Y. Yang and M.-H. Yang, "Fast direct super-resolution by simple functions," Proc. IEEE International Conference on Computer Vision, pp , December [6] D. G. Dansereau, O. Pizarro, and S. B. Williams, "Decoding, calibration and rectification for lenselet-based plenoptic cameras," Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp , June [7] G. Freedman and R. Fattal, "Image and video upscaling from local self-examples," ACM Trans.on Graphics, vol. 30, no. 12, pp. 1-12, April [8] D. Glasner, S. Bagon, and M. Irani, "Super-resolution from a single image," Proc. IEEE International Conference on Computational

11 Photography, pp , September [9] T. E. Bishop and P. Favaro, "The light field camera: Extended depth of field, aliasing, and superresolution," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 34, no. 5, pp , May [10] T. E. Bishop, S. Zanetti, and P. Favaro, "Light field superresolution," Proc. IEEE International Conference on Computational Photography, pp.1-9, April [11] V. Boominathan, K. Mitra, and A. Veeraraghavan, "Improving resolution and depth-of-field of light field cameras using a hybrid imaging system," Proc. IEEE International Conference on Computational Photography, pp.1-10, May [12] D. Cho, M. Lee, S. Kim, and Y.-W. Tai, "Modeling the calibration pipeline of the Lytro camera for high quality light-field image reconstruction," Proc. IEEE International Conference on Computer Vision, pp , December [13] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, "Learning low-level vision," International Journal of Computer Vision, vol. 40, no. 1, pp , October [14] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," Proc. SIGGRAPH '96, pp , August [15] X. Huang, O. Cossairt, "Dictionary Learning Based Color Demosaicing for Plenoptic Cameras," Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp , June [16] M. Levoy and P. Hanrahan, "Light field rendering," Proc. SIGGRAPH '96, pp , August [17] Z. Li, Image patch modeling in a light field, EECS Department, University of California, Berkeley, Doctorate thesis, May [18] K. Mitra and A. Veeraraghavan, "Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior," Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.22-28, June [19] S. Wanner and B. Goldluecke, "Spatial and angular variational superresolution of 4d light fields," Proc. European Conference on Computer Vision, vol. 7576, pp , October [20] C.-Y. Yang, C. Ma, and M.-H. Yang, "Single-image super-resolution: A benchmark," Proc. European Conference on Computer Vision, vol. 8692, pp , September : : ~ : ( ) - ORCID : - : (, Light field ), GPGPU : : : ~ : ~ : Mitsubishi Electric Research Laboratories (MERL) ~ : MIT Media Lab, Visiting Associate Professor ~ : - ORCID : - : ( 3, 3, computational photography), GPGPU