1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, 2019 1 (JBE Vol. 24, No. 1, January 2019) https://doi.org/10.5909/jbe.2019.24.1.3 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) 360 VR a), a) Color and Illumination Compensation Algorithm for 360 VR Panorama Image Da-yoon Nam a) and Jong-Ki Han a) 360 VR. 360 VR,., 360 VR,. 360 VR, 360 VR.,. Abstract Techniques related to 360 VR service have been developed to improve the quality of the stitched image and video, where illumination compensation scheme is one of the important tools. Among the conventional illumination compensation algorithms, Gain-based compensation and Block Gain-based compensation algorithms have shown the outstanding performances in the process of making panorama picture. However, those are ineffective in the 360 VR service, because the disparity between illuminations of the multiple pictures in 360 VR is much more than that in making the panorama picture. In addition, the number of the pictures to be stitched in 360 VR system is more than that in the conventional panorama image system. Thus, we propose a preprocessing tool to enhance the illumination compensation algorithm so that the method reduces the degradation in the stitched picture of 360 VR systems. The proposed algorithm consists of color compensation and illumination compensation. The simulation results show that the proposed technique improve the conventional techniques without additional complexity. Keywords : Illumination Compensation, 360 VR, Image Stitching, Panorama Image a) (Sejong University, Dept. of Electrical Engineering) Corresponding Author : (Jong-Ki Han) E-mail: hjk@sejong.edu Tel: +82-2-3408-3739 ORCID:https://orcid.org/0000-0002-5036-7199 This work was partly supported by the National Research Foundation of Korea (NRF) under Grant NRF-2018R1A2A2A05023117 and partly by Institute for Information & communications Technology Promotion (IITP) under Grant 2017-0-00486 funded by the Korea government (MSIT). Manuscript received September 11, 2018; Revised November 17, 2018; Accepted November 17, 2018. Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND (http://creativecommons.org/licenses/by-nc-nd/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.
(JBE Vol. 24, No. 1, January 2019). ''(stitching) [1]. 360 VR 360 '', [2,3]. 360., 50~70.,,,., 50~70,,.,.,,,., [4],,,...,. 360,,., 360,. OpenCV 3.4 [5] Gain-based [6] Block Gain-based [7]. (gain),. Gain-based, Block Gain-based. Gain-based Block Gainbased, 5~10, 360,.,.,,,. Gain-based Block Gain-based,. Gain-based Block gain-based (color correction) [8]... 360VR, Gain-based Block Gain-based.,...,.. 360 VR
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) 1. 360 VR 360 VR. 1., ERP (spherical warping)., (intensity) (exposure compensation). (seam finder). Seam finder (Blending).. 2 flowchart. 1. 360 VR Fig. 1. Concept of the 360 VR image composition system 1.. 360. (feature). (pairwise matching).. (Estimator) [9]. Estimator (Bundle adjustment). 2. 360 VR Fig. 2. Flowchart of the 360 VR image composition algorithm 2 (Exposure Compensation).
(JBE Vol. 24, No. 1, January 2019) Gain-based Block gain- based... 2. Gain-based Gain-based (gain) [6].. Gainbased. Gain-based., n m (ROI: Region of Interest),. ROI R(), G(), B() (intensity). n m x y n ROI, n. (2). (2), 1 b. 3. Gain-based Fig. 3. Gain-based illuminance compensation
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image). n m, m 0. n.,.. A. n, m. (7). (3) (5) A b 1 g. A g b g. g n n gain. n Gain-based., (8) g 1. gain 1,. n, 1. 1,., (6), n, m n. (7) m n. (6) n, n. (7) (6), n m. (7). 3. Block Gain-based Block Gain-based Gain-based [7]. Gain-based gain, Block Gain-based gain., 1, 1. gain, gain gain, (Low pass filter)., gain.
(JBE Vol. 24, No. 1, January 2019) 4. Block gain-based Fig. 4. Block gain-based illuminance compensation n x y R, G, B,, gain,,, Block Gain-based. 3 Gain-based.,..,, Gain-based., 4 Block Gain-based 5~6 Gain-based.,.,, 360 VR. 5 Gainbased Block Gain-based.,. 5 (a),. 5 (b) Gain-based,. 5 (c) Block Gain-based (b),., 5 (b) (c),.
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) (b) (a) (b) (c) 5. Gain-based Block Gain-based Fig. 5. Comparison between Gain-based compensation and Block Gain-based compensation algorithms. (a) Stitching without illumination compensation, (b) Stitched image with Gain-based illumination compensation algorithm, (c) Stitched image with Block Gain-based illumination compensation algorithm
(JBE Vol. 24, No. 1, January 2019).,. 6,. (a), (b).,.. 7.,,.. 6 7,,,., 6 7 1. 1 Average brightness. 0~1, 1.. 1 Average R:G:B R, G, B,. (a) (b) 6. () Fig. 6. Change of brightness according to obstacle (hand)
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) 7. Fig. 7. Pictures according to the direction of the camera (b) 1. Table 1. Comparison of the averaged brightness and three primary colors in similar pictures Images Average brightness Average R : G : B Fig 6 (a) 0.6413 0.58 : 0.35 : 0.07 Fig 6 (b) 0.3135 0.64 : 0.25 : 0.01 Fig 7 (a) 0.4882 0.35 : 0.33 : 0.32 Fig 7 (b) 0.7557 0.34 : 0.33 : 0.33. 360 VR, 6 7., Gain-based Block Gain-based,,, (contrast).,. 360,, 360.. Gainbased,,.,. II 3 4. II.
(JBE Vol. 24, No. 1, January 2019) 1. R, G, B. R, G, B R, G, B.. R, B, G. R. G B., R (), G(), B(). n R, G, B,,.,, (8). IN. n p,,. p p,,, n R, G, B xy, xy, xy. xy. xy p p p xy p (10), xy, xy, xy p, p, p. p p p p xy. (10) xy, xy, xy, n R, G, B R, G, B. n xy, xy, xy (threshold). xy xy xy xy min min max min min min min YCrCb Cr [4]. max max max YCrCb Cr [10]. n. R, G, B,. 2.. (contrast),. x, y., xy. (15). (1)
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image). (15), 1.,,.,,.., n xy xy xy xy (threshold). xy max threshold min min min max,., Gain-based. (11) (16),.,.,,., 1 2.2~2.4, 2 1.4~1.6. 3. flowchart 8. Gain-based. (threshold), R, G, B R, G, B (intensity).,. 8. Fig. 8. The flowchart of the proposed algorithm
(JBE Vol. 24, No. 1, January 2019) 8 Gain-based.. Block-gain based Gain-based. Block-gain based Gain-based,. Gain-based, Block-based.,..,. Gain-based [6] Block Gain-based [7]. Gain-based Block Gain-based (gain)., gain. Gainbased gain., Block Gain-based gain gain.,., (brightness) (hue).,,.. 15 (a) Gain-based, (b) Block Gainbased, (c). S6 SM-G920K. 9 540x960 jpg 43, 11 540x960 jpg 5, 12 540x960 jpg 3, 14 540x960 9 jpg 11, 15 540x960 jpg 3, 16 960 x 540 1 jpg 59, 17 540x960 jpg 3. 9,.,,,. 10. 10, 11.,,. Gain-based Block Gain-based,,.. (11). (11)... 12 11. 11.
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) The image stitched by Gain-based compensation (b) The image stitched by Block Gain-based compensation (c) The image stitched by the proposed compensatoin 9. Fig. 9. The stitched images resulted from the various compensation algorithms 10. 11 Fig. 10. Input images for Fig 11
(JBE Vol. 24, No. 1, January 2019) (a) (b) (c) 11. Fig. 11. The stitched images resulted from the various compensation algorithms with input images having the different colors, (a) The image stitched by Gain-based compensation, (b) The image stitched by Block Gain-based compensation, (c) The image stitched by the proposed compensatoin algorithm
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) (b) (c) 12. () Fig. 12. The stitched images (hallway) resulted from the various compensation algorithms with input images having the different colors, (a) The image stitched by Gain-based compensation, (b) The image stitched by Block Gain-based compensation, (c) The image stitched by the proposed compensatoin algorithm,. 12 (c),. 13,,,,.,.
(JBE Vol. 24, No. 1, January 2019) 13. 14 Fig. 13. Input images for Fig 14 (a) (b) (c) 14. Fig. 14. The stitched images resulted from the various compensation algorithms with input images having the different brightness, (a) The image stitched by Gain-based compensation, (b) The image stitched by Block Gain-based compensation, (c) The image stitched by the proposed compensatoin algorithm,. 14 (a) (b). Gain-based. Block Gain-based,
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) no compesation (b) only color compesation (c) only brightness compesation (d) proposed compensation 15. Fig. 15. The stitched images resulted from the various compensation algorithms with input images having the different brightness and color, (a) The image stitched by no compensation, (b) The image stitched by only color compensation, (c) The image stitched by only brightness compensation, (d)the image stitched by the proposed compensatoin algorithm,.,. 15 (a). 15 (b), (c).. 15 (d) 15 (a). 15 (c) 15 (d), 15 (b) 15 (d). 16 360
(JBE Vol. 24, No. 1, January 2019) (a) Gain-based compensation (b) Block Gain-based compensation 16. 360 Fig. 16. Horizontality 360 degree panorama (c) Proposed compensatoin,. 16,.,,.,,. Gain-based, Gain.
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) Gain-based compensation (b) Block Gain-based compensation (c) Proposed compensatoin 17. Fig. 17. The stitched images resulted from the various compensation algorithms with input images having the similar colors and brightness
(JBE Vol. 24, No. 1, January 2019) 17., Gain-based... 2. Table 2. Comparison between run times of Gain-based compensation, Block Gain-based compensation, and the proposed compensation algorithms Gain-based compensation [sec] Block gain-based compensation [sec] Proposed compensation [sec] Fig 9 0.622 573.667 0.628 Fig 11 0.015 0.136 0.020 Fig 14 0.059 0.719 0.065 Fig 16 0.632 812.519 0.633 2 4 Gain-based Block gain- based. 10. Fig 9 43, Fig 11 5, Fig 14 11, Fig 16 59. 2, Block Gain-based Gain-based. Gain-based Block gain-based. Block gain-based Gain-based,, (gain), Block gain-based. 2 Gain-based, Fig 11 Fig 14, Fig 9 Fig 16.. Gain-based,, Gain-based.,, Gain-based., Gain-based. 3. Fig 11 0 4 Table 3. Comparison of the averaged brightness and three primary colors in the overlap region between image0 and image4 of Fig 11 No compensation Gain-based compensation Block gain-based compensation Proposed compensation Image 0 brightness Image 4 brightness Image 0 R : G : B Image 4 R : G : B 0.488 0.438 19 : 37 : 44 30 : 32 : 38 0.506 0.477 19 : 37 : 44 30 : 32 : 38 0.476 0.444 20 : 37 : 43 30 : 32 : 38 0.515 0.495 22 : 36 : 42 30 : 32 : 38 3 1. 3 Fig 11 0 4. 3 Image0 Image4,,., Fig 9 0 4
1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (11)..,,.. Gain-based,.,.,.,..,. (References) [1] Wei Xu, "Panoramic Video Stitching," Computer Science Graduate Theses & Dissertations. 47, 2012. [2] Jinwoong Jung, Joon-Young Lee, Byungmoon Kim, and Seungyong Lee, Upright adjustment of 360 spherical panoramas, 2017 IEEE Virtual Reality, pp 251 252, 2017. [3] Myeongah Cho, Junsik Kim, and Kyuheon Kim, Three-Dimensional Rotation Angle Preprocessing and Weighted Blending for Fast Panoramic Image Method, Journal of Broadcast Engineering, Vol. 23, No. 2, pp 235-245, March 2018. [4] David G. Lowe, Distinctive Image Features from Scale-Invariant Key points, International Journal of Computer Vision, Vol 60, No 2, pp 91 110, November 2004. [5] Kaili Chen and Meiling Wang, Image stitching algorithm research based on OpenCV, Proceedings of the 33rd Chinese Control Conference, pp. 7292 7297, 2014. [6] Matthew Brown and David G. Lowe, Automatic Panoramic Image Stitching using Invariant Features, International Journal of Computer Vision, Vol. 74, No. 1, pp. 59-73, 2007. [7] Wei Xu and Jane Mulligan, Performance Evaluation of Color Correction Approaches for Automatic Multi-view Image and Video Stitching, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 263 270, 2010. [8] Meer Sadeq Billah and Ahn Heejune, Stitching Method of Videos Recorded by Multiple Handheld Cameras, Journal of the Korea Industrial Information Systems Research, Vol. 22, No. 3, pp.27-38, June 2017. [9] Heung-Yeung Shum and Richard Szeliski, Panoramic vision, Springer-Verlag New York, Secaucus NJ USA, pp.227-268, 2001. [10] Benoit Payette, Color Space Converter: R G B to Y CrCb, XAPP 637(v1.0), September 2002. - 2016 3 ~ : - ORCID : https://orcid.org/0000-0002-6573-4932 - :
(JBE Vol. 24, No. 1, January 2019) - 1992 : KAIST - 1994 : KAIST - 1999 : KAIST - 1999 3 ~ 2001 8 : DM - 2001 9 ~ : - 2008 9 ~ 2009 8 : University California San Diego (UCSD) Visiting Scholar - ORCID : https://orcid.org/0000-0002-5036-7199 - :,,,