1 : 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, (JBE Vol. 24, No

Size: px

Start display at page:

Download "1 : 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, (JBE Vol. 24, No"

재중 조
4 years ago
Views:

1 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, (JBE Vol. 24, No. 1, January 2019) ISSN (Online) ISSN (Print) 360 VR a), a) Color and Illumination Compensation Algorithm for 360 VR Panorama Image Da-yoon Nam a) and Jong-Ki Han a) 360 VR. 360 VR,., 360 VR,. 360 VR, 360 VR.,. Abstract Techniques related to 360 VR service have been developed to improve the quality of the stitched image and video, where illumination compensation scheme is one of the important tools. Among the conventional illumination compensation algorithms, Gain-based compensation and Block Gain-based compensation algorithms have shown the outstanding performances in the process of making panorama picture. However, those are ineffective in the 360 VR service, because the disparity between illuminations of the multiple pictures in 360 VR is much more than that in making the panorama picture. In addition, the number of the pictures to be stitched in 360 VR system is more than that in the conventional panorama image system. Thus, we propose a preprocessing tool to enhance the illumination compensation algorithm so that the method reduces the degradation in the stitched picture of 360 VR systems. The proposed algorithm consists of color compensation and illumination compensation. The simulation results show that the proposed technique improve the conventional techniques without additional complexity. Keywords : Illumination Compensation, 360 VR, Image Stitching, Panorama Image a) (Sejong University, Dept. of Electrical Engineering) Corresponding Author : (Jong-Ki Han) hjk@sejong.edu Tel: ORCID: This work was partly supported by the National Research Foundation of Korea (NRF) under Grant NRF-2018R1A2A2A and partly by Institute for Information & communications Technology Promotion (IITP) under Grant funded by the Korea government (MSIT). Manuscript received September 11, 2018; Revised November 17, 2018; Accepted November 17, Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND ( which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.

2 (JBE Vol. 24, No. 1, January 2019). ''(stitching) [1]. 360 VR 360 '', [2,3]. 360., 50~70.,,,., 50~70,,.,.,,,., [4],,,...,. 360,,., 360,. OpenCV 3.4 [5] Gain-based [6] Block Gain-based [7]. (gain),. Gain-based, Block Gain-based. Gain-based Block Gainbased, 5~10, 360,.,.,,,. Gain-based Block Gain-based,. Gain-based Block gain-based (color correction) [8] VR, Gain-based Block Gain-based.,..., VR

360 VR 360 VR. 1., ERP (spherical warping)., (intensity) (exposure compensation).

360 VR Fig. 1. Concept of the 360 VR image composition system 1.. 360. (feature).

3 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) VR 360 VR. 1., ERP (spherical warping)., (intensity) (exposure compensation). (seam finder). Seam finder (Blending).. 2 flowchart VR Fig. 1. Concept of the 360 VR image composition system (feature). (pairwise matching).. (Estimator) [9]. Estimator (Bundle adjustment) VR Fig. 2. Flowchart of the 360 VR image composition algorithm 2 (Exposure Compensation).

(JBE Vol. 24, No. 1, January 2019) Gain-based Block gain- based... 2. Gain-based Gain-based (gain) [6].. Gainbased. Gain-based., n m (ROI: Region of Interest),.

4 (JBE Vol. 24, No. 1, January 2019) Gain-based Block gain- based Gain-based Gain-based (gain) [6].. Gainbased. Gain-based., n m (ROI: Region of Interest),. ROI R(), G(), B() (intensity). n m x y n ROI, n. (2). (2), 1 b. 3. Gain-based Fig. 3. Gain-based illuminance compensation

5 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image). n m, m 0. n.,.. A. n, m. (7). (3) (5) A b 1 g. A g b g. g n n gain. n Gain-based., (8) g 1. gain 1,. n, 1. 1,., (6), n, m n. (7) m n. (6) n, n. (7) (6), n m. (7). 3. Block Gain-based Block Gain-based Gain-based [7]. Gain-based gain, Block Gain-based gain., 1, 1. gain, gain gain, (Low pass filter)., gain.

(JBE Vol. 24, No. 1, January 2019) 4. Block gain-based Fig. 4. Block gain-based illuminance compensation n x y R, G, B,, gain,,, Block Gain-based. 3 Gain-based.,..,, Gain-based.

6 (JBE Vol. 24, No. 1, January 2019) 4. Block gain-based Fig. 4. Block gain-based illuminance compensation n x y R, G, B,, gain,,, Block Gain-based. 3 Gain-based.,..,, Gain-based., 4 Block Gain-based 5~6 Gain-based.,.,, 360 VR. 5 Gainbased Block Gain-based.,. 5 (a),. 5 (b) Gain-based,. 5 (c) Block Gain-based (b),., 5 (b) (c),.

7 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) (b) (a) (b) (c) 5. Gain-based Block Gain-based Fig. 5. Comparison between Gain-based compensation and Block Gain-based compensation algorithms. (a) Stitching without illumination compensation, (b) Stitched image with Gain-based illumination compensation algorithm, (c) Stitched image with Block Gain-based illumination compensation algorithm

0~1, 1.. 1 Average R:G:B R, G, B,. (a) (b) 6.

8 (JBE Vol. 24, No. 1, January 2019).,. 6,. (a), (b).,.. 7.,,.. 6 7,,,., Average brightness. 0~1, Average R:G:B R, G, B,. (a) (b) 6. () Fig. 6. Change of brightness according to obstacle (hand)

Comparison of the averaged brightness and three primary colors in similar pictures Images Average brightness Average R : G : B Fig 6 (a) 0.

9 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) 7. Fig. 7. Pictures according to the direction of the camera (b) 1. Table 1. Comparison of the averaged brightness and three primary colors in similar pictures Images Average brightness Average R : G : B Fig 6 (a) : 0.35 : 0.07 Fig 6 (b) : 0.25 : 0.01 Fig 7 (a) : 0.33 : 0.32 Fig 7 (b) : 0.33 : VR, 6 7., Gain-based Block Gain-based,,, (contrast).,. 360,, Gainbased,,.,. II 3 4. II.

10 (JBE Vol. 24, No. 1, January 2019) 1. R, G, B. R, G, B R, G, B.. R, B, G. R. G B., R (), G(), B(). n R, G, B,,.,, (8). IN. n p,,. p p,,, n R, G, B xy, xy, xy. xy. xy p p p xy p (10), xy, xy, xy p, p, p. p p p p xy. (10) xy, xy, xy, n R, G, B R, G, B. n xy, xy, xy (threshold). xy xy xy xy min min max min min min min YCrCb Cr [4]. max max max YCrCb Cr [10]. n. R, G, B,. 2.. (contrast),. x, y., xy. (15). (1)

11 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image). (15), 1.,,.,,.., n xy xy xy xy (threshold). xy max threshold min min min max,., Gain-based. (11) (16),.,.,,., 1 2.2~2.4, 2 1.4~ flowchart 8. Gain-based. (threshold), R, G, B R, G, B (intensity).,. 8. Fig. 8. The flowchart of the proposed algorithm

12 (JBE Vol. 24, No. 1, January 2019) 8 Gain-based.. Block-gain based Gain-based. Block-gain based Gain-based,. Gain-based, Block-based.,..,. Gain-based [6] Block Gain-based [7]. Gain-based Block Gain-based (gain)., gain. Gainbased gain., Block Gain-based gain gain.,., (brightness) (hue).,,.. 15 (a) Gain-based, (b) Block Gainbased, (c). S6 SM-G920K x960 jpg 43, x960 jpg 5, x960 jpg 3, x960 9 jpg 11, x960 jpg 3, x jpg 59, x960 jpg 3. 9,.,,, , 11.,,. Gain-based Block Gain-based,,.. (11). (11)

13 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) The image stitched by Gain-based compensation (b) The image stitched by Block Gain-based compensation (c) The image stitched by the proposed compensatoin 9. Fig. 9. The stitched images resulted from the various compensation algorithms Fig. 10. Input images for Fig 11

14 (JBE Vol. 24, No. 1, January 2019) (a) (b) (c) 11. Fig. 11. The stitched images resulted from the various compensation algorithms with input images having the different colors, (a) The image stitched by Gain-based compensation, (b) The image stitched by Block Gain-based compensation, (c) The image stitched by the proposed compensatoin algorithm

15 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) (b) (c) 12. () Fig. 12. The stitched images (hallway) resulted from the various compensation algorithms with input images having the different colors, (a) The image stitched by Gain-based compensation, (b) The image stitched by Block Gain-based compensation, (c) The image stitched by the proposed compensatoin algorithm,. 12 (c),. 13,,,,.,.

16 (JBE Vol. 24, No. 1, January 2019) Fig. 13. Input images for Fig 14 (a) (b) (c) 14. Fig. 14. The stitched images resulted from the various compensation algorithms with input images having the different brightness, (a) The image stitched by Gain-based compensation, (b) The image stitched by Block Gain-based compensation, (c) The image stitched by the proposed compensatoin algorithm,. 14 (a) (b). Gain-based. Block Gain-based,

17 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) no compesation (b) only color compesation (c) only brightness compesation (d) proposed compensation 15. Fig. 15. The stitched images resulted from the various compensation algorithms with input images having the different brightness and color, (a) The image stitched by no compensation, (b) The image stitched by only color compensation, (c) The image stitched by only brightness compensation, (d)the image stitched by the proposed compensatoin algorithm,.,. 15 (a). 15 (b), (c).. 15 (d) 15 (a). 15 (c) 15 (d), 15 (b) 15 (d)

18 (JBE Vol. 24, No. 1, January 2019) (a) Gain-based compensation (b) Block Gain-based compensation Fig. 16. Horizontality 360 degree panorama (c) Proposed compensatoin,. 16,.,,.,,. Gain-based, Gain.

19 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (a) Gain-based compensation (b) Block Gain-based compensation (c) Proposed compensatoin 17. Fig. 17. The stitched images resulted from the various compensation algorithms with input images having the similar colors and brightness

20 (JBE Vol. 24, No. 1, January 2019) 17., Gain-based Table 2. Comparison between run times of Gain-based compensation, Block Gain-based compensation, and the proposed compensation algorithms Gain-based compensation [sec] Block gain-based compensation [sec] Proposed compensation [sec] Fig Fig Fig Fig Gain-based Block gain- based. 10. Fig 9 43, Fig 11 5, Fig 14 11, Fig , Block Gain-based Gain-based. Gain-based Block gain-based. Block gain-based Gain-based,, (gain), Block gain-based. 2 Gain-based, Fig 11 Fig 14, Fig 9 Fig 16.. Gain-based,, Gain-based.,, Gain-based., Gain-based. 3. Fig Table 3. Comparison of the averaged brightness and three primary colors in the overlap region between image0 and image4 of Fig 11 No compensation Gain-based compensation Block gain-based compensation Proposed compensation Image 0 brightness Image 4 brightness Image 0 R : G : B Image 4 R : G : B : 37 : : 32 : : 37 : : 32 : : 37 : : 32 : : 36 : : 32 : Fig Image0 Image4,,., Fig 9 0 4

21 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (11)..,,.. Gain-based,.,.,.,..,. (References) [1] Wei Xu, "Panoramic Video Stitching," Computer Science Graduate Theses & Dissertations. 47, [2] Jinwoong Jung, Joon-Young Lee, Byungmoon Kim, and Seungyong Lee, Upright adjustment of 360 spherical panoramas, 2017 IEEE Virtual Reality, pp , [3] Myeongah Cho, Junsik Kim, and Kyuheon Kim, Three-Dimensional Rotation Angle Preprocessing and Weighted Blending for Fast Panoramic Image Method, Journal of Broadcast Engineering, Vol. 23, No. 2, pp , March [4] David G. Lowe, Distinctive Image Features from Scale-Invariant Key points, International Journal of Computer Vision, Vol 60, No 2, pp , November [5] Kaili Chen and Meiling Wang, Image stitching algorithm research based on OpenCV, Proceedings of the 33rd Chinese Control Conference, pp , [6] Matthew Brown and David G. Lowe, Automatic Panoramic Image Stitching using Invariant Features, International Journal of Computer Vision, Vol. 74, No. 1, pp , [7] Wei Xu and Jane Mulligan, Performance Evaluation of Color Correction Approaches for Automatic Multi-view Image and Video Stitching, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp , [8] Meer Sadeq Billah and Ahn Heejune, Stitching Method of Videos Recorded by Multiple Handheld Cameras, Journal of the Korea Industrial Information Systems Research, Vol. 22, No. 3, pp.27-38, June [9] Heung-Yeung Shum and Richard Szeliski, Panoramic vision, Springer-Verlag New York, Secaucus NJ USA, pp , [10] Benoit Payette, Color Space Converter: R G B to Y CrCb, XAPP 637(v1.0), September ~ : - ORCID : - :

22 (JBE Vol. 24, No. 1, January 2019) : KAIST : KAIST : KAIST ~ : DM ~ : ~ : University California San Diego (UCSD) Visiting Scholar - ORCID : - :,,,

(JBE Vol. 23, No. 6, November 2018) (Regular Paper) 23 6, (JBE Vol. 23, No. 6, November 2018) ISSN 2

(JBE Vol. 23, No. 6, November 2018) (Regular Paper) 23 6, 2018 11 (JBE Vol. 23, No. 6, November 2018) https://doi.org/10.5909/jbe.2018.23.6.896 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) 360 VR seam