(JBE Vol. 23, No. 6, November 2018) (Regular Paper) 23 6, (JBE Vol. 23, No. 6, November 2018) ISSN 2

Size: px

Start display at page:

Download "(JBE Vol. 23, No. 6, November 2018) (Regular Paper) 23 6, (JBE Vol. 23, No. 6, November 2018) ISSN 2"

미숙 김
4 years ago
Views:

1 (JBE Vol. 23, No. 6, November 2018) (Regular Paper) 23 6, (JBE Vol. 23, No. 6, November 2018) ISSN (Online) ISSN (Print) 360 VR seam finding a), a) Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR Da-yoon Nam a) and Jong-Ki Han a) 360 VR (seam). seam finding (Voronoi), dynamic programming, graph cut (view disparity) ghost.,.,,.,,. Abstract Seam finding algorithm is one of the most important techniques to construct the high quality 360 VR image. We found that some degradations, such as ghost effect, are generated when the conventional seam finding algorithms (for examples, Voronoi algorithm, Dynamic Programming algorithm, Graph Cut algorithm) are applied, because those make the inefficient masks which cross the body of main objects. In this paper, we proposed an advanced seam finding algorithm providing the efficient masks which go through background region, instead of the body of objects. Simulation results show that the proposed algorithm outperforms the conventional techniques in the viewpoint of the quality of the stitched image. Keywords: Seam finder, 360 VR, Image Stitching, Panorama Image a) (Sejong University, Dept. of Electrical Engineering) Corresponding Author : (Jong-Ki Han) hjk@sejong.edu Tel: ORCID: This work was partly supported by the National Research Foundation of Korea (NRF) under Grant NRF-2018R1A2A2A and partly by Institute for Information & communications Technology Promotion (IITP) under Grant funded by the Korea government (MSIT). Manuscript received September 12, 2018; Revised October 29, 2018; Accepted October 29, Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND ( which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.

2 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR). (Image Stitching) [1,2]. 360 VR..,,, (Feature) (Feature Extraction), (Feature Matching), (Warping), (Blending).,.,.,,,,. (Seam). feathering (Pyramid Blend- ing) [3], (Poisson Blending) [4]..,.,.,.. seam finder. seam finder,, (Ghosting) [5]. seam finder,. seam finder Voronoi segment (Vo- ronoi) [6], (Dyna- mic Programming) [7] (Graph Cut) [8], (Minimal Error Boundary Cut) [9,10], [9] [10]. seam finder.,,.., seam finder., seam finder. seam finder.,.. 1. (Image Stitching) (Overlap Roi). (a) (feature extraction), (b) (feature matching), (c) (warping), (d) (blending)

(JBE Vol. 23, No. 6, November 2018) 4. SIFT (Scale Invariant Feature Transform) [11,12], SURF(Speeded Up Robust Features) [13], ORB(Oriented BRIEF) [14], FAST [15] (feature extraction).,,.

3 (JBE Vol. 23, No. 6, November 2018) 4. SIFT (Scale Invariant Feature Transform) [11,12], SURF(Speeded Up Robust Features) [13], ORB(Oriented BRIEF) [14], FAST [15] (feature extraction).,,.. (Descriptor).,, (Feature Matching). RANSAC [16] (Inlier), (Outlier).. (estimator) [17].,,,, (warping). (Spherical Projection), (Cylindrical Projection), (Plane Projection).,,. (seam finder), (blending) (mask) 0 1,.,.., (seam) Fig. 1. Flowchart of the image stitching algorithm

4 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR) 1, 2., , (a) 2. Warping Fig. 2. Warped images for the stitching process (b) (a) 3. Warping stitching Mask Fig. 3. Masks for stitching the warped pictures (b)

5 (JBE Vol. 23, No. 6, November 2018), 2 (a) (b). 3 2, 4 (a). (seam). 4 (b) (a) The image stitched by using masks of Fig.3 (b) The stitched image without mask 4. Fig. 4. Comparison between the stitched images using mask and no mask

1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR). 4 (a) 4 (b)..,..,, 4 (b) (Ghosting).,,.,,,., (seam). seam finder.

6 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR). 4 (a) 4 (b)..,..,, 4 (b) (Ghosting).,,.,,,., (seam). seam finder.. seam finder seam finder 3. (Voronoi) [6], (Dynamic Programming) [7], (Graph Cut) [8].,., seam finder. 1. Voronoi Seam Finder. Voronoi seam finder. 5. 0, 1..,, Fig. 5. Process of Voronoi algorithm

7 (JBE Vol. 23, No. 6, November 2018),. 0,.,, 1, 0. Voronoi,,.,,,. 2. Dynamic programming. 6. u v intensity, u v., u v. (1),.,, 3,... min 6. Fig. 6. Process of Dynamic programming algorithm

8 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR),.,.,. (seam). Dynamic programming Graph cut., [9]. 7 seam finder. 7 (a) The stitched image by using Voronoi (b) The stitched image by using Dynamic programming (c) The stitched image by using graph cut 7. seam finder ( ) Fig. 7. The stitched pictures by using the conventional seam finder algorithms, where the blending process is not applied to compare the performances of the seam finder algorithms

9 (JBE Vol. 23, No. 6, November 2018),. 7 (seam). Seam finder,,,,., seam finder. seam finder,. 3. Parking Lot 8. Fig. 8. Stitching error resulted from Parallax Column in the building Motorcycle in the road 9. Fig. 9. Stitching error resulted from moving objects Walking people

10 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR), ,,.,,,. 9.,,..,... seam finding flowchart 10. seam finder,.,.... seam finding.,.,, Fig. 10. Algorithm of the proposed seam finder (warping), (blending).. 11,.

11 (JBE Vol. 23, No. 6, November 2018) 11. Fig. 11. Mask extraction by using the difference of brightness 1.,.,., (intensity).,.,, x y (3).,,,

12 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR) (5).,.,,,,,., R, G, B 3 (Hue). (3. ). 2. (seam).,. 3.,. RGB HSV, (Hue). 0., HSV GRAY,. 4.,,. 5., 0 1 (Binary image). 0 1 threshold (7). Threshold 0, Threshold ,.,.. [18]., 31, ,. 31.,.,..,.

13 (JBE Vol. 23, No. 6, November 2018).., 11. low pass filter,. 7. image1 image2 6 subimage1, subimage2 seamimage seamimage 1 image2 0, seamimage 0 image1 0.,. 1, 0.,,.,. seamimage,,. 1.,.. seam finding seam finding,. voronoi, dynamic programming, graph cut ,. 14,,. (a) voronoi, (b) Dynamic programming, (c) graph cut, (d). S6 SM-G920K x960 jpg 2, x jpg ,. Voronoi, Dynamic programming, Graph cut, seam finder. 12 seam finder seam. 12, 13, (seam). 13 (a), (b), (c) seam finder,,,., (d)

1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR),. 12 13 (d),.

14 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR), (d),. (a) the image stitched by voronoi (b) the image stitched by dynamic programming (c) the image stitched by graph cut (d) the image stitched by the proposed seam finder 12. Seam finder Fig. 12. The images stitched by using a variety of seam finding algorithms, where the blending process is not applied to compare the performances of the seam finder algorithms

15 (JBE Vol. 23, No. 6, November 2018) (a) the image stitched by voronoi (b) the image stitched by dynamic programming (c) the image stitched by graph cut (d) the image stitched by the proposed seam finder 13. Seam finder Fig. 13. The images stitched by using a variety of seam finding algorithms, where the blending process is not applied to compare the performances of the seam finder algorithms

programming (c) the image stitched by graph cut (d) the image stitched by proposed seam finder 14.

16 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR) (a) the image stitched by voronoi (b) the image stitched by dynamic programming (c) the image stitched by graph cut (d) the image stitched by proposed seam finder 14. Seam finder Fig. 14. The images stitched by using a variety of seam finding algorithms, where the blending process is applied 14,. 14 (view disparity),. seam finder. 14 (a), (b), (c) seam finder,, (seam). 14 (d), seam finder,. seam finder,., seam finder, Fig. 15. The stitched image when there is no clear division between background and objects.,,.

17 (JBE Vol. 23, No. 6, November 2018). seam finder dynamic programming, voronoi,. seam finder,., seam finder,,,. seam finder seam.,.,,,.,,,.,,.,,.,.,,, seam finder. (References) [1] Matthew Brown and David G. Lowe, Automatic Panoramic Image Stitching using Invariant Features, International Journal of Computer Vision, Vol. 74, No. 1, pp , 2007 [2] R. Szeliski, Image Alignment and Stitching: A Tutorial, Foundations and Trends in Computer Graphics and Vision, Vol 2, Issue 1, January [3] Burt, P. J. and Adelson, E. H. A multiresolution spline with applications to image mosaics, ACM Transactions on Graphics, Vol. 2, No. 4, pp , October 1983 [4] Patrick Perez, Michel Gangnet and Andrew Blake, Poisson Image Editing, ACM Transactions on Graphics, Vol 22, Issue 3, pp , July 2003 [5] Matthew Uyttendaele, Ashley Eden and Richard Szeliski, Eliminating Ghosting and Exposure Artifacts in Image Mosaics, 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol 2, pp , 2001 [6] Jun Pan, Mi Wang, Deren Li, and Jonathan Li, Automatic Generation of Seamline Network Using Area Voronoi Diagrams With Overlap, IEEE Transactions on Geoscience and Remote Sensing, Vol. 47, Issue. 6, pp , April 2009 [7] Shai Avidan and Ariel Shamir, Seam Carving for Content-Aware Image Resizing, ACM Transactions on Graphics, Vol. 26, Issue. 3, No 10, pp.1-10, July 2007 [8] Yuri Y. Boykov and Marie-Pierre Jolly, Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images, International Conference on Computer Vision, Vol. 1, pp , July 2001 [9] Alexei A. Efros and William T. Freeman, Image quilting for texture synthesis and transfer, Proceedings of ACM SIGGRAPH 2001, pp , 2001 [10] Myeongah Cho, Junsik Kim, and Kyuheon Kim, Three-Dimensional Rotation Angle Preprocessing and Weighted Blending for Fast Panoramic Image Method, Journal of Broadcast Engineering, Vol. 23, No. 2, pp , March 2018 [11] David G. Lowe, Distinctive Image Features from Scale-Invariant Key- points, International Journal of Computer Vision, Vol. 60, No. 2, pp , November [12] Won-Jun Moon, Young-Ho Seo, and Dong-Wook Kim, Parameter Analysis for Time Reduction in Extracting SIFT Keypoints in the Aspect of Image Stitching, Journal of Broadcast Engineering Vol. 23, No. 4, pp , July 2018 [13] Herbert Bay, Andreas Ess, Tinne Tuytelaars and Luc Van Gool, Speeded-Up Robust Features (SURF), Computer Vision and Image Understanding, Vol. 10, Issue 3, pp , 2008 [14] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, Orb: An efficient alternative to sift or surf, International conference on computer vision, Vol. 1, pp , November 2011 [15] Siyoung Park, Jongho Kim, and Jisang Yoo, Fast Stitching Algorithm by using Feature Tracking, Journal of Broadcast Engineering, Vol. 20, No. 5, pp , September 2015 [16] Martin A. Fischler and Robert C. Bolles, Random sample consensus:

18 1 : 360 VR seam finding (Da-yoon Nam et al.: Seam Finding Algorithm using the Brightness Difference Between Pictures in 360 VR) A paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, Vol. 24, No. 6, pp , June 1981 [17] Heung-Yeung Shum and Richard Szeliski, Construction of Panoramic Image Mosaics with Global and Local Alignment, International Journal of Computer Vision, Vol. 36, pp , 1998 [18] R. Gonzalez and R. Woods, Digital Image Processing, Addison- Wesley, pp , : - ORCID : - : : KAIST : KAIST : KAIST ~ : DM ~ : ~ : University California San Diego (UCSD) Visiting Scholar - ORCID : - :,,,

1 : 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, (JBE Vol. 24, No

1 : 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, (JBE Vol. 24, No 1: 360 VR (Da-yoon Nam et al.: Color and Illumination Compensation Algorithm for 360 VR Panorama Image) (Special Paper) 24 1, 2019 1 (JBE Vol. 24, No. 1, January 2019) https://doi.org/10.5909/jbe.2019.24.1.3