(JBE Vol. 21, No. 6, November 2016) (Special Paper) 21 6, 2016 11 (JBE Vol. 21, No. 6, November 2016) http://dx.doi.org/10.5909/jbe.2016.21.6.878 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) 3 a), a), a) Boundary Artifacts Reduction in View Synthesis of 3D Video System Dohoon Lee a), Yoonmo Yang a), and Byung Tae Oh a) 3., convex set projection onto convex sets (POCS)... Abstract This paper proposes an efficient method to remove the boundary artifacts of rendered views caused by damaged depth maps in the 3D video system. First, characteristics of boundary artifacts with the compression noise in depth maps are carefully studied. Then, the artifacts suppression method is proposed by the iterative projection onto convex sets (POCS) algorithm with setting the convex set in pixel and frequency domain. The proposed method is applied to both texture and depth maps separately during view rendering. The simulation results show the boundary artifacts are greatly reduced with improving the quality of synthesized views. Keyword : Boundary artifact, 3D video, Multiview plus depth map, POCS a) (Korea Aerospace University) Corresponding Author : (Byung Tae Oh) E-mail: byungoh@kau.ac.kr Tel: +82-2-300-0409 ORCID: http://orcid.org/0000-0003-1437-2422 2013 () (NRF-2013R1A1A1057779). 2016. Manuscript received September 7, 2016; Revised October 26, 2016; Accepted October 26, 2016...,,. High-definition (HD), 4K/8K Ultra HD (UHD) Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND (http://creativecommons.org/licenses/by-nc-nd/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.
2: 3 (Dohoon Lee et al.: Boundary Artifacts Reduction in View Synthesis of 3D Video System), 8-bit, High-dynamic-range (HDR) 10-bit 12-bit.,,,. 3, 3.,.,. (Motion parallax), -.,,,. (Multi-view plus depth map, MVD). MVD /, (depth-image based rendering, DIBR). MVD [1-3]. MVD 8-bit.,. DIBR,.,, DIBR.,. Projection onto convex sets (POCS),.. 2, 3. 4, 5.. 1. DIBR., 1D, (1)..,. disparity,
(JBE Vol. 21, No. 6, November 2016) (2).,,. (1), (2). (3), 1. Fig. 1. Model of Boundary artifacts 2. (a), (b) Fig. 2. (a) Original virtual view image, (b) Virtual view image containing boundary artifacts,.,, 1.. 2. 2..,, DIBR. [4-7].,... [8-9]. [10].,. [11-13]..
2: 3 (Dohoon Lee et al.: Boundary Artifacts Reduction in View Synthesis of 3D Video System),,.,,.. POCS. 1. Projection onto convex sets (POCS).,,. 3.,, POCS, convex set convex set. convex set. POCS,, convex set [14-16], convex set (a) (b) (c) 3. (a), (b), (c) (a) (b) Fig. 3. (a) Warped image from right view, (b) Warped image from left view, (c) Virtual view image with blending (a) and (b) 4. Fig. 4. Proposed algorithm
(JBE Vol. 21, No. 6, November 2016) [17-18]. convex set. convex set, convex set. 4. convex set 2 3. 2. convex set.., discrete cosine transform (DCT). 3.,..,, DCT.,. DCT convex set. NxN DCT, 2 DCT zig-zag 1., (4) convex set DCT 5. DCT convex set Fig. 5. Example of the projection onto convex set at DCT domain
2: 3 (Dohoon Lee et al.: Boundary Artifacts Reduction in View Synthesis of 3D Video System), p E., 5,, 3. 5. AC 10% p, E. (4) convex set. 5,.,, POCS. 3. convex set DCT POCS,. convex set., (4). min max.. 1. MVD system JCT-3V Common Test Condition (CTC), 3DV. reference S/W 3D-ATM v14.0 [19], View synthesis reference software (VSRS) general mode. CTC [20]. 2. 3, convex set.,,.., 6.,,.,.,
(JBE Vol. 21, No. 6, November 2016) (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) 6..,,,. (a) ~ (d) : Undo Dancer, (e) ~ (h) : Newspaper, (i) ~ (l) : Balloons. Fig. 6. Performance comparison of proposed algorithm. From left to right, image before processing, image after only texture processing, image after only depth map processing, image after texture and depth map processing. (a) ~ (d) : Undo Dancer, (e) ~ (h) : Newspaper, (i) ~ (l) : Balloons. 1. BDBR Table 1. Performance evaluation in terms of BDBR Sequence Texture Depth Texture+Depth Poznan Hall2 0.00 0.00 +0.02 Poznan Street -0.35-0.64-0.96 Undo Dancer -6.61-7.61-9.84 GT Fly +0.14 +0.11 +0.20 Kendo -0.12-0.13-0.28 Balloons -2.22-2.87-4.54 Newspaper -1.55-1.93-3.15 Shark -5.94-6.98-9.18 Avg. -2.08% -2.51% -3.47% Bjøntegaard delta bit rate (BDBR) [21]. 1,,.,., Quantization Parameter (QP) 2.
2: 3 (Dohoon Lee et al.: Boundary Artifacts Reduction in View Synthesis of 3D Video System) 2. PSNR Table 2. Performance evaluation in terms of PSNR with various QP conditions Sequence QP = 26 QP = 31 QP = 36 QP = 41 Poznan Hall2 0.000 0.000 0.000-0.003 Poznan Street 0.007 0.019 0.034 0.060 Undo Dancer 0.146 0.198 0.294 0.505 GT Fly 0.008 0.006-0.008-0.039 Kendo 0.004 0.007 0.012 0.029 Balloons 0.061 0.142 0.203 0.305 Newspaper 0.056 0.091 0.120 0.162 Shark 0.031 0.127 0.654 1.580 Avg. 0.039 db 0.074 db 0.164 db 0.325 db (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) 7.., QP = 26, 31, 36, 41 Fig. 7. Depth map and texture as intensity of compression. From left to right, original image, QP = 26, 31, 36, 41, 3 8., [6-8].
(JBE Vol. 21, No. 6, November 2016) 3. Table 3. Performance comparison in terms of BDBR Sequence DF [6] BAR [7] BF [8] Prop. DF + Prop. BAR + Prop. Poznan Hall2-0.56-0.38 0.12 0.02-0.06 0.36 Poznan Street -2.72-0.43-0.03-0.96-5.26-3.82 Undo Dancer -18.98-13.94-21.38-9.84-24.58-15.51 GT Fly -2.82-0.86-3.98 0.20-2.51-4.51 Kendo -0.23-0.02 9.84-0.28-1.60-0.88 Balloons -0.45-1.72 17.89-4.54-8.94-12.95 Newspaper -0.50-0.80-5.34-3.15-8.78-6.53 Shark -16.11-12.11-7.56-9.18-17.09-14.28 Avg. -5.29% -3.78% -1.31% -3.47% -8.60% -7.26% (a) (b) (c) (d) (e) (f) 8.. (a), (b), (c) DF [6], (d) BF [7], (e) BAR [8], (f) Fig. 8. Performance comparisons of algorithms. (a) Original image, (b) Virtual view image after compression, (c) Result image by DF [6], (d) Result image by BF [7], (e) Result image by BAR [8], (f) Result image by the proposed scheme 3.,., Undo Dancer, Shark 10% 1., 1,,.., DCT.,.,., 7.,. 2., GT Fly,
2: 3 (Dohoon Lee et al.: Boundary Artifacts Reduction in View Synthesis of 3D Video System)..,, convex set,.,, 3., DF BAR 3.21%, 3.48%.., DCT convex set POCS.,,, 3%.,, 8%. (References) [1] Text of ISO/IEC 14496-10:200X/FDAM 1 Multi-View Video Coding, ISO/IEC JTC1/SC29/WG11, Doc. N9978, Hannover, Germany, 2008. [2] A. Smolic, K. Mueller, N. Stefanoski, J. Ostermann, A. Gotchev, G. B. Akar, G. Triantafyllidis, and A. Koz, Coding algorithms for 3DTV a survey, IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1606-1621, 2007. [3] P. Merkle, A. Smolic, K. Muller, and T. Wiegand, Multiview video plus depth representation and coding, in Proc. IEEE Int. Conf. on Image Proc., vol. 1, pp. 201204, 2007. [4] J. Kopf, M. Cohen, D. Lischinski, and M. Uyttendaele, Joint bilateral upsampling, Proc. ACM Siggraph, vol. 26, no. 3, pp. 96, 2007. [5] Y.S. Kang, S.B. LEE, and Y.S. Ho, Depth map upsampling using depth local features, Electron. Lett., vol. 50, no. 3, pp. 170-171, 2014. [6] S.Y Kim, M.B. Kim, and Y.S. Ho, Depth map denoising based on the common distance Transform, Journal of Broadcast Engineering, vol. 17, no. 4, pp. 565-571, July 2012. [7] D.W. Shin, and Y.S Ho, Real-time depth Image refinement using hierarchical joint bilateral filter, Journal of Broadcast Engineering, vol. 19, no. 2, pp. 140-147, March 2014. [8] 3D-CE3.a results on dilation filter for depth post processing, ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Doc. JCT2-A0038, Stockholm, 2012. [9] Y. Zhao, C. Zhu, Z. Chen, D. Tian, and L. Yu. Boundary artifact reduction in view synthesis of 3D video: from perspective of texture-depth alignment. IEEE Trans. Broadcasting, vol. 57, no. 2, pp. 510-522, 2011. [10] C. Lee, and Y.S. Ho. Boundary filtering on synthesized views of 3D video. IEEE Second Int. Conf. Future Generation Communication and Networking Symposia, FGCNS'08, vol. 3, pp. 15-18, 2008. [11] C.H. Li, J.J. Tsai, and H.M. Hang. A triangular-warping based view synthesis scheme with enhanced artifact reduction for FTV. in Proc. IEEE Int. Conf. on Image Proc., pp. 1969-1972, 2011. [12] L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, and M. Tanimoto. Error suppression in view synthesis using reliability reasoning for FTV. IEEE 3DTV-Conf. The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), pp. 1-4, 2010. [13] L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, and M. Tanimoto. Artifact reduction using reliability reasoning for image generation of FTV. Journal of Visual Communication and Image Representation, vol. 21, no. 5, pp. 542-560, 2010. [14] A. Zakhor, Iterative procedures for reduction of blocking effects in transform image coding, IEEE Trans. on Circuits Syst. Video Technol., vol. 2, no. 1, pp. 91-95, 1992. [15] S. Minami and A. Zakhor, An optimization approach for removing blocking effects in transform coding, IEEE Trans. on Circuits Syst. Video Technol., vol. 5, no. 2, pp. 74-82, Apr. 1995. [16] J.H. Lee, and C.H. Lee, POCS based interpolation method for irregularly sampled image, Journal of Broadcast Engineering, vol. 16, no. 4, pp. 669-679, July 2011. [17] Y. Kim, C.S. Park, and S.J. Ko. Fast POCS based post-processing technique for HDTV. IEEE Trans. Consumer Electron., vol. 24, no. 4, pp. 1438-1447, 2003. [18] G.R. Kwon, H.K. Kim, Y. Kim, and S.J. Ko. An efficient POCS-based post-processing technique using wavelet transform in HDTV. IEEE Trans. Consumer Electron., vol. 51, no. 4, pp. 1283-1290, 2005.
888 방송공학회논문지 제21권 제6호, 2016년 11월 (JBE Vol. 21, No. 6, November 2016) [19] 3D-AVC Test Model 5, ITU-T SG16 WP3 ISO/IEC JTC1/SC29/ WG11, Doc. JCT3V-C1003, Geneva, 2013. [20] Common test condition of 3DV core experiments, ITU-T SG16 WP3 ISO/IEC JTC1/SC29/WG11, Doc. JCT3V-C1100, Geneva, 2013. [21] G. Bjøntegaard, Calculation of average PNSR differences between RD-curves, ITU-T SG.16 Q.6, Doc. VCEG M33, 2001. 저자소개 이도훈 년 8월 : 한국항공대학교 전자 및 항공전자공학 학사 년 8월 : 한국항공대학교 항공전자정보공학과 석사 년 10월 ~ 현재 : (주) Silicon Works 선행제품개발담당 알고리즘개발팀 연구원 주관심분야 : 영상처리 - 2014-2016 - 2016 - 양윤모 년 월 한국항공대학교 전자 및 항공전자공학 학사 년 월 현재 한국항공대학교 항공전자정보공학과 석사과정 주관심분야 영상처리 - 2015 2 : - 2015 3 ~ : - ORCID : http://orcid.org/0000-0003-2816-1685 : 오병태 - 년 8월 : 연세대학교 전기전자공학부 학사 년 8월 : Univ. of Southern California (USC), Dept. of Electrical Eng. 석사 및 박사 년 ~ 2013년 : 삼성종합기술원 전문연구원 년 ~ 현재 : 한국항공대학교 항공전자정보공학부 조교수 : http://orcid.org/0000-0003-1437-2422 주관심분야 : 3차원 영상시스템, 영상처리, 비디오압축 2003 2009 2009 2013 ORCID