1: (Su-Min Hong et al.: Depth Upsampling Method Using Total Generalized Variation) (Regular Paper) 21 6, 2016 11 (JBE Vol. 21, No. 6, November 2016) http://dx.doi.org/10.5909/jbe.2016.21.6.957 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) a), a) Depth Upsampling Method Using Total Generalized Variation Su-Min Hong a) and Yo-Sung Ho a), 3. 3.., KINECT Time-of-flight (ToF).,,., (JBU) (NAFDU)..,.,.,,. Abstract Acquisition of reliable depth maps is a critical requirement in many applications such as 3D videos and free-viewpoint TV. Depth information can be obtained from the object directly using physical sensors, such as infrared ray (IR) sensors. Recently, Time-of-Flight (ToF) range camera including KINECT depth camera became popular alternatives for dense depth sensing. Although ToF cameras can capture depth information for object in real time, but are noisy and subject to low resolutions. Recently, filter-based depth up-sampling algorithms such as joint bilateral upsampling (JBU) and noise-aware filter for depth up-sampling (NAFDU) have been proposed to get high quality depth information. However, these methods often lead to texture copying in the upsampled depth map. To overcome this limitation, we formulate a convex optimization problem using higher order regularization for depth map upsampling. We decrease the texture copying problem of the upsampled depth map by using edge weighting term that chosen by the edge information. Experimental results have shown that our scheme produced more reliable depth maps compared with previous methods. Keyword : ToF camera, depth map upsampling, total generalized variation, diffusion tensor Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND (http://creativecommons.org/licenses/by-nc-nd/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.
(JBE Vol. 21, No. 6, November 2016)., 3 [1]. 3,,. 3 (passive sensor-based method) (active sensor-based method).., 3,. 3,.. Time-of-flight (ToF).,,.,,,. 3, a) (School of Electrical Engineering and Computer Science, GIST) Corresponding Author : (Yo-Sung Ho) E-mail: hoyo@gist.ac.kr Tel: +82-62-715-2211 ORCID: http://orcid.org/0000-0002-7220-1034 Manuscript received August 30, 2016; Revised November 7, 2016; Accepted November 7, 2016.., [2]., [3,4].. (joint bilateral upsampling, JBU) NAFDU (noise-aware filter for depth upsampling) [5,6].,., (Markov random field, MRF) [7].,.. 2 Total generalized variation (TGV),, diffusion tensor.,..,.,.,
1: (Su-Min Hong et al.: Depth Upsampling Method Using Total Generalized Variation),., 2 Total generalized variation (TGV),, anisotropic diffusion tensor. 1.,.,., 3, Sparse depth map ( ).. arg (1).,.., 0, 1.. 1 smoothness Total variation semi norm. Total variation (TV). TV, (staircasing effects)., Total generalized variation [8]. 2. Total generalized variation Total generalized variation Total variation. Total variation. Total variation,. Total generalized variation, 2 Total generalized variation. 2 Total generalized variation. min,,., Total generalized variation 1 2. Ferstl et al. Total generalized variation anisotropic diffusion tensor ATGV (Image Guided Depth Upsampling using Anisotropic Total Generalized Variationa) [9]. ATGV,.
(JBE Vol. 21, No. 6, November 2016).,. ATGV. 1. tensor.,.,.. 1. Table 1. Source of the texture copying problem diffusion tensor, diffusion tensor. 5. 1. ATGV Fig. 1. Enlarged texture copying area of the ATGV 1 ATGV.,. 3. Adaptive weighting tensor,.,, 0.,,,.,, 1.,, 0~1.,. 6. 7.
1: (Su-Min Hong et al.: Depth Upsampling Method Using Total Generalized Variation) 2. Fig. 2. Edge information 7 6,,. 3.., tensor. anisotropic diffusion tensor,. 8 tensor. exp 3. Fig. 3. Blending function ATGV,. 5,, anisotropic diffusion tensor. tensor.,.. TGV,., Legendre Fenchel saddle point, primal-dual [9]. exp. 8, min
962 방송공학회논문지 제21권 제6호, 2016년 11월 (JBE Vol. 21, No. 6, November 2016) 으로 실험하였다. 그리고 블렌딩 함수의 는 2배 / 4배 / 8배각각의 경우에 대하여 0.11, 20 / 0.1, 0.005 / 4.8, 0.01로 실험하였다. 이러한 파라미터 값들은 실험을 통해 책정되었다. 그림 5는 제안하는 방법을 통해 업샘플링 한 깊이맵들을 보여준다. 그림 6은 업샘플링된 깊이맵의 특정 부분을 확대한 영상 이다. 그림에서 볼 수 있듯이, 기존의 ATGV 방법이 가지고 0.267, 0.03 그림 4. 깊이맵 업샘플링 실험을 위한 이미지 세트: Art", "Cones" Fig. 4. Image sets for experiments on depth upsampling: Art, Cones 에서 제공하는 테스트 영상 Cones, Tsukuba, Middlebury Venus, Books, Bowling, Baby, Art, Moebius, Monopoly, 를 사용하였다. 제안하는 방법의 성능을 살펴보기 위해 원본 깊이맵을 각각 1/2배, 1/4배 그리고 1/8배로 다운샘플링 하였다. 다운 샘플링 방법은 n배 다운샘플 깊이맵을 만들 때, 원본 깊이 맵에서 x축과 y축에 대하여 각각 n번째 화소마다 그 값을 추출하였다. 실험을 진행하는 동안, diffusion tensor의 파라 미터 는 모든 영상과 업샘플링 배수에 대하여 10, 0.75 를 사용하였다. 또한, 에너지 모델의 은 2배 / 4배 / 8배 각각의 경우에 대하여 0.154, 0.023 / 0.05, 0.0056 / Aloe 그림 5. 제안하는 방법을 통해 업샘플링된 깊이맵 그림 6. 질감 복사 영역 확대 ( 4) Fig. 6. Enlarged upsampled depth maps by ATGV and proposed method ( 4) Fig. 5. Upsampled depth maps by the proposed algorithm
1: (Su-Min Hong et al.: Depth Upsampling Method Using Total Generalized Variation). root- mean-square error (RMSE). RMSE. RMSE XY X Y x y f x y gx y X, Y. ground- truth 2. RMSE Table 2. Comparison of root-mean-square error. ground-truth. 2, 3, 4 Middelbury 2, 4, 8. 2 JBU, NAFDU, MRF, ATGV, RMSE... total generalized variation.,.,,., ATGV, 0.26% RMSE, 5 0.41% RMSE.,. (References) [1] Smolic, K. Mueller, P. Merkle, C. Fehn, P. Kauff, P. Eisert, and T. Wiegand, "3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards," in Proc. of IEEE International Conference on Multimedia and Expo, pp. 2161-2164, July 2006. [2] Jong In Gil, Saeed Mahmoudpour, and Manbae Kim, Analysis of Relationship between Objective Performance Measurement and 3D
(JBE Vol. 21, No. 6, November 2016) Visual Discomfort in Depth Map Upsampling, JBE, vol 19, no.1, pp. 31-43, January 2014. [3] Jaehun Kim, Kibaek Kim, Gwanggil Jeon, and Jechang Jeong, New Adaptive Interpolation Based on Edge Direction extracted from the DCT Coefficient Distribution, JBE, vol 18, no.1, pp. 10-20, January 2013. [4] Dong-Won Shin and Yo-Sung Ho, Temporally-Consistent High-Resolution Depth Video Generation in Background Region, JBE, vol 20, no.3, pp. 414-420, May 2015. [5] J. Kopf, M.F. Cohen, D. Lischinski, and M. Uyttendaele, Joint bilateral upsampling, ACM Transactions on Graphics. vol 26, no. 3, pp. 1-6, July. 2007. [6] D. Chan, H. Buisman, C. Theobalt, and S. Thrun, "A noise-aware filter for real-time depth upsampling", in Proc. of ECCV Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications, pp. 1-12, 2008. [7] J. Diebel and S. Thrun, "An Application of Markov Random Fields to Range Sensing," in Proc. of Advances in Neural Information Processing Systems, vol. 18, pp. 291-298, Dec. 2006. [8] K. Bredies, K. Kunisch, and T. Pock, Total generalized variation, SIAM J. Imaging Sci., vol. 3, no. 3, pp. 492526, 2010. [9] D. Ferstl, C. Reinbacher, R. Ranftl, Matthias Ruther, and H. Bischof, Image guided upsampling using anisotropic total generalized variation, in Proc. IEEE ICCV, 2013-2014 : () - 2016 : () - ORCID : http://orcid.org/0000-0003-0239-718x - : 3D,, - 1981 : () - 1983 : () - 1989 : Univ. of California, Santa Barbara, Dept. of Electrical and Computer Engineering() - 1983 ~ 1995 : - 1990 ~ 1993 : Philips, Senior Research Member - 1995 ~ : - ORCID : http://orcid.org/0000-0002-7220-1034 - :,,, MPEG, 3 TV,