(Special Paper) 20 2, 2015 3 (JBE Vol. 20, No. 2, March 2015) http://dx.doi.org/10.5909/jbe.2015.20.2.204 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) 3D a), a), a) Comparison of Objective Metrics and 3D Evaluation Using Upsampled Depth Map Saeed Mahmoudpour a), Changyeol Choi a), and Manbae Kim a)., 3D. PSNR,.,.. 3D,.. DSCQS., SSIM Edge-PSNR. Abstract Depth map upsampling is an approach to increase the spatial resolution of depth maps obtained from a depth camera. Depth map quality is closely related to 3D perception of stereoscopic image, multi-view image and holography. In general, the performance of upsampled depth map is evaluated by PSNR (Peak Signal to Noise Ratio). On the other hand, time-consuming 3D subjective tests requiring human subjects are carried out for examining the 3D perception as well as visual fatigue for 3D contents. Therefore, if an objective metric is closely correlated with a subjective test, the latter can be replaced by the objective metric. For this, this paper proposes a best metric by investigating the relationship between diverse objective metrics and 3D subjective tests. Diverse reference and no-reference metrics are adopted to evaluate the performance of upsampled depth maps. The subjective test is performed based on DSCQS test. From the utilization and analysis of three kinds of correlations, we validated that SSIM and Edge-PSNR can replace the subjective test. Keyword : Depth map, Upsampling, Objective metric, 3D subjective test, Correlation
2 : 3D (Saeed Mahmoudpour et al.: Comparison of Objective Metrics and 3D Evaluation Using Upsampled Depth Map). 3D. Time of Flight(ToF) (depth map)., RGB, [1-5]. HD(High Definition) FHD (Full HD), FHD UHD(Ultra High-Definition) (upsampling). RGB+Depth, (encoder), (decoder) [1]. 3D,. (bilinear upsampling, BLU) 4. (bicubic upsampling, BCU) 16. (bilateral upsampling, BU) [2]. BU a) (Dept. of Computer and Communications Engineering, Kangwon National University) Corresponding Author : (Manbae Kim) E-mail: manbae@kangwon.ac.kr Tel: +82-33-250-6395 ORCID:http://orcid.org/0000-0002-4702-8276 / () [KI002058, IP ], [10041082]. 2014. Manuscript received December 2, 2014; revised January 27, 2015; accepted January 27, 2015. (joint bilateral upsampling, JBU) [3]. (variance based upsampling, VBU), [4]. JBU, (non-edge).,, (adaptive bilateral upsampling, ABU) [5]. (gradient) (blur) (distance transform-based bilateral upsampling, DTBU) [6]. (quality assessment, QA) PSNR(Peak Signal-to-Noise Ratio). PSNR, 3D RGB DSCQS, SSCQS (subjective evaluation),. QA,. PSNR, 3D. [7] PSNR, (Blur metric) (sharpness degree) 3D., 3D. 3D. 7 3D (correlation),.,,
. (full-reference metric) (no-reference metric). PSNR, SSIM, (sharpness degree), sharpness degree, blur metric, BIQI, NIQE. [7],,.. 2. 3, 4. 5 3 4, 6. 7.., 3D, 3D., 3D. 1. RGB (low-resolution depth map, LRD), (high-resolution depth map, HRD)., 7. HRD. DIBR(depth image based rendering) RGB HRD, [1][5]. 1(a) QA (correlation), 1(b). (Pearson), (Spearman), (Kendall) QA.. (full-reference quality assessment, FRQA) (a) 1.. (a) (b) 3D Fig. 1. Block diagram of the experiment methodology. (a) Correlation between two measurements and (b) measurement of 3D perception using objective metric (b)
2 : 3D (Saeed Mahmoudpour et al.: Comparison of Objective Metrics and 3D Evaluation Using Upsampled Depth Map) (no-reference quality assessment, NRQA)..,. FRQA NRQA. 1. FRQA 1.1 PSNR PSNR(Peak Signal to Noise Ratio) (1). log., PSNR PSNR, Edge PNSR, Non-edge PNSR. 1.2 SSIM SSIM(structural similarity index measure) [8]. SSIM,,. f, g SSIM., SSIM lfgcfgsfg l(f,g), c(f,g) s(f,g),. σ f σ g,μ f μ g, σ fg (covariance). C 1, C 2 C 3 0. SSIM [0,1], 1. 1.3 VIF VIF(Visual Information Fidelity) (fidelity) [9]. (natural scene statistics), HVS(human visual system). 2. NRQA 2.1 (Sharpness degree) [10] (3).. 2.2 (blur) (blur metric) [10]., (local maximum), (local minimum). 1., BM (4). SD
BM 2.3 BIQI BIQI(blind image quality index), NSS [11]., JPEG, JPEG2000, white noise(wn), Gaussain Blur(Blur) Fast fading(ff). p i.. BIQI q i 5 5. 2.4 NIQE NIQE(natural image quality evaluator) [12]. BIQI NIQE [0,100], 0.. 3D DIBR 3D. 423DTV DSCQS(Double Stimulus Continuous Quality Scale) [14],[15]. 12. 7, 5. 3D 3D.,. (visual fatigue). 10. 5.0,. 1. 1. Table 1. Visual fatigue evaluation 5 (slight) 4 (moderate) 3 2 (severe) 1. 3 4 3D 3. 1. (Pearson Correlation) (Pearson's correlation coefficient). [-1,1].,. 0. ρ p.
2 : 3D (Saeed Mahmoudpour et al.: Comparison of Objective Metrics and 3D Evaluation Using Upsampled Depth Map) Cov(x,y), σ x σ y. 2. (Spearman Correlation). n 1, n, 1 n. ρ s. i f x i f x i f x. 3. (Kendall's Tau Correlation) (τ) [-1, +1],... MSR, Middlebury [13], HHI, GIST 16. RGB 2., (downsampling),. 3 4 Bowling, Ballet 7. 2. RGB. MSR, Middlebury, GIST HHI. Fig. 2. RGB images and related depth maps. (Provided by MSR, Middlebury, GIST and HHI)
3. Middlebury Bowling. (a) (b)~(h) 7. (BLU, BCU, BU, JBU, VBU, ABU, DTBU ) Fig. 3. Upsampled depth maps of Middlebury Bowling using seven upsampling methods. (a) is an original depth map and (b)~(h) are upsampled depth maps obtained by seven methods. (BLU= bilinear upsampling, BCU=bicubic upsampling, BU= bilateral upsampling, JBU=joint bilateral upsampling, VBU=variance-based bilateral upsampling, ABU=adaptive bilateral upsampling, and DTBU=distance transform-based bilateral upsampling) 4. MSR Ballet. (a) (b)~(h) 7. Fig. 4. Upsampled depth maps of MSR Ballet using seven upsampling methods. (a) is an original depth map and (b)~(h) are upsampled depth maps obtained by seven methods
2 : 3D (Saeed Mahmoudpour et al.: Comparison of Objective Metrics and 3D Evaluation Using Upsampled Depth Map) 5. RGB Fig. 5. Stereoscopic images in interlaced format generated by original RGB images and upsampled depth maps. 6. 5 Fig. 6. Close-ups of images in Fig. 5 5 DIBR. 5 6,.., Image PSNR, Edge PSNR, Non-edge PSNR, Sharpness Degree (SD), Blur Metric (BM), SSIM, VIF, BIQI, NIQE, 16 2.
2.. PSNR db. (BLU, BCU, BU, JBU, VBU, ABU, DTBU ) Table 2. Average results of objective QA metrics. (BLU= bilinear upsampling, BCU=bicubic upsampling, BU= bilateral upsampling, JBU=joint bilateral upsapling, VBU=variance-based bilateral upsampling, ABU=adaptive bilateral upsampling, and DTBU=distance transform-based bilateral upsampling) Depth map BLU BCU BU JBU VBU ABU DTBU Image PSNR Edge PSNR Non-edge PSNR Sharpness Degree 35.85 35.71 35.64 34.15 35.64 33.16 34.86 23.68 23.55 23.66 22.82 23.38 20.97 22.93 38.07 37.94 37.78 37.50 37.93 35.43 36.92 39.5 42.2 49.51 49.09 31.92 88.31 68.14 Blur Metric 8.48 11.38 10.29 10.87 10.51 9.00 9.89 SSIM 0.946 0.955 0.975 0.956 0.971 0.962 0.972 VIF 0.518 0.539 0.424 0.422 0.478 0.398 0.438 BIQI 57.8 66.34 61.11 32.81 41.94 29.15 72 NIQE 15.95 13.11 13.94 11.82 12.47 13.41 13.82 non-edge PSNR... (high frequency) [14]. SSIM PSNR. SSIM, 3.. VIF. BIQI. BIQI,. NIQI VIF. 3. SSIM Edge-PSNR. 4.,, Table 4. Correlations of Pearson, Spearman and Kendall for visual fatigue and objective metrics 3.. scale=[1,5]. Table 3. Average results of 3D subjective test at scale [1,5] visual Fatigue BLU BCU BU JBU VBU ABU DTBU 3.76 3.64 3.89 3.84 4.03 3.46 3.99 3. 3.76(BLU), 3.64(BCU), 3.89(BU), 3.84(JBF), 4.03(VBU), 3.46(ABU) 3.99(DTBU). 0 1,,, 4. 7. 4, 3 PSNR Edge-PSNR PSNR Pearson Spearman Kendall Sum PSNR 0.582 0.0357 0.0476 0.6653 Edge-PSNR 0.608 0.1429 0.1429 0.8938 Non-edge PSNR Sharpness Degree 0.554 0.0357 0.0476 0.6373-0.522-0.3214-0.1429-0.9863 Blur Metric 0.293 0.1429 0.0476 0.4835 SSIM 0.505 0.3571 0.1429 1.005 VIF 0.019 0.1071 0.1429 0.269 BIQI -0.339-0.3214-0.2381-0.8985 NIQE 0.132 0.1071 0.1429 0.382
2 : 3D (Saeed Mahmoudpour et al.: Comparison of Objective Metrics and 3D Evaluation Using Upsampled Depth Map) 7.,, Fig. 7. Bar diagram of correlations of Pearson, Spearman and Kendall. 3D 3D. 2D, 3D.., SSIM. SSIM. : IW-PSNR, NQM, UQI, MS-SSIM, IWSSIM, IFC, VIF, M-SVD, PSNR-HVSM, VSNR, MAD, FSIM, GSM, IGM, GMSD, VSQA.. (References) [1] A. B. Dehkordi, M. T. Pourazad, P. Nasiopoulos, "A study on the relationship between depth map quality and the overall 3D video quality of experience," IEEE 3DTV Conference, 2013. [2] C. Tomasi and R. Manduchi, "Bilateral Filtering for Gray and Color Image", In Proc. IEEE Int. Conf. on Computer Vision, pp.836-846(1998). [3] J. Kopf, M. F. Cohen, D. Lischinski and M. Uyttendaele, "Joint bilateral upsampling", ACM Trans. on Graphics, Vol. 26, No. 3, pp.1-6, 2007. [4] C. Pham, S. Ha, and J. Jeon, "A local variance-based bilateral filtering for artifact-free detail and edge-preserving smoothing", PSIVT, Part, LNCS 7088, 99.60-70, 2011. [5] D. Yeo, E. Haq, J. Kim, M. Baig, H. Shin, "Adaptive Bilateral Filtering for Noise Removal in Depth Upsampling", SoC Design Conf., pp. 36-39, 2010. [6] S. Jang, D. Lee, S. Kim, H. Choi, M. Kim, "Depth Map Upsampling with Improved Sharpness", Journal of Broadcast Engineering, Vol. 17, No. 6, pp. 933-944, 2012. [7] J. Gil, S. Mahmoudpour and M. Kim, "Analysis of relationship between objective performance measurement and 3D visual discomfort in depth map upsampling", Journal of Broadcast Engineering, Vol. 19, No. 1, pp. 31-43, Jan. 2014. [8] Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity", IEEE Trans. On Image Processing, Vol. 13, No. 4, 2004. [9] H. R. Sheikh and A. C. Bovik, "Image information and image quality", IEEE Trans. On Image Processing, Vol. 12, No. 2, pp. 430-444, 2006. [10] P. Marziliano, F. Dufaux, S. Winkler, and T. Ebrahimi, "Perceptual blur and ringing metrics:applicaation to JPEG2000", Int. Workshop on Multimedia Signal Processing, pp. 403-408, 2008. [11] A. K. Moorthy and A. Bovik, "A two-step framework for constructing
blind image quality assessment", IEEE Signal Processing Letters, Vol. 17, No. 5, pp. 513-516, 2010 [12] A. Mittal, R. Soundararajan, and A. C. Bovik, "Making a completely blind' image quality analyzer", IEEE Signal Processing Letters, Vol. 20, No. 3, pp. 209-212, 2013. [13] D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms", Int. J. of Computer Vision, Vol. 47, No.1-3, pp. 7-42, 2002. [14] E. Lee, H. Heo, and K. Park, "The comparative measurements of eye strain caused by 2D and 3D displays", IEEE Trans. on Consumer Electronics, Vol. 56, No. 3, pp. 1677-1683, 2010. [15] D. Kim and K. Sohn, "Visual fatigue prediction for stereoscopic image", IEEE Trans. Circuits Syst. Video Technol., Vol. 21, No. 3, pp. 231-236, 2011. - 2007 9 : B.S degree, Islamic Azad University, Iran - 2011 9 : M.S degree, Islamic Azad University, Iran - 2013 3 ~ : Ph.D student, Kangwon National University, Repubic of Korea - : 3D image processing, 3D human factor, human detection - 1979 : - 1981 : - 1995 : - 1984 ~ 1996 : ETRI / - 2009 ~ 2011 : IT - 1996 ~ : - ORCID : http://orcid.org/0000-0002-8340-4195 - :, 3D, - 1983 : - 1986 : University of Washington, Seattle - 1992 : University of Washington, Seattle - 1992 ~ 1998 : - 1998 ~ : - ORCID : http://orcid.org/0000-0002-4702-8276 - : 3D,,