(JBE Vol. 20, No. 5, September 2015) (Regular Paper) 205, 2015 9 (JBE Vol. 20, No. 5, September 2015) http://dx.doi.org/10.5909/jbe.2015.20.5.756 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) HDR SDR a), a), a), a), a), b) Layered Coding Method for Scalable Coding of HDR and SDR videos Jeongyun Lim a), Yong-Jo Ahn a), Woong Lim a), Seanae Park a), Donggyu Sim a), and Jung-Won Kang b) SHVC (Scalable High Efficiency Video Coding) HDR (High Dynamic Range) SDR (Standard Dynamic Range). SHVC SDR,, HDR., (Global inverse tone mapping) HDR. SHVC SHM7.0 43%, 76.3% BD-Bitrate, 15.7%, 31% BD-Bitrate. Abstract In this paper, we propose a scalable coding method for high dynamic range (HDR) and standard dynamic range (SDR) videos based on Scalable High Efficiency Video Coding (SHVC). The proposed method has multi-layer coding architecture that consists of base layer for SDR videos and enhancement layer for HDR videos to support the backward compatibility with legacy codec and display devices. Also, to improve coding efficiency of enhancement layers, a global inverse tone mapping is applied to the reconstructed SDR video and the compensated frames are referred for coding of the enhancement layer. The proposed method is found to achieve BD-Rate gain of 43.0% on average (maximum 76.3%) for the enhancement layer and 15.7% on average (maximum 31%) for dual-layer against the SHM 7.0 reference software. Keyword : HEVC, SHVC, HDR, Tone mapping, Inverse tone mapping a) (Dept. of Computer Engineering, Kwangwoon University) b) (ETRI) Corresponding Author : (Donggyu Sim) E-mail: dgsim@kw.ac.kr Tel: +82-2-941-6470 ORCID: http://orcid.org/0000-0002-2794-9932 [B0101-15-295, UHD // ] [NRF- 2014R1A2A1A11052210]. Manuscript received August 18, 2015; Revised September 24, 2015; Accepted September 24, 2015.
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos).,. ISO/IEC MPEG (Moving Picture Expert Group) ITU-T VCEG (Video Coding Expert Group) 2010 JCT-VC (Joint Collaborative Team on Video Coding) HEVC (High Efficiency Video Coding). 2013 1 HEVC version 1, H.264/ AVC 50%, 40% [1~2]., JCT-VC HEVC version 1, HEVC RExt (Range Extension), SHVC (HEVC Scalable Extension), MV-HEVC (HEVC Multi-view Extension) HEVC version 2. SHVC, [3][4].,, HDR(High Dynamic Range)/WCG (Wide Color Gamut). [5]. MPEG 2014 7 HDR/WCG, 2015 2 HDR/WCG CfE (Call for Evidence) [6]. CfE HEVC SHVC HDR/WCG,, (backward compatibility) 3 (category). 2015 6 MPEG JCT-VC CfE HDR/WCG. CfE Category 1, 2, 3. Category 1 HEVC Main 10 Profile HDR., Category 1, HDR/WCG. [7~10]. Category 2 HEVC Main 10 Profile SHVC Main 10 Profile SDR HDR (Dual-layer coding), SDR HEVC/SHVC Main 10 Profile,. Category 3 HEVC Main 10 Profile SHVC Main 10 Profile,, HDR, -. Category 2 Category 3 [11~15]. SHVC Main 10 Profile, HDR SDR. 'Category 3' - HDR SHVC,, SDR HDR. (base layer) (enhancement layer) SDR HDR.
(JBE Vol. 20, No. 5, September 2015) (Inter-layer prediction, ILP) [4] (Inverse tone mapping) SDR. SDR HDR SDR HDR.. HDR HDR (tone mapping),, HEVC HDR. HDR/SDR,,.. HDR 2 HDR., 2.1 HDR,. 2.2 HDR. 1. HDR / (light intensity), nits f-stops [16]., f-stops, (contrast ratio) 1 [17]. HDR 16 f-stops, HDR HDR [18~22]., HDR TIFF [23], OpenEXR [24] (physical scene luminance). OpenEXR HDR ILM (Industrial Light & Magic), IEEE 754-2008 16-bit 32-bit floating point [24]. OpenEXR 8-bit 7~10 f-stops 16-bit floating point, 30 f-stops [24]. HDR SDR,. HDR SDR, SDR HDR (conversion). 1. f-stop Table 1. Dynamic range depending on f-stop Dynamic range Minimum f-stop Maximum f-stop Contrast ratio SDR (Standard Dynamic Range) EDR (Enhanced Dynamic Range) HDR (High Dynamic Range) 0 f-stops 10 f-stops 2 10 : 1 10 f-stops 16 f-stops 2 14 : 1 16 f-stops - 2 16 : 1
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos), HDR SDR. HDR SDR TMO(Tone Mapping Operator) (global tone mapping) (local tone mapping). HDR (monotonously increasing tone mapping curve) HDR SDR [25~27]. SDR HDR HDR [28~30]. (edge) [31], HVS(Human Visual System) HDR SDR [32]. 1 3. Mapping operation, HDR (Luminance value) Lin SDR 0 1., Gamma encoding,, Mapping operation HDR Luma Lin Lg. Quantization floating point [0;2n-1](n ) (integer code value) HDR Luma Lout [33]. SDR HDR., SDR ITMO (Inverse Tone Mapping Operator) 2. SDR HDR TMO, TMO, TMO ITMO. SDR HDR. EO(Expand Operator). SDR HDR TMO, HDR, SDR [34~35]. 2. HDR HDR. OpenEXR, TIFF HDR [6]. HEVC HDR, HDR. 2 HEVC HDR., HDR HEVC 16bit floating point 4:4:4 RGB HDR 10bit 4:2:0 Y'CbCr. HDR HEVC, HEVC., 10bit 4:2:0 Y'CbCr HDR 1. 3 Fig. 1. Three steps to perform a tone mapping operation
(JBE Vol. 20, No. 5, September 2015) 2. HEVC HDR Fig. 2. HDR Video coding architecture based on HEVC HDR. HDR. 2 Coding TF (Transfer Function) HDR RGB, PQ-TF(Perceptual Quantizer) [36~37] (1), HDR R, G, B R', G', B'. maxmin maxmin maxmin (1) PQ-TF [36~37] EOTF(Electro-Optical Transfer Function), CRT (Gamma correction). PQ-TF HDR, HDR R, G, B R, G, B. Color conversion R'G'B floating Color gamut Container gamut Y'CbCr floating point, Color gamut BT.709 [38], (2). Color gamut BT.2020 [39], (3) Color conversion. (2) (3) R', G', B' PQ-TF HDR, HEVC Y'CbCr. Quantization Y'CbCr floating point bit-depth, Sampling conversion 4:4:4 Y'CbCr 4:2:0 [6]. 2 HDR HEVC. HDR HDR
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos). HDR HDR HDR HEVC [7~10]. SDR. HDR HDR HDR, SDR. SDR HDR SDR.. HDR, HDR/ SDR. SDR, HDR.. HDR/SDR,. 1. HDR/SDR, HDR/SDR. SDR HDR,, 3. Fig. 3. Block diagram of proposed scalable coding architecture
(JBE Vol. 20, No. 5, September 2015) SDR. 3 SHVC SHM7.0 HDR/SDR. 10bit YCbCr SDR, 10bit Y'CbCr HDR., SDR SDR,. SDR, (4). SDR, SDR.,. SDR,, (flickering artifacts) [40]. = (4) L'SDR SDR, L'HDR SDR SDR. SDR, HDR SDR.,,,., i j, 0 8. SDR/HDR., 2. SDR SDR, HDR., SDR. HDR, SDR. SDR. 4 SDR. SDR HDR 16bit RGB 4:4:4 OpenEXR 4. SDR Fig. 4. Process for SDR generation
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos) 10bit Y CbCr 4:2:0., HDR SDR. 4 Conversion HDR, 2, HDR PQ-TF, Color conversion, Quantization Sampling conversion 10bit YUV 4:2:0., HDR SDR.. HDR SDR Luma SDR.,,, SDR. CfE [6] HDR SDR POC (Picture order count) Luma. 5(a) HDR Balloon- Festival [6] SDR, X Y HDR SDR Luma [0;2n](n; bit-depth)., HDR Luma SDR Luma., HDR 170~200 SDR 80~120. HDR Luma 500 SDR 430~510. HDR, (5). (5),,., i j 0 8. 5(b) 5(a) HDR/SDR. HDR Luma 8 ([R0;R1], [R1;R2],..., [R7;R8]), HDR Luma avg(ri) SDR Luma ([R 0;R 1], [R 1;R 2],..., [R 7;R 8]) avg(r'i)., 5. HDR SDR (a) (b) Fig. 5. The pixel mapping relation between HDR video and tone mapped SDR video (a) and the normalized global tone mapping curve based on the corresponding relation (b)
(JBE Vol. 20, No. 5, September 2015) HDR/SDR (6), (5). (6) pi, [Ri-1;Ri] pi-1 pi., SDR,., SDR,. 2,.., SHVC SHM 7.0. SHM7.0 SDR HDR, 2. Table 2. Test environment and encoding configuration option Encoder Parameter Profile Bit Depth Input Sequence Rate Control CGS (Color Gamut Scalability) Configuration Option SHVC Main 10 Profile Basement Layer Enhancement Layer Basement Layer Enhancement Layer Off Off 10 bit 10 bit SDR video HDR Video. 2., QP [6] CfE, (random access). 3 HDR. 1920 1080p, SDR HDR HDR SDR. 4 QP. CfE[6] QP, 2 dqp(deltaqp; dqp1 = -2, dqp2 = 0) QP., SDR 3. HDR Table 3. HDR test sequences Class Sequence name Seq number Fps Frames FireEater2Clip4000r1 S00 25 200 A Tibul2Clip4000r1 S01 30 240 Market3Clip4000r2 S02 50 400 G BalloonFestival S03 24 240
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos) 4. QP Table 4. QPs for Base layer and Enhancement layer Base layer SDR QP_BL = {QP1, QP2, QP3, QP4} QP_BL = {QP1, QP2, QP3, QP4} Enhancement layer HDR QP_EL1 = {QP1+dQP1, QP2+dQP1, QP3+dQP1, QP4+dQP1} QP_EL1 = {QP1+dQP1, QP2+dQP1, QP3+dQP1, QP4+dQP1}, HDR Luma Chroma HDR. 6 6. Fig. 6. Example for referencing in the enhancement layer with inverse tone mapping to reconstructed picture(base layer),, S03 SDR, 47.8%., SDR HDR,,. SHM 7.0, 5., 76.3% BD-Bitrate, 4, dqp1 dqp2 43.0% BD- Bitrate., S02 S03 56% BD-Bitrate, HDR Luma ([0;210]) 5. BD-Bitrate Table 5. BD-Bitrate result for the proposed method vs. anchor on the enhancement layer Class HDR seq dqp A G S00 S01 S02 S03 BD-Bitrate Y U V dqp1-12.8% -9.0% -12.5% dqp2-17.8% -8.7% -13.9% dqp1-19.4% -27.4% -19.6% dqp2-32.3% -32.9% -20.8% dqp1-56.0% -50.3% -50.8% dqp2-68.9% -59.4% -60.1% dqp1-60.4% -51.9% -52.9% dqp2-76.3% -65.0% -66.5% Average -43.0% -38.1% -37.1%
(JBE Vol. 20, No. 5, September 2015) 6. BD-Bitrate Table 6. BD-Bitrate gains for the proposed method vs. anchor dual-layer (HDR and SDR) Class HDR seq dqp A G S00 S01 S02 S03 BD-rate Y U V Encoding Time dqp1-3.5% 1.1% -2.9% 98.53% dqp2-4.8% 7.5% 1.0% 98.52% dqp1-4.4% -14.0% -4.8% 97.51% dqp2-7.0% -8.5% 13.2% 96.98% dqp1-22.7% -14.0% -14.8% 97.60% dqp2-28.0% -9.6% -10.7% 97.60% dqp1-24.1% -11.1% -12.7% 97.61% dqp2-31.0% -4.6% -7.9% 97.60% Average -15.7% -6.7% -5.0% 97.74%.,. S01 17.8 BD-Bitrate S02 S03. Luma,,. S00, HDR SDR,,. 6, SHM 7.0. 4 dqp1 dqp2 QP 15.7%, 31% BD-Bitrate.. HDR, SDR HDR/SDR. SDR. SHM 7.0 76.3% BD- Rate, 15.7%, 31% BD-Rate. Color gamut. (References) [1],, HEVC,, 2014. Donggyu Sim, Hyunho Jo, "Understanding of HEVC Standard Technology," Hongrung publishing, 2014. [2] Yong-Jo Ahn, Tae-Jin Hwang, Dong-Gyu Sim and Woo-Jin Han, "Implementation of fast HEVC encoder based on SIMD and data-level parallelism," EURASIP Journal on Image and Video Processing 2014, 2014:16, Mar. 2014. [3],,,, Scalable HEVC, ( ), 30, 9, 49-57, 2013 8. Kyeong-Hye Kim, Hyunho Jo, Donggyu Sim, Youngmin Jang, "Trend of Scalable HEVC Standard technology," Information and Communications, vol. 30, no. 9, pp. 49-57, Aug. 2013. [4] J. Chen, J. Boyce, Y. Ye, M. M. Hannuksela, G. J. Sullivan and Y. -K. Wang, "High efficiency video coding (HEVC) scalable extension Draft 6," Q-1008_v5, 17th JCT-VC meeting, Mar. 2014, Valencia. [5] MovieLabs Specification for Next Generation Video: http://www. movielabs.com/ngvideo
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos) [6] A. Luthra, E. Francois and W. Husak, "Call for Evidence (CfE) for HDR and WCG Video Coding," N-15083, February. 2015, Geneva. [7] D. Baylon, Z. Gu, A. Luthra, K. Minoo, P. Yin, F. Pu, T. Lu, T. Chen, W. Husak, Y. He, L. Kerofsky, Y. Ye, B. Yi, "Response to Call for Evidence for HDR and WCG Video Coding: Arris, Dolby and InterDigital," M-36264, 112th MPEG meeting, June. 2015, Warsaw. [8] S. Lasserre, F. Le Leannec, E. Francois, T. Poirier, "Technicolor's response to CfE for HDR and WCG (Category 1)," M-36263, 112th MPEG meeting, June. 2015, Warsaw. [9] Rocco Goris, Robert Brondijk, Rene van der Vleuten, "Philips response to CfE for HDR and WCG," M-36266, 112th MPEG meeting, June. 2015, Warsaw. [10] Jonathan Hatchett, Kurt Debattista, Joshua McNamee, Alan Chalmers, "gohdr10+: A Category 1 HDR video compression method," M-36261, 112th MPEG meeting, June. 2015, Warsaw. [11] Jeongyun Lim, Woong Lim, Yong-jo Ahn, Seanea Park, Donggyu Sim, "Reporting: Coding efficiency of the scalable coding for HDR and SDR video using inverse tone mapping on SHVC," M-36463, 112th MPEG meeting June. 2015, Warsaw. [12] Jacob Strom, Jonatan Samuelsson, Martin Pettersson, Kenneth Andersson, Per Wennersten, Rickard Sjorberg, "Ericsson's response to CfE for HDR and WCG," M-36184, 112th MPEG meeting, June. 2015, Warsaw. [13] A. Cotton, T. Borer, M. Pindoria, S. Thompson, M. Naccari, S. Schwarz, M. Mark, "BBC's response to CfE for HDR Video Coding (Category 3a)," M-36249, 112th MPEG meeting, June. 2015, Warsaw. [14] A.K Ramasubramonian, J. Sole, M. Karczewicz, S. Lee, D. Rusanovskyy, D. Bugdayci, X. Li, "Dual layer non-normative (category 3b) response to the Call for Evideonce on HDR/WCG," M-36280, 112th MPEG meeting, June. 2015, Warsaw. [15] D. Rusanovskyy, S. Lee, D. Bugdayci, A. Ramsubramonian, J. Sole, M. Karczewicz, A.M. Tourapis, Y. Su, D. Singer, C. Fogg, A. Duenas, F. Bossen, "Single layer non-normative (category 3a)NCL and CL responses to the Call for Evidence on HDR/WCG," M-36256, 112th MPEG meeting, June. 2015, Warsaw. [16] ITU-R BT.2246-2 (2012), "The present state of ultra-high definition television," http://www.itu.int/dms_pub/itu-r/opb/rep/r-rep-bt.2246-2-2012-pdf-e.pdf [17] Ajay Luthra, Edouard Francois, Walt Husak, "Draft Requirements and Explorations for HDR and WCG Content Distribution," M-36436, 110th MPEG meeting, Oct. 2014, Strasbourg. [18] Banterle, Francesco, et al. Advanced high dynamic range imaging: theory and practice. CRC Press, 2011. [19] P.E. Devec and J. Malik, "Recovering high dynamic range radiance maps from photographs," Proc. ACM SIGGRAPH'97, pp. 369-378, 1997. [20] T. Mitsunaga and S. K. Nayar, "High dynamic range imaging: Spatially varying pixel exposures," Proc. CVRP' 2000, vol. 1, pp. 472-479, 2000. [21] S. Mann and R. W. Picard, "On being 'undigital' with digital cameras: extending dynamic range by combining differently exposed pictures," IS&T's 48th Annual Conference, Society for Imaging Science and Yechnology, Washington D. C., pp. 422-428, 1995. [21] S.B. Kang, M. Uyttendale, S. Winder and R. Szeliski, "High dynamic range video," ACM Transactions on Graphics, vol.22, no. 3, pp. 319-325, July. 2003. [22] R. Mantiuk, G. Krawczyk, K. Myszkowski, and H-P Seidel, "Perception-motivated High Dynamic Range Video Encoding," Proc. of SIGGRAPH'2004, pp. 733-741, 2004. [23] Larson, Gregory Ward. "LogLuv encoding for full-gamut, high-dynamic range images," Journal of Graphics Tools 3.1, pp. 15-31, 1998. [24] Kainz, Florian, and Rod Bogart, "Technical introduction to OpenEXR," Industrial light and magic, Nov. 2009 [25] J. Tumbin, and H. Rushmeier, "Tone reproduction for realistic images," Computer Graphics and Application IEEE, 1993. [26] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda, "Photographic tone reproduction for digital images," ACM Trans. Graph, pp. 267-276, 2002. [27] R. Mantiuk, S. Daly and L. Kerofsky, "Display adaptive tone maping," ACM Transactions on Graphics, 27(3):1, 2008. [28] K. Chiu, M. Herf, P. Shirley, S. Swamy, K. Zimmerman, "Spatially nonuniform scaling functions for high contrast images," in Graphics Interface, pp. 245-245, 1993. [29] S. N. Pattanaik, J. A. Ferwerda, M. D. Fairchild, and D. P. Greenberg, "A multiscale model of adaptation and spatial vision for realistic image display," In Proceedings of model of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH' 98, pp. 287-298, 1998. [30] Y. Li, L. Sharan, and H. E. Adelson, "Compressing and companding high dynamic range images with subband architectures," ACM Transactions on Graphics, 24(3):836, 2005. [31] Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski, "Edge-preserving decompositions for multi-scale tone and detail manipulation," In ACM SIGGRAPH 2008 papers on - SIGGRAPH'08, pp. 1, 2008. [32] P. Ledda, A. Chalmers, T. Troscianko, and H. Seetzen, "Evaluation of tone mapping operators using a high dynamic range display," In ACM SIGGRAPH 2005 papers on - SIGGRAPH'05, pp. 640-648, 2005. [33] Boitard, Ronan, et al. "Motion-guided quantization for video tone mapping," Multimedia and Expo (ICME), 2014 IEEE International Conference on. IEEE, 2014. [34] Akyuz, Ahmet Oguz, et al, "Do HDR displays support LDR content?: a psychophysical evaluation," In ACM Transactions of Graphics (TOG), Vol. 26, p. 38, 2007. [35] Banterle, Francesco, et al. "Inverse tone mapping," Processings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia, ACM, pp. 349-356, 2006. [36] S. Miller, M. Nezamabadi, and S. Daly, "Perceptual Signal Coding for More Efficient Usage of Bit Codes," SMPTE Motion Imaging Journal, vol. 122, no. 4, pp. 52-59, May-June 2013. [37] Society of Motion Picture and Television Engineers, FCD ST 2084 (2014), Electro-Optical Transfer Function for High Dynamic Range
(JBE Vol. 20, No. 5, September 2015) Reference Display. [38] ITU-R Recommendation BT.709-5, "Parameter values for the HDTV standards for production and international programme exchange," 2002. [39] ITU-R Recommendation BT.2020-1, "Parameter values for ultra-high definition television systems for production and international programme exchange," 2014. [40] Boitard, Ronan, et al, "Survey of temporal brightness artifacts in video tone mapping," HDRi2014-Second International Conference and SME Workshop on HDR imaging. 2014. [41] Vadim, Yong He, "Common SHM test conditions and software reference configurations," JCTVC-Q1009, Valencia, Es, 27 March - 4 April 2014. - 2014 2 : - 2014 3 ~ : - ORCID : orcid.org/0000-0002-7035-8572 - :,, - 2010 2 : - 2012 2 : - 2012 3 ~ : - ORCID : http://orcid.org/0000-0002-0012-0905 - :, - 2008 2 : - 2010 2 : - 2010 3 ~ : - ORCID : http://orcid.org/0000-0002-7095-9116 - :,, - 2004 2 : - 2006 2 : - 2011 8 : - 2014 11 ~ : - ORCID : http://orcid.org/0000-0002-2057-2212 - :,,
5 : HDR SDR (Jeongyun Lim et al.: Layered Coding Method for Scalable Coding of HDR and SDR videos) - 1993 2 : - 1995 2 : - 1999 2 : - 1999 3 ~ 2000 8 : - 2000 9 ~ 2002 3 : - 2002 4 ~ 2005 2 : University of Washington Senior research engineer - 2005 3 ~ : - ORCID : http://orcid.org/0000-0002-2794-9932 - :,, - 1993 2 : - 1995 2 : - 2003 8 : Georgia Institute of Technology Department of ECE - 2003 10 ~ : (ETRI) - 2007 3 ~ : - ORCID : orcid.org/0000-0002-7722-6206 - :,,,,