2 : : 3D-HEVC (Young Su Heo et al.: A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode) (Special Paper) 20 4, 2015 7 (JBE Vol. 20, No. 4, July 2015) http://dx.doi.org/10.5909/jbe.2015.20.4.509 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) : 3D-HEVC a), b), a) A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode Young Su Heo a), Gun Bang b), and Gwang Hoon Park a), HEVC (High Efficiency Video Coding), 3D-HEVC.,. 3D-HEVC HEVC, HEVC., HEVC, 3D-HEVC., 51.4%,. Abstract A design for the merge mode of three dimensional High Efficiency Video Coding (3D-HEVC) is proposed in this paper. The proposed design can reduce the implementation complexity by removing the duplicated modules of the HEVC. For the extension codec, the implementation complexity is as crucial as coding efficiency, meaning if possible, extension codec needs to be easily implemented through by reusing the design of the legacy codec as-is. However, the existing merging process of 3D-HEVC had been built-in integrated in the inside of the HEVC merging process. Thus the duplicated merging process of HEVC had to be fully re-implemented in the 3D-HEVC. Consequently the implementation complexity of the extension codec was very high. The proposed 3D-HEVC merge mode is divided into following two stages; the process to reuse the HEVC modules without any modification; and the reprocessing process for newly added and modified merging modules in 3D-HEVC. By applying the proposed method, the re-implemented HEVC modules, which accounted for 51.4% of 3D-HEVC merge mode confirmed through the operational analysis of algorithm, can be eliminated, while maintaining the same coding efficiency and computational complexity. Keyword : HEVC, 3D-HEVC, Merge mode
(JBE Vol. 20, No. 4, July 2015). (ultra-high definition: UHD) 3D., Moving Picture Experts Group (MPEG) Video Coding Experts Group (VCEG) 2D 3D, (multi-view video coding: MVC)., 2D., (POC) [1]., (disparity compensated prediction: DCP). DCP, POC,.,,. (Syntax),, (Slice header) (High level syntax). [1]. DCP 1996 MPEG-2 Multi-view Profile [2] (temporal scalability coding tool) [3],., Advanced Video Coding (AVC)/H.264 [4] Amendment Multi-view Video Coding (MVC) [5], High Efficiency Video Coding (HEVC)/H.265 [6] MVC HEVC extension to multiple views (MV- HEVC) [7]. DCP, (frame-compatible format). (extension codec),,.,,,. MVC.,., MVC macro-block (MB) [8]-[13] Joint Multi-view Video Model (JMVM) [14], MB MVC [15], MVC [5]., (S/W) (H/W)., a) (Media Lab., Dept. of Computer Engineering, Kyung Hee University) b) (Electronics and Telecommunications Research Institute) Corresponding Author : (Gwang Hoon Park) E-mail: ghpark@khu.ac.kr Tel: +82-31-201-3680 ORCID: http://orcid.org/0000-0001-7133-8285 2015 ( ) (No. B0101-15-295, UHD / / ). Manuscript received May 18, 2015; Revised July 6, 2015; Accepted July 6, 2015.
2 : : 3D-HEVC (Young Su Heo et al.: A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode),.,,., H/W. H/W,, (killer application).,. MPEG VCEG Joint Collaborative Team on 3D Video (JCT-3V), 3D- AVC [16], 2015 3D-HEVC [1] 3D,..,, MVC MV-HEVC, 25% [17].,,. MVC MV-HEVC,. 3D-HEVC [18], Coding Unit(CU) HEVC. S/W H/W, HEVC CU. 3D-HEVC [19],, HEVC. HEVC [20], 3D-HEVC. 3D-HEVC [18], HEVC,,. 3D-HEVC, HEVC., HEVC, 3D-HEVC.. II. III 3D-HEVC, IV. V. VI... 1, (base layer) (enhancement layer).,
(JBE Vol. 20, No. 4, July 2015). 3D-HEVC, (base view), (dependent view).,. 1 (a), (built-in integration).,. 1 (a).,,. 3D-HEVC [18]..., 1 (b).,.,.,, Layer information Base layer encoder Input video sequences Enhancement layer encoder M U X Bitstream Base layer encoder (a) Layer information Input video sequences Base layer encoder Refined process for extension M U X Bitstream (b) 1.. (a), (b) Fig. 1. Illustration of the design of enhancement layer codec. (a) Integrating inside the existing module, (b) Reusable design
2 : : 3D-HEVC (Young Su Heo et al.: A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode)..,.,.,,.,.. 3D-HEVC, PU (Prediction Unit), PU. (merge candidate list), - (rate-distortion).,, Advanced motion vector prediction(amvp) [21].,, PU [20].,,. HEVC PU (A0), (A1), (B0), (B1), (B2) 5 (spatial candidate), PU (temporal candidate: Col) [20]. /, [22]-[23]. /,., L0, L1 (combined bi-predictive candidate), 0 (zero candidate) [24]. 3D-HEVC, 3D. 3D.. (T):., PU PU [19]. (IvDC):, DV(disparity vector) [25]. (IvMC):., POC [26]. (VSP): VSP (warping) PU [28]-[29]., PU VSP. IvMCShift, IvDCShift: DV. IvMCShift 1 DV IvMC. IvDCShift, IvMCShift IvMC,, DV 1 [27]., T Zero [18]. 3D-HEVC 2. NVSP, VSP
(JBE Vol. 20, No. 4, July 2015) VSP., {T, IvMC, A1, B1, B0, IvDC, VSP, A0, B2, IvMCShift, IvDCShift, Temporal(Col), Combined bi-predictive, Zero}. N VSP T IvMC A 1 B 1 B 0. Step. step step (skip).. II, 3D- IvDC VSP A 0 B 2 IvMCShift IvDCShift HEVC HEVC [31]-[32]. 3. Temporal (Col) Combined Bi-predictive Zero Merge candidate list Stage 1 2. 3D-HEVC Fig. 2. Integrated design of merge candidate list construction in the exisiting 3D-HEVC Derivation of HEVC candidates Base merge candidate list construction Base merge candidate list Step 1: VSP. Step 2: T. Step 3: IvMC T,. Step 4: A 1 IvMC. Step 5: B 1 IvMC. Step 6: B 0. Step 7: IvDC A 1 B 1,. Step 8: VSP. Step 9: A 0. Step 10: B 2. Step 11: IvMC, IvMC IvMCShift IvMC, IvMCShift. Step 12: IvMCShift IvDCShift. Step 13:. Step 14: PU B,. Step 15:. Derivation of 3D candidates Stage 2 Reprocessing of merge candidate list Extended merge candidate list 3. 3D-HEVC Fig. 3. Proposed reusable design of merge mode in 3D-HEVC / HEVC 1 : (base merge candidate list construction). 1,,, HEVC, 3D-HEVC. 1 HEVC.
2 : : 3D-HEVC (Young Su Heo et al.: A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode) 2, 1 3D HEVC 3D (extended merge candidate list). (reprocessing). 1. HEVC, 1. 3D-HEVC. HEVC, A1,B1,B0,A0,B2,, 4. A1,B1,B0,A0 B2., 3D- HEVC 5 6, 4 [18]., B2. HEVC step, 3D-HEVC Step 3. Step 1: PU, B 2. B 2. Step 2: B 2, B 2 A 1 B 1, B 2. Step 3: B 2, A 1,B 1,B 0,B 0, B 2. 3D-HEVC, HEVC Step 3,., IvMC A1, B2, B2 0.17% ( ). B2, B2 HEVC. 3D-HEVC VSP. VSP flag True PU, [30]., VSP 2, 1. 2. 3D-HEVC, HEVC 3D. HEVC 3D., (mutual dependency)., VSP True [18]. HEVC VSP (causal dependency).,, 3D-HEVC 5 6, 12 20 8., HEVC, 1. 3D-HEVC 6 HEVC, 12, MCL. HEVC,. 3D-HEVC, HEVC [32]-[33]., 1
(JBE Vol. 20, No. 4, July 2015) 3D-HEVC. 3., 1 HEVC. 4, 1 HEVC ( 4 (a)), 2 ( 4 (b)). 2, 1 3D. VSP (VSP), 3D T, IvMC, IvDC, VSP, IvMCShift, IvDCShift., 3D 2. Step,. step step. A 1 B 1 B 0 Temporal Base merge candidate list VSP IvMCShift Combined Bi-predictive N VSP (a) IvDCShift (b) A 0 B 2 Zero Base merge candidate list T IvMC IvDC Extended merge candidate list 4.. (a) HEVC, (b) 3D-HEVC Fig. 4. Block diagram of proposed reusable design for merge candidate list construction. (a) Merge candidate list construction process in HEVC, (b) Proposed refined process for reprocessing of 3D-HEVC merge candidate list Step 1: VSP. Step 2: T. Step 3: IvMC. IvMC A 1 B 1. Step 4: IvDC A 1 B 1,. Step 5: VSP. Step 6: IvMC, IvMC IvMCShift IvMC, IvMCShift. Step 7: IvMCShift IvDCShift.. 3D-HEVC S/W HTM-8.2 [34], HTM-8.2. 1, HEVC, 2 3D., JCT-3V 3D-HEVC Common Test Condition (CTC) [35]. Bjontegaard-Delta rate (BD-rate) [36]. 3D-HEVC, HEVC,,,.,,., ( /, T, IvMC, IvMCShift),.
2 : : 3D-HEVC (Young Su Heo et al.: A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode).,.,,, 1.,, 12 7 41.7%., 6, 3D T, IvMC, IvMCShift 1, 9., 9 4, 55.6%., 9 T,, 8 3, 62.5%. 10, 3D-HEVC, 5 50.0%. 1 HEVC, 3D-HEVC 20 0 100%. 2., (views 1-3 ) BD-rate 0.0%, 1. Table 1. Reduction of major operations on proposed merging process Number of operations Before refinement After refinement Refinement rate Reduction rate Availability determination 12 7 58.3% 41.7% Total motion data fetch 9 4 44.4% 55.6% Motion data fetch for dependent texture 8 3 37.5% 62.5% Redundancy check 10 5 50.0% 50.0% Calculations of combined bi-predictive candidate 20 0 eliminated 100.0% 2. BD-rate Table 2. BD-rate results for the proposed method Sequences Inter-view video View 0 View 1 View 2 Total bitrate Synthesized bitrate Balloons 0.0% -0.1% 0.0% 0.0% 0.0% Kendo 0.0% 0.1% 0.1% 0.1% 0.1% Newspaper_CC 0.0% -0.1% 0.0% 0.0% 0.0% GT_Fly 0.0% -0.6% -0.2% 0.0% 0.0% Poznan_Hall2 0.0% -0.2% 0.1% 0.0% 0.0% Poznan_Street 0.0% -0.1% -0.2% 0.0% 0.0% Undo_Dancer 0.0% -0.1% -0.1% 0.0% 0.0% Shark 0.0% -0.1% 0.0% 0.0% 0.0% Average 0.0% -0.1% 0.0% 0.0% 0.0%
(JBE Vol. 20, No. 4, July 2015). 3.,. 3. Table 3. Encoding and decoding time results Sequences Encoding time Deccoding time Balloons 0.0% 0.0% Kendo 0.0% 0.0% Newspaper_CC 0.0% -0.1% GT_Fly 0.0% 0.1% Poznan_Hall2 0.0% 0.1% Poznan_Street 0.0% -0.1% Undo_Dancer 0.0% -0.1% Shark 0.0% -0.2% Average 0.0% 0.0%,, 51.4% (, 41.7%, 62.5%, 50.0% ) S/W H/W.. 3D-HEVC., 3D-HEVC HEVC, HEVC., HEVC, 3D-HEVC,.,, HEVC,.,, 51.4%, S/W H/W. 3D- HEVC [37]-[38], 3D-HEVC FDAM 4 [30] HTM-14 [39]. (References) [1] G. J. Sullivan, J. M. Boyce, Y. Chen, J. R. Ohm, C. A. Segall, and A. Vetro, Standardized Extensions of High Efficiency Video Coding (HEVC), IEEE Journal of Selected Topics in Signal Processing, vol. 7, no. 6, pp. 1001 1016, Dec. 2013. [2] B. L. Tseng and D. Anastassiou, Multiviewpoint video coding with MPEG-2 compatibility, IEEE Trans. Circuits Syst. Video Technol., vol. 6, no. 4, pp. 414 419, Aug. 1996. [3] A. Puri, R. V. Kollarits, and B. G. Haskell, Stereoscopic Video Compression Using Temporal Scalability, SPIE Conf. Visual Communications and Image Processing (VCIP' 95), vol. SPIE 2501, pp. 745 756, Apr. 1995. [4] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, Overview of the H.264/AVC video coding standard, IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 560 576, Jul. 2003. [5] A. Vetro, T. Wiegand, and G. J. Sullivan, Overview of the stereo and multiview video coding extensions of the H.264/AVC standard, Proc. IEEE, Special Issue 3D Media Displays, vol. 99, no. 4, pp. 626 642, Apr. 2011. [6] ISO/IEC JTC 1, Information technology High efficiency coding and media delivery in heterogeneous environments Part 2: High Efficiency Video Coding, ISO/IEC 23008-2:2013 (ver. 1), Nov. 2013. [7] M. Domański, T. Grajek, D. Karwowski, J. Konieczny, M. Kurc, A. Łuczak, R. Ratajczak, J. Siast, O. Stankiewicz, J. Stankowski, and K. Wegner, "Coding of Multiple Video+Depth Using HEVC Technology and Reduced Representations of Side Views and Depth Maps," in Proc. Picture Coding Symp., pp. 5 8, May 2012. [8] Y. L. Lee, J. H. Hur, Y. K. Lee, K. H. Han, S. H. Cho, N. H. Hur, J. W. Kim, J. H. Kim, P. Lin Lai, A. Ortega, Y. Su, P. Yin, and C. Gomila, CE11: Illumination compensation, JVT-U052 Document JVT,
2 : : 3D-HEVC (Young Su Heo et al.: A Design for Extension Codec based on Legacy Codec: 3D-HEVC Merge Mode) Hangzhou, China, Oct. 2006. [9] J. H. Hur, S. Cho, and Y. L. Lee, Adaptive local illumination change compensation method for H.264/AVC-based multiview video coding, IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1496 1505, Nov. 2007. [10] W. S. Shim, M. W. Park, G. H. Park, D. Y. Suh, H. S. Song, Y. H. Moon, and J. B. Choi, CE5 results- Joint proposal for MVC deblocking, JVT-W024 Document JVT, San Jose, CA, USA, Apr. 2007. [11] G. H. Park, M. W. Park, S. C. Lim, W. S. Shim, and Y. L. Lee, Deblocking Filtering for Illumination Compensation in Multiview Video Coding, IEEE Trans. Circuits and Systems for Video Tech., vol. 18, no. 10, pp. 1457 1461, Oct. 2008. [12] H. S. Koo, Y. J. Jeon, and B. M. Jeon, MVC motion skip mode, JVT-W081 Document JVT, San Jose, CA, USA, Apr. 2007. [13] H. S. Koo, Y. J. Jeon, and B. M. Jeon, Motion information inferring scheme for multi-view video coding, IEICE transactions on communications, vol. E91-B(4), no. 4, pp. 1247 1250, Apr. 2008. [14] A. Vetro, P. Pandit, H. Kimata, A. Smolic, and Y. K. Wang, Joint Multiview Video Model (JMVM) 8.0, ISO/IEC JTC1/SC29/WG11/ N9762, Archamps, France, April 2008. [15] G. J. Sullivan, J. R. Ohm, T. Wiegand, and A. Luthra, Meeting Report of the 26th JVT Meeting, JVT-Z200 DocumentJVT, Antalya, Turkey, January 2008. [16] M. M. Hannuksela, D. Rusanovskyy, W. Su, L. Chen, R. Li, P. Aflaki, D. Lan, M. Joachimiak, H. Li, and M. Gabbouj, Multiview-Video- Plus-Depth Coding Based on the Advanced Video Coding Standard, IEEE Trans. Image Processing, vol. 22, no. 9, pp. 3449 3458, Sep. 2013. [17] J. R. Ohm, Overview of 3D Video Coding Standardization, International conference on 3D systems and applications (3DSA), Keynote speech 2, Kitaku, Osaka, Japan, Jun. 2013. [18] G. Tech, K. Wegner, Y. Chen, and S. Yea, 3D-HEVC Draft Text 1, JCT3V-E1001 Document JCT3V, Vienna, Austria, Aug. 2013. [19] K Müller, H. Schwarz, D. Marpe, C. Bartnik, S. Bosse, H. Brust, T. Hinz, H. Lakshman, P. Merkle, F. H. Rhee, G. Tech, M. Winken, and T. Wiegand, 3D High-efficiency video coding for multi-view video and depth data, IEEE Trans. Image Processing, vol. 22, no. 9, pp. 3366 3378, Sep. 2013. [20] P. Helle, S. Oudin, B. Bross, D. Marpe, M. O. Bici, K. Ugur, J. Jung, G. Clare, and T. Wiegand, Block Merging for Quadtree-Based Partitioning in HEVC, IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 12, pp. 1720 1731, Dec. 2012. [21] K. McCann, W. J. Han, and I. K. Kim, Samsung s response to the call for proposals on video compression technology, JCTVC-A124 Document JCTVC, Dresden, Germany, Apr. 2010. [22] O. Bici, J. Lainema, and K. Ugur, CE9: Results of SP Experiments on Simplification of Merge Process, JCTVC-H0252 Document JCTVC, San Jose, CA, USA, Feb. 2012. [23] B. Li, J. Xu, and H. Li, Non-CE9/Non-CE13: Simplification of Adding New Merge Candidates, JCTVC-G397 Document JCTVC, Geneva, Switzerland, Nov. 2011. [24] T. Sugio and T. Nishi, Parsing Robustness for Merge/AMVP, JCTVC-F470, Document JCTVC, Torino, Italy, Jul. 2011. [25] A. Fusiello, E. Trucco, and A. Verri, A compact algorithm for rectification of stereo pairs, Machine Vision and Application, vol. 12, no. 1, pp. 16 22, July 2000. [26] J. An, Y. W. Chen, J. L. Lin, Y. W. Huang, and S. Lei, 3D-CE5.h related: Inter-view motion prediction for HEVC-based 3D video coding, JCT3V-A0049 Document JCT3V, Stockholm, Sweden, July 2012. [27] V. Thirumalai, L. Zhang, Y. Chen, M. Karczewicz, C. Guillemot, L. Guillo, J. L. Lin, Y. W. Chen, and Y. L. Chang, CE3.h: Merge candidates derivation from vector shifting, JCT3V-E0126 Document JCT3V, Vienna, Austria, Aug. 2013. [28] E. Martinian, A. Behrens, J. Xin, and A. Vetro, View synthesis for multiview video compression, in Proc. Picture Coding Symp., vol. 27, pp. 38 39, Apr. 2006. [29] D. Tian, F. Zou, and A. Vetro, Backward view synthesis prediction for 3D-HEVC, in Proc. IEEE Int. Conf. Image Proc., pp. 1729 1733, Melbourne, Australia, Sep. 2013. [30] G. Tech, K. Wegner, Y. Chen, and S. Yea, 3D-HEVC Draft Text 7 (Text of ISO/IEC 23008-2 FDAM4 for ISO/IEC ballot, and submitted for ITU consent), JCT3V-K1001 of JCT-3V, Geneva, Switzerland, Feb. 2015. [31] G. Bang, Y. S. Heo, K. Y. Kim, G. H. Park, G. S. Lee, and N. Hur, 3D-CE3.h related: Simple merge candidate list construction for 3D V, JCT3V-E0213 Document JCT3V, Vienna, Austria, July 2013. [32] G. Bang, Y. S. Heo, Y. J. Lee, G. H. Park, G. S. Lee, and N. Hur, 3D-CE3.h: Results on simple merge candidate list construction for 3DV, JCT3V-F0093 Document JCT3V, Geneva, Switzerland, Oct. 2013. [33] L. Zhang and Y. Chen, CE3 related: combined bi-predictive merging candidates for 3D-HEVC, JCT3V-F0129 Document JCT3V, Geneva, Switzerland, Oct. 2013. [34] JCT-3V, Aachen, Germany. (2014). Subversion repository for the 3D-HEVC test model version HTM-8.2 [Online]. Available: https://hevc.hhi.fraunhofer.de/svn/svn_3dvcsoftware/tags/htm-8.2/ [35] D. Rusanovskyy, K. Müller, and A. Vetro, Common Test Conditions of 3DV Core Experiments, JCT3V-F1100 Document JCT3V, Geneva, Switzerland, Oct. 2013. [36] G. Bjontegaard, Calculation of average PSNR differences between RD-curves, in Proc. 13th VCEG Meeting, pp. 1 5, Austin, TX, USA, 2001. [37] G. Tech, K. Wegner, Y. Chen, and S. Yea, 3D-HEVC Draft Text 2, JCT3V-F1001 Document JCT3V, Geneva, Switzerland, Oct. 2013. [38] JCT-3V, Aachen, Germany. (2014). Subversion repository for the 3D-HEVC test model version HTM-9.0 [Online]. Available: https:// https://hevc.hhi.fraunhofer.de/svn/svn_3dvcsoftware/tags/htm-9.0/ [39] JCT-3V, Aachen, Germany. (2015). Subversion repository for the 3D-HEVC test model version HTM-14.0 [Online]. Available: https:// https://hevc.hhi.fraunhofer.de/svn/svn_3dvcsoftware/tags/htm-14.0/
(JBE Vol. 20, No. 4, July 2015) - 2010 2 : - 2012 2 : - 2012 3 ~ : - ORCID : http://orcid.org/0000-0003-4786-6744 - :,, - 1995 2 : - 1997 2 : - 2014 8 : - 2000 4 ~ : - 2002 10 ~ 2006 2 : ATSC T3/S2 ACAP - 2011 9 ~ 2012 10 : MIT RLE ATSP group - ORCID : http://orcid.org/0000-0003-4355-599x - :,, - 1985 2 : - 1987 7 : - 1991 1 : Case Western Reserve University M.S. - 1995 1 : Case Western Reserve University Ph.D. - 1995 3 ~ 1997 2 : - 1997 3 ~ 2001 2 : ( ) - 2001 3 ~ : - ORCID : http://orcid.org/0000-0001-7133-8285 - :,,,,