3 : 360 (Dongmin Jang et al.: Implementing Renderer for Viewport Dependent 360 Video) (Special Paper) 23 6, (JBE Vol. 23, No. 6, November 2018

Transcription

1 (Special Paper) 23 6, (JBE Vol. 23, No. 6, November 2018) ISSN (Online) ISSN (Print) 360 a), a), a), a) Implementing Renderer for Viewport Dependent 360 Video Dongmin Jang a), Jang-Woo Son a), JongBeom Jeong a), and Eun-Seok Ryu a) , MCTS (Motion Constrained Tile Sets) EIS (Extraction Information Sets) SEI (Supplemental Enhancement Information), (extractor)..,,. Abstract In this paper, we implement viewport dependent tile partitioning for high quality 360 video transmission and rendering method to present a HMD (Head Mounted Display) screen for 360 video quality evaluation. As a method for high-quality video transmission based on a user s viewport, this paper introduces MCTS (Motion Constrained Tile Sets) technique for solving the motion reference problem and EIS (Extraction Information Sets) SEI including pre-configured tile information, and extractor that extracts tiles. In addition, it explains tile extraction method based on user s viewport and implementation contents of the method of expressing on an HMD. Therefore, if 360 video is transferred by the proposed implementation which only transfers video from the user viewport area, it is possible to express higher quality video with lower bandwidth while avoiding unnecessary image transmission. Keyword : 360 Video, Viewport, HEVC, MCTS, Renderer a) (Department of Computer Engineering, Gachon University) Corresponding Author : (Eun-Seok Ryu) esryu@gachon.ac.kr Tel: ORCID: ( : R17XA05-68). This research was supported by Korea Electric Power Corporation. (Grant number : R17XA05-68). Manuscript received September 4, 2018; Revised November 5, 2018; Accepted November 5, Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND ( which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.

2 (JBE Vol. 23, No. 6, November 2018) UHD [1]. UHD, 360. MCTS [2]. 360,,. 360,. 360 MCTS H.264/AVC H.265/HEVC 360 [3]. MCTS.,,., (renderer). MCTS EIS SEI.,. 1 MCTS, MCTS EIS SEI, Fig. 1. The conceptual architecture of tile extracting system based on user viewport VR MVP (Merciless Video Processing) MVP. 2 MVP. MVP, HEVC(High Efficiency Video Coding) HEVC

3 2. MVP(Merciless Video Processing) Fig. 2. The conceptual architecture of merciless video processing project SHVC(Scalable HEVC) ROI(Region Of Interest) (base layer), (enhancement layer)., ROI (asymmetric) big/ little. big/ little [4]., DSP(Digital Signal Processor) (mmwave) [5]. MVP 360,. 2. (viewport) , (bitstream).,.. MPEG

4 (JBE Vol. 23, No. 6, November 2018) MCTS. MCTS HEVC. MCTS., MCTS. A. Zare HEVC MCTS. 3, 3%~6% [6]. FOV(Field Of View) 30%~40% (bitrate). R. Skupin MCTS HEVC (full resolution) (half resolution), 30% BDrate [7]. MCTS HM PSNR.,. 360 MCTS., QoE(Quality of Experience) [8]., (packet) (request) (buffering), (rate control),.. F. Qian ( ), WLR(Weighted Linear Regression) %±2.0%, %±3.7% [9]., MCTS MCTS HEVC SHVC.,,. MCTS HEVC, NAL 1:1 [2]., HM #if MCTS_ENC_CHECK [10] SHM HM, 8. 8, 3 4., 3 4., PU,. 3 TIP(Temporal Inter Prediction)

5 ,. PU PU,., TIP... 3 H, AMVP Merge H. (column) H TIP Fig. 3. Interpolation area problem due to the tile reference to the same location in TIP 2.1. EIS SEI EIS SEI MCTS [4].. VPS, SPS, PPS., EIS SEI 2,000 EIS AMVP Merge SHM HM AMVP(Advanced Motion Vector Prediction) Merge..., PU CTU(Coding Tree Unit) 5. EIS SEI Message Fig. 5. Structure of EIS SEI Message 4. AMVP Merge Fig. 4. Problems determining candidate on vertical boundaries between tiles in AMVP and Merge mode 2.2 EIS SEI MPEG-DASH, MMT(MPEG Media Transport)., 6

6 (JBE Vol. 23, No. 6, November 2018) VPS, SPS, PPS., NAL ,,,,.. 6. Fig. 6. Information of original and extracted bitstream HEVC HEVC., HEVC., 35., HEVC EIS SEI VPS, SPS, PPS. 7. target EIS id, MCTS set id, highest temporal id. NAL. PPS, NAL. EIS SEI target EIS id. / NAL nuh_temporal_id_plus1 1 target highest temporal id. temporal id. temporal id nuh_layer_id target MCTS set id /, / first_slice_segment_in_pic_ flag slice_segment_address. / / first_slice_ segment_in_pic_flag 1 0. EIS SEI slice_reordering_flag 1, slice_segment_address EIS SEI output_slice_segment_address.

7 slice_segment_address., EIS MCTS sets, VPS, SPS, PPS. SPS PPS. 7. Fig. 7. Flow chart of the extractor , yaw, pitch, roll., (dynamic viewport) yaw, pitch [11]., yaw, pitch. (angle) 3.,. (RA: Random Access) GOP(Group Of Picture) 8 GOP I-., DPB(Decoded Picture Buffer). 8. Fig. 8. Block diagram of extracting tiles corresponding to the user viewport

8 (JBE Vol. 23, No. 6, November 2018), GOP I-., I-., GOP. GOP s1 p1, p1 p1 s1.,. v0 v1, i0, v0 i0 v1 i0 v0 v1 v0, v1, Fig. 9. The intersection of the viewport edge plane and the tile edge line 10. Fig. 10. Example of tile decision based on viewport

9 3. HM Oculus Rift SDK, OpenGL, MCTS EIS SEI [12,13]., (shader) OpenGL GLU, ERP 360 [14,15]. glusphere. 2 x 40, y 40. ERP 360 YUV 420 RGB..,. Oculus SDK HMD. Oculus Rift 90Hz, HMD Oculus Rift. 11. Fig. 11. The conceptual architecture of renderer 11, HMD,, Oculus SDK OpenGL 13. (Trolley_ ) Fig. 13. Rendered display (Trolley_ ) 12. Fig. 12. Original bitstream and extracted bitstream

10 (JBE Vol. 23, No. 6, November 2018). JVET(Joint Video Exploration Team) [16] x 4096, 300, 30 fps, 10bit RA(Random Access) QP 27. FoV(Field of View) 90 ~ 110 3x Table 1. Information of test sequences Name Resolution Frame length Frame rate Bit depth ChairliftRide 8192x fps 10-bit Gaslamp 8192x fps 10-bit Harbor 8192x fps 10-bit KiteFlite 8192x fps 10-bit SkateboardInLot 8192x fps 10-bit Trolley 8192x fps 10-bit JVET 360Lib, [17] x4096 ERP(Equirectangular Projection) 360Lib 4096x2048 ERP. ERP MCTS HM Main 10. (dynamic viewport) GOP GOP. PSNR 8192x4096 ERP End- to-end PSNR. 2 End-to-End MCTS WS-PSNR S-PSNR-NN [17,18]. ChairliftRide PSNR, Trolley PSNR yaw pitch. FoV 75, 4. 4~5, 51.8%. 5 MCTS PSNR Lib 360 Fig video testing procedure using 360Lib

11 2. MCTS End-to-End PSNR Table 2. End-to-End PSNR with MCTS applied bitstream and non-applied bitstream Name MCTS Bit-rate Increasing End-to-End Y-WS-PSNR (Non-MCTS) End-to-End Y-SPSNR-NN (Non-MCTS) End-to-End Y-WS-PSNR (MCTS) End-to-End Y-SPSNR-NN (MCTS) ChairliftRide 8.2% 39.7 db 39.7 db 39.6 db 39.6 db Gaslamp 1.6% 41.9 db 42.0 db 41.9 db 42.0 db Harbor 1.5% 41.1 db 41.1 db 41.1 db 41.1 db KiteFlite 1.3% 38.4 db 38.4 db 38.4 db 38.4 db SkateboardInLot 2.9% 38.8 db 38.8 db 38.8 db 38.8 db Trolley 0.7% 38.7 db 38.7 db 38.7 db 38.8 db 3. dynamic viewport yaw and pitch Table 3. Dynamic viewport s yaw, pitch considering user s region of interest Name Field of View Scenario 1 Angle of Start Scenario 1. Angle of End Scenario 2. Angle of Start Scenario 2. Angle of End ChairliftRide 75 x 75-45, , , , -43 Gaslamp 75 x 75-51, , 19 61, , 15 Harbor 75 x 75-83, -56 7, , 12-50, 42 KiteFlite 75 x , , , , -11 SkateboardInLot 75 x , , 5-65, -5 25, 25 Trolley 75 x , , 12 30, , bit-rate Table 4. Average bit-rate savings and number of tiles per frame in scenarios Name Scenario 1 Number of Tiles (Average) Scenario 1. Bit-rate Saving (Average) Scenario 2. Number of Tiles (Average) Scenario 2. Bit-rate Saving (Average) ChairliftRide % % Gaslamp % % Harbor % % KiteFlite % % SkateboardInLot % % Trolley % % 5. MCTS Dynamic Viewport PNSR Table 5. Dynamic Viewport PSNR with MCTS applied bitstream and non-applied bitstream Name Scenario 1. Y-PSNR on Viewport (None MCTS) Scenario 1. Y-PSNR on Viewport (MCTS) Scenario 2. Y-PSNR on-viewport (None MCTS) Scenario 2. Y-PSNR on Viewport (MCTS) ChairliftRide 40.7 db 40.5 db 39.2 db 39.1 db Gaslamp 40.8 db 40.8 db 42.0 db 42.0 db Harbor 39.5 db 39.5 db 41.3 db 41.3 db KiteFlite 40.0 db 40.0 db 38.7 db 38.7 db SkateboardInLot 39.1 db 39.1 db 39.1 db 39.0 db Trolley 37.9 db 37.9 db 37.0 db 37.0 db

12 (JBE Vol. 23, No. 6, November 2018) , Oculus rift OpenGL., %., 360., MCTS MPEG-I 3DoF+ 6DoF.. (References) [1] M. Champel, T. Stockhammer, T. Fautier, E. Thomas, and R. Koenen, Quality Requirements for VR, 116th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG 116/m39532, [2] R. Skupin, Y. Sanchez, K. Suehring, T. Schierl, E. Ryu, and J. Son, Temporal MCTS Coding Constraints Implementation, 120th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG 120/m41626, [3] J. Boyce, A. Ramasubramanian, R. Skupin, G. J. Sullivan, A. Tourapis, and Y. Wang, HEVC Additional Supplemental Enhancement Information (Draft 4), 29th JCT-VC meeting of ISO/IEC JTC1/SC29/ WG11, JCTVC-AC1005, [4] Y. Ryu, and E. S. Ryu. Video on Mobile CPU: UHD Video Parallel Decoding for Asymmetric Multicores, Proceedings of the 8th ACM International Conference on Multimedia System (MMSys 2017), Taipei, Taiwan, pp , [5] T. T. Le, D. N. Van, and E. S. Ryu. Real-time 360-Degree Video Streaming over Millimeter Wave Communication, International Conference on Information Networking 2018 (ICOIN2018), Chiang Mai, Thailand, pp. 5-17, [6] A. Zare, A. Aminlou, M. Hannuksela, and M. Gabbouj. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications, In Proceedings of the 24th ACM Multimedia Conference, Amsterdam, Netherlands, pp , [7] R. Skupin, Y. Sanchez, D. Podborski, C. Hellge, and T. Schierl, "Viewport-dependent 360 degree video streaming based on the emerging Omnidirectional Media Format (OMAF) standard," Image Processing (ICIP), 2017 IEEE International Conference on. IEEE, Beijing, China, pp , [8] K. Brunnström, S. A. Beker, K. De Moor, A. Dooms, S. Egger, M.-N. Garcia, T. Hossfeld, S. Jumisko-Pyykkö, C. Keimel, M.-C. Larabi et al. "Qualinet white paper on definitions of quality of experience," (accessed Sep. 03, 2018). [9] F. Qian, L. Ji, B. Han, and V. Gopalakrishnan, Optimizing 360 video delivery over cellular networks, Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges. ACM, New York, USA, pp. 1-6, [10] High Efficiency Video Coding (HEVC), de/ (accessed Sep. 03, 2018). [11] T. Ikai, Y. Yasugi, and T. Aono, AHG8: Dynamic viewport generation for 360 video evaluation, 5th JVET Meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC29/WG11, JVET-E0024, [12] J. Son, D. Jang, and E. S. Ryu, Implementing Motion-Constrained Tile and Viewport Extraction for VR Streaming, ACM Network and Operating System Support for Digital Audio and Video 2018 (NOSSDAV2018), Amsterdam, Netherlands, pp , [13] J. Son, D. Jang, J. Chung, E. S. Ryu, Implementing 360 Video Tiled Streaming System, ACM Multimedia Systems 2018 (ACM MMSys 2018), Amsterdam, Netherlands, pp , [14] L. Bassbouss, S. Steglich, and M. Lasak, High quality 360 video rendering and streaming, NEM Summit proceedings, Porto, Portugal, [15] Oculus rift, (accessed Sep. 03, 2018). [16] P. Hanhart, J. Boyce, and K. Choi, JVET common test conditions and evaluation procedures for 360 video, 11th JVET Meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC29/WG11, JVET-K1012-v1, [17] Y. Ye, and J. Boyce, Algorithm descriptions of projection format conversion and video quality metrics in 360Lib Version 7, 11th JVET Meeting of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC29/WG11, JVET-K1004, [18] K. K. Sreedhar, A. Aminlou, and M. M. Hannuksela, "Viewport-adaptive encoding and streaming of 360-degree video for virtual reality applications," IEEE International Symposium Multimedia (ISM) 2016 on. IEEE, San Jose, USA, pp , 2016.

13 ~ : - ORCID : -, HMD/VR, : : ~ : Fraunhofer HHI - ORCID : -,, : ~ : - ORCID : -,, : : : ~ : ~ : ~ : Staff Engineer ~ : / ~ : - ORCID : -,, HMD/VR