MPEG 3D Video Coding 요구사항및응용분야 2011. 08. 16. 호요성
발표내용 차세대방송기술과 3 차원 TV 3 차원비디오압축을위한 MPEG 표준화작업 3D Audio Visual (3DAV) Multi-view Video Coding g( (MVC) Free Viewpoint TV (FTV) 3D Video Coding (3DVC) Final Call for Proposals Test Conditions and Requirements Subjective Test and Time Line 맺음말
차세대방송기술 실감형방송서비스 고화질방송 + 3 차원방송
3 차원영상시대 3 차원디스플레이장치의상용화 3 차원영화의흥행성공
3D Audio-Visual
3D Audio-Visual (3DAV) 3D Video 3 차원영상의표현방법과부호화형식 영상획득장치의기하학적정보를포함 주요특징 상호작용 (interactivity): 자유로운시점변환 자연스러운영상 (photo-realistic image)
3DAV 표준화작업 2001 /12 2002 /05 2002 /12 2003 /10 2004 /10 time First Proposal on 3D Video EEs on 3DAV 3DAV Seminar CfC on 3DAV CfE on MVC 3D Video 3DAV activities Applications and requirements Representation format and camera parameters Expolation Experiemnts on 3DAV
3DAV 탐색실험 기존의 MPEG 기술과연동가능성을실험함 EE1: Omni-directional video EE2: FTV/Free viewpoint video EE3: Stereoscopic video (coding efficiency test) EE4: Stereoscopic video (depth based rendering)
Multi-view Video Coding
다시점비디오부호화 Multi-view i Video Coding (MVC) z Decoded Multiple Views Scope of MPEG Standardization Multiple Cameras x From Y Axis MVC Bitstream Multiview Video Encoder Multiview Video Decoder Displays 2D TV/HDTV Stereo Display Multi-view Display
MVC 표준화작업 2005 /01 2005 /04 2005 /07 2006 /01 2006 /04 2006 /07 time First Draft CfP Second Draft CfP CfP on MVC Evaluation of Proposals Core Experiments MVC Work in JVT MVC works in MPEG MVC works in JVT Test Data Fix Test Conditions MVC Work
다시점비디오카메라배열 1D Parallel with 8 Cameras 2D Parallel with 5 Cameras Convergent 4 cameras 1D Arc with 8 Cameras 2D Array with 128 Cameras Divergent 4 cameras
응용분야 : 3DTV VIEW-1 TV/HDTV VIEW-1 TV/HDTV VIEW-2 VIEW-3 TV/HDTV VIEW-2 VIEW-3 TV/HDTV VIEW-3 Stereo system Channel Multi Multi-view view video video encoder encoder Multi Multi-view view video video decoder decoder VIEW-3 Stereo system Channel Multi Multi-view view video video encoder encoder Multi Multi-view view video video decoder decoder - - - Multi-view - - - Multi-view - - VIEW-N 3DTV - - VIEW-N 3DTV 3DTV 3DTV
응용분야 : 자유시점 TV Provides The ability to change viewpoint freely Multiple l Views Available Render one view (real or virtual) to legacy 2D display Useful lfor Surveillance, broadcast TV and stored interactive video
MVC 예측구조 MVC using H.264/AVC Fully compatible to H.264/MPEG-4 AVC Uses hierarchical-b pictures combined in interview and temporal dimension Reorganization of input images into a single stream prior to encoding Inter-view-temporal prediction structure based on AVC, using hierarchical B pictures Associated reordering of multi-view input for compression with AVC
Summary: Coding Results
3D Video Coding
3차원비디오 (3DV) 3DV 주요기능 자유로운시청시점선택 Auto-stereoscopic 재생장치를이용 다시점비디오와깊이영상의부호화기술 다시점깊이영상이용 디스플레이의형식에맞는가상시점영상생성 3 차원워핑 (warping) 을이용한시점이동
3DV 기본구조
Vision of 3D Video 3 차원비디오포맷 스테레오디스플레이뿐만아니라다시점디스플레이장치까지도지원할수있는보다진보되고상호사용이가능한기술들을포함 제한된수의카메라를이용하여스테레오디스플레이와다시점디스플레이에모두사용할수있어야함 Limited Camera Inputs Data Format Constrained Rate (based on distribution) ib i Stereoscopic displays Variable stereo baseline Adjust depth perception Data Format Left Right Auto-stereoscopic N-view displays Wide viewing angle Large number of output views
2D+ 깊이영상 vs. 다시점비디오 2D+ 깊이영상과다시점비디오성능비교 2D+ 깊이영상 : 현재의비디오포맷과호환이가능하지만, 시야각 (viewing angle) 이좁고폐색영역 (occlusion) 을처리하기어려움 다시점비디오 : 넓은시야각을제공할수있지만, 데이터양이카메라개수에비례하여증가하기때문에효율적인부호화기술필수적 Simulcast Bit Rate 3DV should be compatible with: existing standards mono and stereo devices existing or planned infrastructure MVC 3DV 2D 2D+Depth 3D Rendering Capability
3DV 표준화작업 2007 /04 2008 /01 2008 /04 2008 2010 /07 /04 time Request for FTV Work Call for 3D Test Data EEs on 3DV Vision on Preparing 3DV for CfP FTV/3DV Applications and requirements on 3DV Viewing i test for evaluation Updating DERS and VSRS
3 차원비디오주요기능 자유로운시청시점선택 다시점재현장치를이용한입체화면재현 Pos1 R L Pos2 Pos3 R L R L MV 3D Display V1 V2 V3 V4 V5 V6 V7 V8 V9 DIBR DIBR V1 D1 V5 D5 V9 D9 Decoded MVD Data
FTV 표준화주요이슈 FTV 데이터형식 Hardware-independent FTV data format 디코더모듈 Light decoder 보간모듈 To guarantee QoS (Quality of Service)
FTV 표준화현황 표준화의기술적검토를위한환경구축단계 주요기술의기준 SW 를요청한단계 Call for Test Material (CfT) 3DV needs multi-view video and its depth video Test data : multi-view video Depth map generation: DERS View synthesis: VSRS
다시점비디오획득과정 Camera Setting Color Correction Capturing Camera Calibration Multi-view Image Image Video Cropping Rectification Multiple Camera Array 1D parallel camera rig Camera distance: 5~6.5 cm
깊이맵생성 두시점이상의영상을이용한깊이정보추정 Multiview depth map estimation Multiview camera system Multiview images Multiview depth map
DERS 의깊이영상생성방법 Automatic Mode Segmentation Mode Semi-automatic Mode <Automatic mode> <Segmentation mode> <Semi-automatic mode>
깊이맵생성소프트웨어 깊이영상 카메라로부터각화소의거리정보 자유로운시점의선택을위한중간시점생성에사용
깊이맵생성소프트웨어 Nagoya 대학에서모든 SW 를통합 Temporal Enhancement Segmentation based Depth Estimation Semi-automatic Depth Estimation Stereo Matching Algorithm 이용 Graph Cuts Algorithm 을이용하여변이값정제 DERS S/W 다운로드 h // 11 29 / / /MPEG http://wg11.sc29.org/svn/repos/mpeg- 4/test/trunk/3D/depth_estimation/DERS/DERS
영상합성의필요성 시청자의거리와위치에따른시점선택 Pos1 R L Pos2 Pos3 R L R L MV 3D Display V1 V2 V3 V4 V5 V6 V7 V8 V9 DIBR DIBR V1 D1 V5 D5 V9 D9 Decoded MVD Data
3차원워핑 (warping) Depth based 3D Warping Direct texture warping causes contours caused by round-off errors 3D warp the depth image instead of texture image Median filtering 3D warped depth image
영상합성절차 Left view Right view Texture image Depth image Depth image Texture image Depth preprocessing Depth-based 3D warping Depth-based 3D warping Depth-based histogram matching Depth-based histogram matching Base and assistant view blending Depth-based in-painting Final synthesized view
영상합성소프트웨어 Two Approaches 3D warping based view synthesis: VSRS 1.0 -> VSRS 2.0 Disparity based view synthesis: ViSBD 1.0 -> ViSBD 2.1 Integrated version: VSRS 3.0 Latest version: VSRS 3.5 Download: http://wg11.sc29.org/svn/repos/mpeg- g p 4/test/tags/3D/view_synthesis/VSRS_3_5 General Mode 1D Mode Providerofprototype of Nagoya University Thomson Viewpoint Shifting Method 3D warping Disparity based viewpoint shifting Sub-pel Precision Valid Valid
Final Call for Proposals
3DV 표준화작업 2006 /01 2008 /01 2008 /04 2009 2010 /02 /04 time Request for FTV works Call for 3D Test Materials EEs in 3DV Vision on 3DV Preparing for CfP Applications and requirements on 3DV Viewing test for evaluation Development of depth estimation and view synthesis Coding experiments
3 차원비디오코덱 입력 : 다시점카메라로획득한다시점비디오 부호기 : 깊이영상생성, 3 차원표현, 부호화 복호기 : 3 차원비디오복원, 임의시점영상생성 깊이영상생성 3 차원영상복원 3 차원영상생성 전송 장면생성 3D Video Coding 다시점카메라 부호기복호기 3 차원영상
Test Sequences Class Test Sequence Resolution Frame rate (fps) Poznan_Street 1920x1088 25 Class A Poznan_Hall2 1920x1088 25 Undo_Dancer 1920x1080 25 GT_Fly 1920x1080 25 Kendo 1024x768 30 Class C Balloons 1024x768 30 Lovebird1 1024x768 30 Newspaper 1024x768 30
양안식영상처리 Advanced Stereoscopic Applications Two-view configuration Adjust the perceived depth of the 3D scene Inputs: two color videos + two depth videos Reconstructed output is used to synthesize a virtual view 좌 우 깊이영상 3 차원비디오다시점영상가상시점생성코덱영상재현 스테레오디스플레이
다시점영상처리 Auto-stereoscopic Displays Three-view configuration Wider set of views Inputs: three color videos + three depth videos Reconstructed output is used to synthesize a virtual view 좌 중 3 차원비디오코덱 N 시점의다시점영상 우 깊이영상생성 다시점영상재현 가상시점영상
Test Conditions and Parameters Pre-processing on input data: highly discouraged Post-processing after reconstruction is allowed, but need to describe Quantization settings should be kept static Proponents are discouraged from optimizing encoding parameters and any processing steps using non-automatic means
Rendering Conditions Data set View to Synthesize from 2-view test scenario Views to Synthesize from 3-view test scenario (and stereo pair) Kendo 4 (4-5) All 1/8 positions between views 1 and 5 (2.75-3.25) Balloons 4(45) (4-5) All 1/8 positions between views 1 and 5 (2.75-3.25) 25) Lovebird1 7 (7-8) All 1/8 positions between views 4 and 8 (5.75-6.25) Newspaper 5(5-6) All 1/8 positions between views 2 and 6 (3.75 75-4.25) Poznan_Street 3.5 (3.5-3) All 1/16 positions between views 5 and 3 (4.125-3.875) Poznan_Hall2 6.5 (6.5-6) 6) All 1/16 positions between views 7 and 5 (6.125-5.875) 5.875) Undo_Dancer 3 (3-5) All 1/4 positions between views 1 and 9 (4.5-5.5) GT_ Fly 4 (4-2) All 1/4 positions between views 9 and 1 (5.5-4.5)
Submission Requirements Bitstreams for all test classes and test scenarios Decoded and synthesized sequences for all test classes Binary decoder executable Excel sheet for coding results Technical description: C/C++ programming language Complexity assessment Encoding time, decoding time, rendering time Expected memory usage of encoder and decoder Complexity characteristics
Compatibility AVC-CompatibleCompatible Satisfying the requirement on forward compatibility with AVC Partial fulfillment of requirements, if the base view is AVC-coded, d the second (and third) view coded d with so me new approach HEVC-Compatible & Unconstrained Satisfying the requirement on forward compatibility with HEVC, or submissions without any compatibility constraints Indicate the HM version number
Registration 1TByte 2.5 5 thin disc with USB 3.0 high speed (720 0 rounds/min) per proposal send to Vittorio Follow the directory structure t that t will be provided d b y Vittorio to proponents 24 proposal registrations, roughly 50% AVC and 50 % HEVC/unconstrained 9000 Euro per Proposal Withdrawal possible until August 15
Subjective Testing Reduce random stereo pair selection for the 3-view case to one test class (A or B) This will be done secretly by Vittorioand i not tto be annou nced before presenting the final results Anonymized data of proponents will be used by test labs for own scientific work, unless indicated by pro ponents otherwise Slightly updated test method for CfP as extra output document
Time Line 2011/01/28: Preliminary Call for Proposals (public) 2011/03/25: Final Call for Proposals 2011/03/25: Formal registration period opens. 2011/06/15: Formal registration ti period ends 2011/09/01: Coded test material shall be available at the test site. 2011/10/01: Subjective assessment starts 2011/11/21: Registration of documents describing the proposals 2011/11/22: Submission of documents 2011/11/26-12/02: 12/02: Evaluation of proposals at 98th MPEG meeting 47/51
요약 3 차원오디오- 비주얼 (3DAV) 3 차원입체감을제공하는오디오와비디오기술 전방향비디오와양안식영상부호화논의 다시점비디오부호화 (MVC) 시점간상관도를고려한다양한기술이검토됨 계층적 B 화면예측구조를기본으로사용 3 차원비디오 (3DV) 깊이맵을포함하는 3 차원비디오부호화 깊이맵추정과중간시점영상합성기술이검토 향후 2 년안에표준안이제정될것으로보임
감사합니다 호요성교수 062-715-2211, 010-3162-3669 hoyo@gist.ac.kr http://vclab.gist.ac.kr/ 실감방송연구센터 062-715-2263 2263 http://rbrc.gist.ac.kr/ 광주과학기술원 http://www.gist.ac.kr/
두양사 02-3417-4417 www.dooyangsa.co.kr 참고문헌