(Special Paper) 21 3, 2016 5 (JBE Vol. 21, No. 3, May 2016) http://dx.doi.org/10.5909/jbe.2016.21.3.330 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) a), b) A Study on Multiple Sensorial Media Application Format Jung Yup Oh a) and Sang-Kyun Kim b) MPEG-A(ISO/IEC 23000). /,,., MPEG-4 12(ISO/IEC 14496-12) ISO MPEG-V 3(ISO/IEC 23005-3). MPEG-V 3 (SEM) SEM XML,. MPEG-V Binary MPEG format for XML(BiM) gzip. Abstract This paper explains about the structure of multiple sensorial media application format (ISO/IEC 23000-17), which is newly standardized as a project of MPEG-A. This format facilitates effective storage, playing, and management of media with multiple sensorial effects. The ISO base media file format from MPEG-4 Part 12 and sensory effect metadata (SEM) from MPEG-V Part 3 are used to composed the multiple sensorial media application format. In this paper, a fragmentation method to break a SEM XML document into valid SEM samples is presented. Several binarization methods to compress the SEM samples are compared and evaluated as well. The compression ratio and processing time using the MPEG-V binary representation and the Binary MPEG format for XML (BiM) are superior to the gzip compression. Keyword : Sensorial effects, MulSeMedia, MPEG-V, MPEG-A, ISO base media file format a) (Telecommunications Technology Association) b) (Myongji Univ. Computer Eng. Dept.) Corresponding Author : (Kim, Sang-Kyun) E-mail: goldmunt@gmail.com Tel: +82-31-330-6443 ORCID:http://orcid.org/0000-0002-2359-8709 IWAIT 2016. ( : 10053655). Manuscript received March 21, 2016. Revised May 24, 2016; Accepted May 24, 2016.. IMAX 4D HD 3D.,,.
1 : (Jung Yup Oh et al.: A Study on Multiple Sensorial Media Application Format) 2008 ISO/IEC 23005( MPEG-V). 3(Sensory information),, XML Sensory Effect Metadata ( SEM). SEM. SEM 4D. XML file,,,. MPEG. ISO/ IEC 23000 ( MPEG-A) [1]. MPEG-A Multimedia Application Format ( MAF). MPEG-A 13. MAF 8 Portable Video MAF (ISO/IEC 23000-8), 9 Digital Multimedia Broadcasting MAF ((ISO/IEC 23000-9)), 11 Stereoscopic Video MAF (ISO/IEC 23000-11). MAF,, DMB,,,,, 3D. MAF. DVD,.. XML., (packer) (unpacker).. II SEM XML. III. IV SEM, V III.. SEM XML ISO (ISO base media file format). ISO,,. SEM. MPEG-V Part 3 (ISO/IEC 23005-3) SEM Effect, GroupOfEffects, ReferenceEffect,. XML (validate). XML Streaming Instruction SEM XML. 1 SEM XML. 1 XML 1,, 1 ( ). ISO/IEC 23005-3 (MPEG-V Part 3). MPEG XML MPEG- 21 7(ISO/IEC 21000-7) XML Streaming In- struction ( XSI). MPEG-V XSI SEM
1. Table 1. A source SEM XML document to be fragmented <?xml version="1.0" encoding="utf-8" standalone="yes"?> <SEM xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns="urn:mpeg:mpeg-v:2010:01-sedl-ns" xmlns:sev="urn:mpeg:mpeg-v:2010:01-sev-ns" xmlns:mpeg7="urn:mpeg:mpeg7:schema:2004" xmlns:ct="urn:mpeg:mpeg-v:2010:01-ct-ns" xmlns:si="urn:mpeg:mpeg21:2003:01-dia-xsi-ns" xsi:schemalocation="urn:mpeg:mpeg-v:2010:01-sev-ns MPEG-V-SEV.xsd" si:pumode="ancestorsdescendants" si:anchorelement="true" si:timescale="1000"> <Effect xsi:type="sev:temperaturetype" si:anchorelement="true" intensity-range="0.0 100.0" intensity-value="40.0" duration="5000" activate="true" si:pts="6000" <GroupOfEffects si:pumode="ancestorsdescendants" si:anchorelement="true" id="explosion_effect" si:pts="20000"> <Effect xsi:type="sev:temperaturetype" intensity-range="0.0 100.0" intensity-value="40.0" duration="5000" activate="true" <Effect xsi:type="sev:windtype" intensity-range="0.0 100.0" intensity-value="50.0" duration="5000" activate="true" <Effect xsi:type="sev:vibrationtype" intensity-range="0.0 100.0" intensity-value="60.0" duration="5000" activate="true" </GroupOfEffects> </SEM> 1. process unit mode ancestorsdescendants Fig. 1. A example case of a process unit mode - ancestorsdescendants. MPEG-V 3, Group- OfEffects, Effect, ReferenceEffect si- AttributeList siattributelist XSI. XML XSI anchor element process unit mode encodes- AsRAP. process unit mode 7 ancestorsdescendants, anchor,. 1 process unit mode ancestorsdescendants anchor. Fragmenter XML XSI. Fragmenter XML anchorelement. pumode. 2 Fragmenter.
1 : (Jung Yup Oh et al.: A Study on Multiple Sensorial Media Application Format) 2. fragmenter Fig. 2. Normative functions of a fragmenter 2 1 XSI Fragmenter. 1 SEM 1, Effect 1, GroupOfEffects 1. Fragmenter 2. SEM XSI pumode time- Scale. SEM timescale pts. 2. XSI fragmenter 1 Table 2. Process units of the XML document in Table 1 generated by the XSI fragmenter PU #1 <?xml version="1.0" encoding="utf-8" standalone="yes"?> <SEM xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns="urn:mpeg:mpeg-v:2010:01-sedl-ns" xmlns:sev="urn:mpeg:mpeg-v:2010:01-sev-ns" xmlns:mpeg7="urn:mpeg:mpeg7:schema:2004" xmlns:ct="urn:mpeg:mpeg-v:2010:01-ct-ns" xmlns:si="urn:mpeg:mpeg21:2003:01-dia-xsi-ns" xsi:schemalocation="urn:mpeg:mpeg-v:2010:01-sev-ns MPEG-V-SEV.xsd" si:pumode="ancestorsdescendants" si:anchorelement="true" si:timescale="1000"> <Effect xsi:type="sev:temperaturetype" si:anchorelement="true" intensity-range="0.0 100.0" intensity-value="40.0" duration="5000" activate="true" si:pts="6000" </SEM> PU #2 <?xml version="1.0" encoding="utf-8" standalone="yes"?> <SEM xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns="urn:mpeg:mpeg-v:2010:01-sedl-ns" xmlns:sev="urn:mpeg:mpeg-v:2010:01-sev-ns" xmlns:mpeg7="urn:mpeg:mpeg7:schema:2004" xmlns:ct="urn:mpeg:mpeg-v:2010:01-ct-ns" xmlns:si="urn:mpeg:mpeg21:2003:01-dia-xsi-ns" xsi:schemalocation="urn:mpeg:mpeg-v:2010:01-sev-ns MPEG-V-SEV.xsd" si:pumode="ancestorsdescendants" si:anchorelement="true" si:timescale="1000"> <GroupOfEffects si:pumode="ancestorsdescendants" si:anchorelement="true" id="explosion_effect" si:pts="20000"> <Effect xsi:type="sev:temperaturetype" intensity-range="0.0 100.0" intensity-value="40.0" duration="5000" activate="true" <Effect xsi:type="sev:windtype" intensity-range="0.0 100.0" intensity-value="50.0" duration="5000" activate="true" <Effect xsi:type="sev:vibrationtype" intensity-range="0.0 100.0" intensity-value="60.0" duration="5000" activate="true" </GroupOfEffects> </SEM>
. SEM (packetizing) (access unit). SEM. Rigid body motion effect 1, XML XML... 1. Binary MPEG format for XML( BiM) MPEG-7 MPEG-B XML 3. Table 3. Box structures of the multiple sensorial media application format Mandatory(*) Box Type Box description * ftyp file type and compatibility pdin Progressive download Information * moov container for all the metadata * mvhd movie header, overall declarations * trak container for an individual track or stream * tkhd track header, overall information about the track edts edit list container elst an edit list * mdia container for the media information in a track * mdhd media header, overall information about the media * hdlr handler, declares the media (handler) type soun for audio data vide for visual data semh for SEM * minf media information container vmhd video media header, overall information (video track only) smhd sound media header, overall information(sound track only) hmhd hint media header, overall information (hint track only) nmhd null media header, overall information(some tracks only) * dinf data information box, container dref data reference box, declares soure(s) of media data * in track * stbl sample table box, container for the time/space map * stsd sample descriptions (codec types, initialization etc.) mpms for SEM data * stts (decoding) time-to-sample ctts (composition) time to sample * stsc sample-to-chunk, partial data-offset information * stsz sample sizes (framing) stz2 compact sample sizes (framing) * stco chunk offset, partial data-offset information co64 64-bit chunk offset stss sync sample table (random access points) mdat Ipmc IPMP Control Box media data container
1 : (Jung Yup Oh et al.: A Study on Multiple Sensorial Media Application Format). MPEG-7 XML XML XML MPEG-B(ISO/IEC 23001). BiM XML. BiM XML simple type complex type FSA (Finite State Automata). gzip. complex type FSA [2]. 3.. XML.,,.,. MPEG-V. MPEG-V (MPEG-V binary representation) [4]. 2. XML XML. gzip (deflate). LZ77. LZ77,. 40-50% [3].. ISO. ISO,. 3 (*). SEM. SEM. 3 3. SEM SEM Fig. 3. A SEM track and its references to the SEM samples
SEM ( mdat') SEM. SEM (stsd), (stts), (stsz), (stco) SEM. (stsd) III SEM. (stts) SEM pts. (stsz) SEM. (stco) SEM. SEM, SEM (mdat). V. SEM. XML SEM III BiM, gzip, MPEG-V. 5,000 Rigid body [4]. 1 5, 1,053,068 XML. Effect 5 SEM. II. 2 Rigid Body 23,156 XML. SEM. BiM, gzip, MPEG-V. 4. 4. ( 1) 5,000 SEM Fig. 4. A SEM instance composed of 5,000 effect elements
1 : (Jung Yup Oh et al.: A Study on Multiple Sensorial Media Application Format) 5. ( 2) SEM Fig. 5. A rigid body SEM instance composed of one huge effect element 4. Table 4. Test environment CPU OS Experimental setup 2.4 GHz Intel Core i5 Windows 7 Enterprise edition IDE Visual Studio 2010 Service Pack 1 BiM MPEG-7 reference software, gzip zlib. MPEG-V c++. SEM MPEG-21 Part 7 reference software DIA-Fragment-A-1., 5. MPEG-V / JAVA MPEG-V reference software. zlib / C++ JAVA. zlib. LOC(Lines Of Code) MPEG-V reference soft 5. XML Table 5. Characteristics of the binarization libraries Comp. Alg. BiM Library MPEG-7 reference software LOC (Lines Of Code) Multi-core support? SW Lang. Open source? 57,290 X JAVA X gzip zlib 9,063 O C++ O MPEG-V Implemented 14,846 X C++ X MPEG-V MPEG-V reference software 100,304 X JAVA X
6. Table 6. Experimental results of compression ratio per binarization method ware 100,304 zlib 9,063. 6. Test set SEM size (byte) Fragmented SEM size (byte) 1 1,053,068 3,071,250 2 23,156-1 BiM MPEG-V 99.34%, 96%. gzip 20.71% SEM Compression method Compressed size (byte) Comp. ratio BiM 112,280 99.34% gzip 2,435,000 20.71% MPEG-V 122,800 96% BiM 1,389 94.0% gzip 19,283 16.7% MPEG-V 977 95.7%. gzip BiM MPEG-V SEM. 2 Rigid body. Rigid 6. Fig. 6. Compression ratio comparison 7. / Fig. 7. Processing time comparison of en/decoding a XML document per binarization method
1 : (Jung Yup Oh et al.: A Study on Multiple Sensorial Media Application Format) body. 2 BiM 94%, gzip 16.7%. MPEG-V 95.7%. 1 SEM /. BiM MPEG-V gzip. SEM 1 Rigid Body 2 BiM MPEG-V. / BiM MPEG-V. SEM BiM MPEG-V. VI. MPEG-A 17(ISO/IEC 23000-17) Multiple Sensorial Media application format, 2014 [5][6][7][8][9]. MPEG-4 12(ISO/IEC 14496-12) ISO MPEG-V 3(ISO/IEC 23005-3). MPEG-V 3 (SEM) XML. ISO SEM (stsd) mpms SEM. (packer) (unpacker). SEM MPEG-B 1(ISO/IEC 23001-1) BiM(Binary MPEG for XML) [2], gzip [3] MPEG-V 3 [4]. 50,00 SEM Rigid body SEM. 50,00 SEM BiM(99.34%), gzip(20.71%), MPEG-V (96%), SEM MPEG-V (95.7%), BiM(94.0%), gzip(16.7%). SEM gzip. SEM, MPEG-2 TS SEM PES [10][11]., SEM SEM PES.. (References) [1] Klaus Diepold, Fernando Pereira, Wo Chang, "MPEG-A: Multimedia Application Formats", MultiMedia, IEEE, vol.12, no. 4, pp. 34-41, October-December 2005. [2] ISO/IEC 23001-1 Information technology - MPEG systems technologies Part 1: Binary MPEG format for XML [3] Deutsch, L. Peter. "GZIP file format specification version 4.3.", 1996. [4] ISO/IEC 23005-3 Information technology - Media context and control - Part 3: Sensory Information. [5] Sang-Kyun Kim, Jungyup Oh, Minwoo Kim, Yong Soo Joo, "Implementation of 4D Media Player Application Format", m32402, 2014.01 MPEG 107th San Jose Meeting. [6] Jungyup Oh, Minwoo Kim, Yong Soo Joo, Sang-Kyun Kim, "Progress on fragmentation of XML document for multi-sensorial application format", m34156, 2014.07 MPEG 109th Sapporo Meeting. [7] Sang-Kyun Kim, Jaejoon Han, Seungju Han, Jungyup Oh, "SEM document fragmentation using Fragment Request Unit (FRU)", m34965, 2014.10 MPEG 110th Strasbourg Meeting. [8] Sang-Kyun Kim, Jungyup Oh, Minwoo Kim, In-Su Jang, "Progress on
MulSeMedia Application Format(MSMAF) Reference Software", m36420, 2015.06 MPEG 112th Warsaw Meeting. [9] ISO/IEC 23000-17:20xx CD Multiple Sensorial Media Application Format, w15873, 2015.10 MPEG 113th Geneva Meeting. [10] Kyoungro Yoon, Bumsuk Choi, Eun-Seo Lee, Tae-Beom Lim, 4-D Broadcasting with MPEG-V, Multimedia Signal Processing (MMSP), 2010 IEEE International Workshop on, 257-262. [11] Kyoungro Yoon, End-to-end framework for 4-D broadcasting based on MPEG-V standard, Signal Processing: Image Communication 28 (2013) 127 135. - 2014 : - 2016 : - 2015 10 ~ : - ORCID : http://orcid.org/0000-0002-0859-9383 - : metadata encoding, multimedia file format, 4D media, intelligent sensorial effect system, standardization strategy roadmap - 1991 : (U of Iowa) BS - 1994 : (U of Iowa) MS - 1997 : (U of Iowa) PhD - 1997 3 ~ 2007 2 : - 2007 3 ~ : - 2016 3 ~ : IT - MPEG (MPEG-7, MPEG-A, MPEG-V, MIoT) / - ORCID : http://orcid.org/0000-0002-2359-8709 - : digital content (image, video, and music) analysis and management, fast image search and indexing, color adaptation, 4D media, sensors and actuators, VR, Internet of Things, and multimedia standardization