3주차_Core Audio_ key

Size: px

Start display at page:

Download "3주차_Core Audio_ key"

영빈 후
6 years ago
Views:

1 iphone OS Sound Programming 5 Core Audio For iphone OS Dept. of Multimedia Science, Sookmyung Women's University JongWoo Lee 1

2 Index 1. Introduction 2. What is Core Audio? 3. Core Audio Essentials 1) API Architectural Layers 2) Frameworks 3) Proxy Objects 4) Properties, Scopes, and Elements 5) Callback Functions: Interacting with Core Audio 6) Audio Data Formats 7) Data Format Conversion 8) Sound Files 9) Sound Streams 10) Audio Sessions: Cooperating with Core Audio 2

3 1. Introduction ios Mac OS X, ios Core Audio,,, (positioning),, (equalizer) (mixer) /, APIs ios Objective-C Cocoa Touch Core Audio Core Audio DRM ios,,, MIDI 3

4 2. What Is Core Audio? Core Audio ios Mac OS X (infrastructure) (high performance), (low latency) ios Mac OS X Core Audio in ios Mac OS X Audio Session Services; ipod 1-1 ios Core Audio architecture 4

5 2. What Is Core Audio? Digital Audio and Linear PCM Core Audio Linear PCM(pulse-code-modulated) PCM (sampling rate) CD (resolution bit depth) 16bit 44kHz sampling rate Sample Frame Packet (time-coincident). ) ( / ). 5

6 2. What Is Core Audio? Digital Audio and Linear PCM ios ios (unit converter) Audio Converter Services 6

7 2. What Is Core Audio? Audio Units (Audio unit) ios, (low-latency) ios, IOS 7

8 2. What Is Core Audio? The Hardware Abstraction Layer HAL(hardware abstraction layer), IOS AURemoteIO, HAL Mac OS X AUHAL 1-2 Hardware input through the HAL and the AUHAL unit 8

9 Core Audio (layerd) (cooperative) (task-focused),,, Core Audio 9

10 API Architectural Layers Core Audio The three API layers of Core Audio 10

11 API Architectural Layers Core Audio Layer APIs Description High-Level Services Mid-Level Services Low-Level Services Audio Queue Services AVAudioPlayer class Extended Audio File Services OpenAL Audio Converter Services Audio File Services Audio Unit Services Audio Processing Graph Services Audio File Stream Services Core Audio Clock Services Audio Format Services I/O Kit Audio HAL(hardware abstraction layer) Core MIDI Host Time Services,,,,. ios Objective-C Audio File Services Audio Converter services positional audio OpenAL Core Audio implementation. ( ), DSP(digital signal processing), ( API, ) MIDI 11

12 Frameworks ios Core Audio framework Audio Toolbox framework (AudioToolbox.framework) Audio Unit framework (AudioUnit.framwork) AV Foundation framework (AVFoundation.framework) Core Audio,. ios ipod Audio Session Services, Objective-C AVAudioPlayer (ios ) Core Audio framework (CoreAudio.framework) Core Audio OpenAL framework (OpenAL.framework) positional audio OpenAL 12

13 Proxy Objects Core Audio,, (Proxy Object) AudioFileCreateWithURL (instantiate),, Core Audio 13

14 Properties, Scopes and Element (1/2) s Core Audio (property) (key-value) (property key) (property value), (accessor) Core Audio ( ) 14

15 Properties, Scopes and Element (2/2) s ) kaudioqueueproperty_enablelevelmetering true Core Audio,, (audio unit) / ( ) kaudiounitproperty_audiochannellayout AudioUnitGetProperty ( / ) (0,1,2, ) 15

16 Callback Functions: Interacting with Core Audio dd d Core Audio (Callback Functions) Core Audio. ) ) 16

17 Callback Functions: Interacting with Core Audio dd d. ( ), ) Audio Queue Services, (, AudioQueue.h ) typedef void (*AudioQueuePropertyListenerProc) ( void * inuserdata, AudioQueueRef inaq, AudioQueuePropertyID inid ); Code 2-1 A template for a callback function 17

18 Callback Functions: Interacting with Core Audio dd d 1. Objective-C, body ( inuserdata) statement Code 2-2 static void propertylistenercallback ( void *inuserdata, AudioQueueRef queueobject, AudioQueuePropertyID propertyid ) { AudioPlayer *player = (AudioPlayer *) inuserdata; // [player.notificationdelegate updateuserinterfaceonaudioqueuestatechange: player]; // notificationdelegate UI } Code 2-2 A property listener callback implementation 18

19 Callback Functions: Interacting with Core Audio dd d 2. (dedicated) Code 2-3 AudioQueueAddPropertyListener ( self.queueobject, // the object that will invoke your callback kaudioqueueproperty_isrunning, // the ID of the property you want to listen for propertylistenercallback, // a reference to your callback function self ); Code 2-3 Registering a property listener callback 19

20 Audio Data Formats Core Audio Audio Data Format (Sample rate), (bit depth), (packetization ) Audio File Format,, (, MP3), (, Apple CAF) 20

21 Audio Data Formats Universal Data Types in Core Audio (1/2) Core Audio AudioStreamBasicDescription AudioStreamPacketDescription CoreAudioTypes.h AudioStreamBasicDescription struct AudioStreamBasicDescription { Float64 msamplerate; UInt32 mformatid; UInt32 mformatflags; UInt32 mbytesperpacket; UInt32 mframesperpacket; UInt32 mbytesperframe; UInt32 mchannelsperframe; UInt32 mbitsperchannel; UInt32 mreserved; // }; 0 typedef struct AudioStreamBasicDescription Code 2-4 The AudioStreamBasicDescription data type Stream, Core Audio ( ).. Stream (, ). audio stream basic description ASBD. 21

22 Audio Data Formats Universal Data Types in Core Audio (2/2) AudioStreamPacketDescription struct AudioStreamPacketDescription { SInt64 mstartoffset; UInt32 mvariableframesinpacket; // 0. }; UInt32 mdatabytesize; typedef struct AudioStreamPacketDescription AudioStreamPacketDescription; Code 2-5 The AudioStreamPacketDescription data type Audio Data Format, AudioStreamPacketDescription (bit-rate) mvariableframesinpacket 0 22

23 Audio Data Formats Obtaining a Sound File s Data Format ASBD, 0 Core Audio. - (void) openplaybackfile: (CFURLRef) soundfile { AudioFileOpenURL ( (CFURLRef) self.audiofileurl, 0x01, // kaudiofilecaftype, &audiofileid ); UInt32 sizeofplaybackformatasbdstruct = sizeof ([self audioformat]); } AudioFileGetProperty ( [self audiofileid], kaudiofilepropertydataformat, &sizeofplaybackformatasbdstruct, &audioformat // ASBD ); Code 2-6 Obtaining an audio stream basic description for playing a sound file 23

24 Core Audio (canonical) Core Audio,,, ASBD Core Audio Audio Data Formats Canonical Audio Data Formats (1/2) Format Sample ios / ios Mac OS x / Mac OS X 16bit Linear PCM 8.24bit Noninterleaved Linear PCM 32bit Linear PCM 32bit Noninterleaved Linear PCM 24

25 Audio Data Formats Canonical Audio Data Formats (2/2) ASBD ), 44.1kHz sample rate iphone struct AudioStreamBasicDescription { msamplerate = ; mformatid = kaudioformatlinearpcm; mformatflags = kaudioformatflagsaudiounitcanonical; mbytesperpacket = 1 * sizeof (AudioUnitSampleType); // 8 mframesperpacket = 1; mbytesperframe = 1 * sizeof (AudioUnitSampleType); // 8 mchannelsperframe = 2; mbitsperchannel = 8 * sizeof (AudioUnitSampleType); // 32 mreserved = 0; };, CoreAudioTypes.h AudioUnitSampleType ios Mac OS X ASBD (agnostic) 25

26 Magic Cookies (1/2) Audio Data Formats Core Audio (Magic Cookie) (decoder) Black box Core Audio 26

27 Magic Cookies (2/2) Audio Data Formats ) - (void) copymagiccookietoqueue: (AudioQueueRef)queue fromfile: (AudioFileID)file { } UInt32 propertysize = sizeof (UInt32); OSStatus result = AudioFileGetPropertyInfo ( file, kaudiofilepropertymagiccookiedata, &propertysize, NULL ); if (!result && propertysize) { } char *cookie = (char *) malloc (propertysize); AudioFileGetProperty ( ); file, kaudiofilepropertymagiccookiedata, &propertysize, cookie AudioQueueSetProperty ( ); queue, kaudioqueueproperty_magiccookie, // cookie, propertysize free (cookie); // // ID, ID, byte, (1=, 0= ) // // ID, ID,, ID Audio Queue Services, //, ID,, Audio File Services, AudioToolbox/AudioToolbox.h, AudioFile.h AudioToolbox/AudioToolbox.h, AudioQueue.h Code 2-7 Using a magic cookie when playing a sound file 27

28 Audio Data Packets (Packet) Audio Data Formats Core Audio (Synchronization) (counting) (, Code 2-8, ). ASBD mbytesperpacket mframesperpacket 28

29 Audio Data Packets Audio Data Formats - (void) calculatesizesfor: (Float64) seconds { UInt32 maxpacketsize; UInt32 propertysize = sizeof (maxpacketsize); AudioFileGetProperty ( audiofileid, kaudiofilepropertypacketsizeupperbound, &propertysize, &maxpacketsize ); static const int maxbuffersize = 0x10000; static const int minbuffersize = 0x4000; // 64K // 16K if (audioformat.mframesperpacket) { Float64 numpacketsfortime = audioformat.msamplerate / audioformat.mframesperpacket * seconds; [self setbufferbytesize: numpacketsfortime * maxpacketsize]; } else { // 0, [self setbufferbytesize: maxbuffersize > maxpacketsize? maxbuffersize : maxpacketsize]; } // if (bufferbytesize > maxbuffersize && bufferbytesize > maxpacketsize) { [self setbufferbytesize: maxbuffersize]; } else { if (bufferbytesize < minbuffersize) {[self setbufferbytesize: minbuffersize]; } } } [self setnumpacketstoread: self.bufferbytesize / maxpacketsize]; Listing 2-8 Calculating playback buffer size based on packetization 29

30 Audio Data Packets Packet CBR (Constant bit rate, ) VBR (Variable bit rate, ) VFR (Variable frame rate, ) Audio Data Formats Core Audio VBR VFR AudioStreamPacketDescription (code 2-5) VBR VFR / CBR VBR ( ) Linear PCM, IMA/ADPCM. AAC, Apple Lossles, MP3.. 30

31 Data Format Conversion, (Converter) Sample rate interleaving/deinterleaving / Linear PCM ) AAC, Advanced Audio Coding Linear PCM Linear PCM ) 16bit (signed integer) Linear PCM 8.24 (fixed point) Linear PCM Audio Queue Services 31

32 Sound Files Core Audio Audio File Services. Audio File Services,,,, SMPTE ( ID,, ) AudioFileGetGlobalInfoSize : AudioFileGetGlobalInfo : AudioFile.h, 32

33 Sound Files Creating a New Sound File CFURL NSURL ) CAF kaudiofilecaftype ASBD Audio File Services AudioFileCreateWithURL ( AudioFileID ) AudioFileCreateWithURL ( audiofileurl, kaudiofilecaftype, &audioformat, kaudiofileflags_erasefile, &audiofileid // the function provides the new file object here ); Code 2-9 Creating a sound file, 33

34 Sound Files Opening a Sound File AudioFileOpenURL URL,, ID ( Audio File Services ) kaudiofilepropertyfileformat kaudiofilepropertydataformat kaudiofilepropertymagiccookiedata kaudiofilepropertychannellayout VBR (podcast) kaudiofilepropertypacketsizeupperbound kaudiofilepropertyestimatedduration 34

35 Sound Files Reading From and Writing To a Sound File Audio File Services. (block), ( ) / VBR Audio File Stream Services( Sound Stream ) 35

36 Sound Files Extended Audio File Services Linear PCM ( ) Audio File Services Audio Converter Services iphone Audio File Format ios Format name AIFF CAF MPEG-1, layer 3 MPEG-2 or MPEG-4 ADTS MPEG-4 WAV Format filename extensions.aif,.aiff.caf.mp3.aac.m4a, mp4.wav 36

37 Sound Files CAF Files ios, Mac OS X Core Audio Format(CAF) ios

38 Sound Streams, (dropouts),, Audio File Stream Service (parsing) Audio File Stream Service AudioFileStreamID 38

39 Sound Streams ( kaudiofilestreamproperty_readytoproducepackets.) Audio File Stream Services 3.. kaudiofilestreamproperty_readytoproducepackets true( 1) 4. Audio File Stream Services kaudiofilestreamproperty_readytoproducepackets ID 5. Audio File Stream Services 39

40 Audio Sessions: Cooperating with Core Audio ios, iphone iphone (Audio Session) ios iphone / ipod 40

41 Next Chapter 1. Audio Session 2. Using Audio in iphone 41

(JBE Vol. 21, No. 3, May 2016) HE-AAC v2. DAB+ 120ms..,. DRM+(Digital Radio Mondiale plus) [3] xhe-aac (extended HE-AAC). DRM+ DAB HE-AAC v2 xhe-aac..

3 : xhe-aac (Bongho Lee et al.: A Study on the Variable Transmission of xhe-aac Audio Frame) (Special Paper) 21 3, 2016 5 (JBE Vol. 21, No. 3, May 2016) http://dx.doi.org/10.5909/jbe.2016.21.3.357 ISSN