iphone OS Sound Programming 5 Core Audio For iphone OS 2010-2 Dept. of Multimedia Science, Sookmyung Women's University JongWoo Lee 1
Index 1. Introduction 2. What is Core Audio? 3. Core Audio Essentials 1) API Architectural Layers 2) Frameworks 3) Proxy Objects 4) Properties, Scopes, and Elements 5) Callback Functions: Interacting with Core Audio 6) Audio Data Formats 7) Data Format Conversion 8) Sound Files 9) Sound Streams 10) Audio Sessions: Cooperating with Core Audio 2
1. Introduction ios Mac OS X, ios Core Audio,,, (positioning),, (equalizer) (mixer) /, APIs ios Objective-C Cocoa Touch Core Audio Core Audio DRM ios,,, MIDI 3
2. What Is Core Audio? Core Audio ios Mac OS X (infrastructure) (high performance), (low latency) ios Mac OS X Core Audio in ios Mac OS X Audio Session Services; ipod 1-1 ios Core Audio architecture 4
2. What Is Core Audio? Digital Audio and Linear PCM Core Audio Linear PCM(pulse-code-modulated) PCM (sampling rate) CD (resolution bit depth) 16bit 44kHz sampling rate Sample Frame Packet (time-coincident). ) ( / ). 5
2. What Is Core Audio? Digital Audio and Linear PCM ios ios (unit converter) Audio Converter Services 6
2. What Is Core Audio? Audio Units (Audio unit) ios, (low-latency) ios, IOS 7
2. What Is Core Audio? The Hardware Abstraction Layer HAL(hardware abstraction layer), IOS AURemoteIO, HAL Mac OS X AUHAL 1-2 Hardware input through the HAL and the AUHAL unit 8
Core Audio (layerd) (cooperative) (task-focused),,, Core Audio 9
API Architectural Layers Core Audio 3 2-1 The three API layers of Core Audio 10
API Architectural Layers Core Audio Layer APIs Description High-Level Services Mid-Level Services Low-Level Services Audio Queue Services AVAudioPlayer class Extended Audio File Services OpenAL Audio Converter Services Audio File Services Audio Unit Services Audio Processing Graph Services Audio File Stream Services Core Audio Clock Services Audio Format Services I/O Kit Audio HAL(hardware abstraction layer) Core MIDI Host Time Services,,,,. ios Objective-C Audio File Services Audio Converter services positional audio OpenAL Core Audio implementation. ( ), DSP(digital signal processing), ( API, ) MIDI 11
Frameworks ios Core Audio framework Audio Toolbox framework (AudioToolbox.framework) Audio Unit framework (AudioUnit.framwork) AV Foundation framework (AVFoundation.framework) Core Audio,. ios ipod Audio Session Services, Objective-C AVAudioPlayer (ios ) Core Audio framework (CoreAudio.framework) Core Audio OpenAL framework (OpenAL.framework) positional audio OpenAL 12
Proxy Objects Core Audio,, (Proxy Object) AudioFileCreateWithURL (instantiate),, Core Audio 13
Properties, Scopes and Element (1/2) s Core Audio (property) (key-value) (property key) (property value), (accessor) Core Audio ( ) 14
Properties, Scopes and Element (2/2) s ) kaudioqueueproperty_enablelevelmetering true Core Audio,, (audio unit) / ( ) kaudiounitproperty_audiochannellayout AudioUnitGetProperty ( / ) (0,1,2, ) 15
Callback Functions: Interacting with Core Audio dd d Core Audio (Callback Functions) Core Audio. ) ) 16
Callback Functions: Interacting with Core Audio dd d. ( ), ) Audio Queue Services, (, AudioQueue.h ) typedef void (*AudioQueuePropertyListenerProc) ( void * inuserdata, AudioQueueRef inaq, AudioQueuePropertyID inid ); Code 2-1 A template for a callback function 17
Callback Functions: Interacting with Core Audio dd d 1. Objective-C, body ( inuserdata) statement Code 2-2 static void propertylistenercallback ( void *inuserdata, AudioQueueRef queueobject, AudioQueuePropertyID propertyid ) { AudioPlayer *player = (AudioPlayer *) inuserdata; // [player.notificationdelegate updateuserinterfaceonaudioqueuestatechange: player]; // notificationdelegate UI } Code 2-2 A property listener callback implementation 18
Callback Functions: Interacting with Core Audio dd d 2. (dedicated) Code 2-3 AudioQueueAddPropertyListener ( self.queueobject, // the object that will invoke your callback kaudioqueueproperty_isrunning, // the ID of the property you want to listen for propertylistenercallback, // a reference to your callback function self ); Code 2-3 Registering a property listener callback 19
Audio Data Formats Core Audio Audio Data Format (Sample rate), (bit depth), (packetization ) Audio File Format,, (, MP3), (, Apple CAF) 20
Audio Data Formats Universal Data Types in Core Audio (1/2) Core Audio AudioStreamBasicDescription AudioStreamPacketDescription CoreAudioTypes.h AudioStreamBasicDescription struct AudioStreamBasicDescription { Float64 msamplerate; UInt32 mformatid; UInt32 mformatflags; UInt32 mbytesperpacket; UInt32 mframesperpacket; UInt32 mbytesperframe; UInt32 mchannelsperframe; UInt32 mbitsperchannel; UInt32 mreserved; // }; 0 typedef struct AudioStreamBasicDescription Code 2-4 The AudioStreamBasicDescription data type Stream, Core Audio ( ).. Stream (, ). audio stream basic description ASBD. 21
Audio Data Formats Universal Data Types in Core Audio (2/2) AudioStreamPacketDescription struct AudioStreamPacketDescription { SInt64 mstartoffset; UInt32 mvariableframesinpacket; // 0. }; UInt32 mdatabytesize; typedef struct AudioStreamPacketDescription AudioStreamPacketDescription; Code 2-5 The AudioStreamPacketDescription data type Audio Data Format, AudioStreamPacketDescription (bit-rate) mvariableframesinpacket 0 22
Audio Data Formats Obtaining a Sound File s Data Format ASBD, 0 Core Audio. - (void) openplaybackfile: (CFURLRef) soundfile { AudioFileOpenURL ( (CFURLRef) self.audiofileurl, 0x01, // kaudiofilecaftype, &audiofileid ); UInt32 sizeofplaybackformatasbdstruct = sizeof ([self audioformat]); } AudioFileGetProperty ( [self audiofileid], kaudiofilepropertydataformat, &sizeofplaybackformatasbdstruct, &audioformat // ASBD ); Code 2-6 Obtaining an audio stream basic description for playing a sound file 23
Core Audio (canonical) Core Audio,,, ASBD Core Audio Audio Data Formats Canonical Audio Data Formats (1/2) Format Sample ios / ios Mac OS x / Mac OS X 16bit Linear PCM 8.24bit Noninterleaved Linear PCM 32bit Linear PCM 32bit Noninterleaved Linear PCM 24
Audio Data Formats Canonical Audio Data Formats (2/2) ASBD ), 44.1kHz sample rate iphone struct AudioStreamBasicDescription { msamplerate = 44100.0; mformatid = kaudioformatlinearpcm; mformatflags = kaudioformatflagsaudiounitcanonical; mbytesperpacket = 1 * sizeof (AudioUnitSampleType); // 8 mframesperpacket = 1; mbytesperframe = 1 * sizeof (AudioUnitSampleType); // 8 mchannelsperframe = 2; mbitsperchannel = 8 * sizeof (AudioUnitSampleType); // 32 mreserved = 0; };, CoreAudioTypes.h AudioUnitSampleType ios Mac OS X ASBD (agnostic) 25
Magic Cookies (1/2) Audio Data Formats Core Audio (Magic Cookie) (decoder) Black box Core Audio 26
Magic Cookies (2/2) Audio Data Formats ) - (void) copymagiccookietoqueue: (AudioQueueRef)queue fromfile: (AudioFileID)file { } UInt32 propertysize = sizeof (UInt32); OSStatus result = AudioFileGetPropertyInfo ( file, kaudiofilepropertymagiccookiedata, &propertysize, NULL ); if (!result && propertysize) { } char *cookie = (char *) malloc (propertysize); AudioFileGetProperty ( ); file, kaudiofilepropertymagiccookiedata, &propertysize, cookie AudioQueueSetProperty ( ); queue, kaudioqueueproperty_magiccookie, // cookie, propertysize free (cookie); // // ID, ID, byte, (1=, 0= ) // // ID, ID,, ID Audio Queue Services, //, ID,, Audio File Services, AudioToolbox/AudioToolbox.h, AudioFile.h AudioToolbox/AudioToolbox.h, AudioQueue.h Code 2-7 Using a magic cookie when playing a sound file 27
Audio Data Packets (Packet) Audio Data Formats Core Audio (Synchronization) (counting) (, Code 2-8, ). ASBD mbytesperpacket mframesperpacket 28
Audio Data Packets Audio Data Formats - (void) calculatesizesfor: (Float64) seconds { UInt32 maxpacketsize; UInt32 propertysize = sizeof (maxpacketsize); AudioFileGetProperty ( audiofileid, kaudiofilepropertypacketsizeupperbound, &propertysize, &maxpacketsize ); static const int maxbuffersize = 0x10000; static const int minbuffersize = 0x4000; // 64K // 16K if (audioformat.mframesperpacket) { Float64 numpacketsfortime = audioformat.msamplerate / audioformat.mframesperpacket * seconds; [self setbufferbytesize: numpacketsfortime * maxpacketsize]; } else { // 0, [self setbufferbytesize: maxbuffersize > maxpacketsize? maxbuffersize : maxpacketsize]; } // if (bufferbytesize > maxbuffersize && bufferbytesize > maxpacketsize) { [self setbufferbytesize: maxbuffersize]; } else { if (bufferbytesize < minbuffersize) {[self setbufferbytesize: minbuffersize]; } } } [self setnumpacketstoread: self.bufferbytesize / maxpacketsize]; Listing 2-8 Calculating playback buffer size based on packetization 29
Audio Data Packets Packet CBR (Constant bit rate, ) VBR (Variable bit rate, ) VFR (Variable frame rate, ) Audio Data Formats Core Audio VBR VFR AudioStreamPacketDescription (code 2-5) VBR VFR / CBR VBR ( ) Linear PCM, IMA/ADPCM. AAC, Apple Lossles, MP3.. 30
Data Format Conversion, (Converter) Sample rate interleaving/deinterleaving / Linear PCM ) AAC, Advanced Audio Coding Linear PCM Linear PCM ) 16bit (signed integer) Linear PCM 8.24 (fixed point) Linear PCM Audio Queue Services 31
Sound Files Core Audio Audio File Services. Audio File Services,,,, SMPTE ( ID,, ) AudioFileGetGlobalInfoSize : AudioFileGetGlobalInfo : AudioFile.h, 32
Sound Files Creating a New Sound File CFURL NSURL ) CAF kaudiofilecaftype ASBD Audio File Services AudioFileCreateWithURL ( AudioFileID ) AudioFileCreateWithURL ( audiofileurl, kaudiofilecaftype, &audioformat, kaudiofileflags_erasefile, &audiofileid // the function provides the new file object here ); Code 2-9 Creating a sound file, 33
Sound Files Opening a Sound File AudioFileOpenURL URL,, ID ( Audio File Services ) kaudiofilepropertyfileformat kaudiofilepropertydataformat kaudiofilepropertymagiccookiedata kaudiofilepropertychannellayout VBR (podcast) kaudiofilepropertypacketsizeupperbound kaudiofilepropertyestimatedduration 34
Sound Files Reading From and Writing To a Sound File Audio File Services. (block), ( ) / VBR Audio File Stream Services( Sound Stream ) 35
Sound Files Extended Audio File Services Linear PCM ( ) Audio File Services Audio Converter Services iphone Audio File Format ios Format name AIFF CAF MPEG-1, layer 3 MPEG-2 or MPEG-4 ADTS MPEG-4 WAV Format filename extensions.aif,.aiff.caf.mp3.aac.m4a, mp4.wav 36
Sound Files CAF Files ios, Mac OS X Core Audio Format(CAF) ios 2.0. 37
Sound Streams, (dropouts),, Audio File Stream Service (parsing) Audio File Stream Service AudioFileStreamID 38
Sound Streams ( kaudiofilestreamproperty_readytoproducepackets.) 1. 2. Audio File Stream Services 3.. kaudiofilestreamproperty_readytoproducepackets true( 1) 4. Audio File Stream Services kaudiofilestreamproperty_readytoproducepackets ID 5. Audio File Stream Services 39
Audio Sessions: Cooperating with Core Audio ios, iphone iphone (Audio Session) ios iphone / ipod 40
Next Chapter 1. Audio Session 2. Using Audio in iphone 41