4 : CNN (Sangwon Suh et al.: Dual CNN Structured Sound Event Detection Algorithm Based on Real Life Acoustic Dataset) (Regular Paper) 23 6, (J

Size: px

Start display at page:

Download "4 : CNN (Sangwon Suh et al.: Dual CNN Structured Sound Event Detection Algorithm Based on Real Life Acoustic Dataset) (Regular Paper) 23 6, (J"

윤혜 부
4 years ago
Views:

1 (Regular Paper) 23 6, (JBE Vol. 23, No. 6, November 2018) ISSN (Online) ISSN (Print) CNN a), a), a), a), a) Dual CNN Structured Sound Event Detection Algorithm Based on Real Life Acoustic Dataset Sangwon Suh a), Wootaek Lim a), Youngho Jeong a), Taejin Lee a), and Hui Yong Kim a). DCASE. DCASE,.,., CNN, DCASE. Abstract Sound event detection is one of the research areas to model human auditory cognitive characteristics by recognizing events in an environment with multiple acoustic events and determining the onset and offset time for each event. DCASE, a research group on acoustic scene classification and sound event detection, is proceeding challenges to encourage participation of researchers and to activate sound event detection research. However, the size of the dataset provided by the DCASE Challenge is relatively small compared to ImageNet, which is a representative dataset for visual object recognition, and there are not many open sources for the acoustic dataset. In this study, the sound events that can occur in indoor and outdoor are collected on a larger scale and annotated for dataset construction. Furthermore, to improve the performance of the sound event detection task, we developed a dual CNN structured sound event detection system by adding a supplementary neural network to a convolutional neural network to determine the presence of sound events. Finally, we conducted a comparative experiment with both baseline systems of the DCASE 2016 and Keyword : Machine learning, Deep learning, Audio signal processing, Sound event detection, Dataset Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND ( which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.

2 (JBE Vol. 23, No. 6, November 2018).,., (sound event detection) [1-4].,. DCASE(Detection and Classification of Acoustic Scenes and Events), (onset) (offset). 2013, DCASE [5-7],. DCASE (non-negative matrix factorization) [2] (gaussian mixture model) [8], RNN(recurrent neural network) [9] CNN(convolutional neural network) [10] (deep learning) CNN RNN CRNN(convolutional recurrent neural network) [11], a) AV (Realistic AV Research Group, Electronics and Telecommunications Research Institute) Corresponding Author : (Youngho Jeong) yhcheong@etri.re.kr Tel: ORCID: This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No , Development of Human Enhancement Technology for auditory and muscle support) 2018 () (No , ) Manuscript received August 20, 2018; Revised October 24, 2018; Accepted October 24, ,., 5~10 % 3.6% [12] 2015 ILSVRC(ImageNet Large Scale Visual Recognition Challenge) [13]. (ImageNet) [14]. ILSVRC 1, DCASE ~5 24, 659. (imbalanced dataset), DCASE.,. state-of-the-art CNN.,.. II,. III CNN. IV, DCASE V.

3 . 1.. (onset) (offset) (annotation) [15]. DCASE TUT [8] 1. Table 1. Sound event classes I n d o o r O u t d o o r Event class label Number of Hazard instances event Kettle whistle 227 O Children crying 255 O Children playing Children shouting 94 - Dish cleanup Dish rinse sound Dishwasher Doorbell Drawer sound Drop impact sound 287 O Fire alarm sound 218 O Footfall Keyboard sound 95 - Scream 236 O Speech Water flowing Bicycle idle horn 127 O Bird singing Car crash 82 O Car idle horn 90 O Car passing 173 O Car passing horn 130 O Drop impact sound 231 O Footfall Motorcycle idle horn 141 O Motorcycle passing horn 212 O Scream 122 O Speech 3 - Truck idle horn 108 O Truck passing 123 O Truck passing horn 206 O Wind sound 593 -,.. 1, 16., DCASE [5,6]. 2,. 3~5. 2. Table 2. Recording signal specifications Signal type Sampling rate Bit resolution Audio format Binaural / Stereo 44.1 khz 24 bits PCM WAV 2..,. Soundman OKM II Klassik/studio A3 electret in-ear microphones, Rode NT4 X/Y Stereo Condenser Microphone. TASCAM DR-100mkII PCM.

4 (JBE Vol. 23, No. 6, November 2018),., 10.,,. 2,.,,. binaural stereo , DCASE , Table 3. Examples of sound event metadata onset offset event class label footfall footfall water flowing dishwashing drop impact sound III. 1. CNN CNN.,.,. (confusion matrix), 2.. DCASE,,, (a), (b). 44.1kHz 40ms (analysis window) 50% 40. (context information)

(fully-connected layer). 3x3 64, ReLU. 20% dropout 2x2 (max-pooling). 2 128, ReLU sigmoid.

5 1. CNN Fig. 1. Schematic diagram of dual CNN based sound event detection algorithm 3 (Convolution layer) 2 (fully-connected layer). 3x3 64, ReLU. 20% dropout 2x2 (max-pooling) , ReLU sigmoid. (minibatch) 64, (learning rate) 0.001, Adam [16], 100 epochs., (over-adaptation) 10 epoch (loss)., (validation) (evaluation criterion). 3. 3x3 32, ReLU. 20% dropout., 1 ( ) 0( )., 0.2.

(JBE Vol. 23, No. 6, November 2018),. TensorFlow [17] Keras [18]. 2. DCASE 2. (Gaussian Mixture Model, GMM) (Multi-Layer Perceptron, MLP).. n EM(Expectation Maximization).

6 (JBE Vol. 23, No. 6, November 2018),. TensorFlow [17] Keras [18]. 2. DCASE 2. (Gaussian Mixture Model, GMM) (Multi-Layer Perceptron, MLP).. n EM(Expectation Maximization). DCASE 2016 [8], MFCC MFCC-delta MFCC-acceleration. 40ms 50% -, 0 19 MFCC. MFCCdelta MFCC-acceleration MFCC, MFCC ,.,,..,.,. (Backpropagation),. 2. Fig. 2. Block diagram of multi-layer perceptron model DCASE 2017 [7]. 0 ~ Hz 40 -, 40ms 50% (over-

fitting) 20% dropout. Adam, 0.001 learning rate 200 epoch. 100 epoch, 10 epoch. IV. 1. II III. DCASE [5,6], 16. 4 (4-fold cross validation). DCASE (metrics) F1- (F1-score) (error rate) [19].

7 fitting) 20% dropout. Adam, learning rate 200 epoch. 100 epoch, 10 epoch. IV. 1. II III. DCASE [5,6], (4-fold cross validation). DCASE (metrics) F1- (F1-score) (error rate) [19]. 1 (ground truth), 3. true positives(tp), true negatives(tn)., false positives(fp), false negatives(fn). F1- (1) k (precision, P) (recall, R).,., where, 0. (2). N(number of reference events), S(substitutions), I(insertions) FP S, D(deletions) FN S Ground truth Fig. 3. Visualization of system output to ground truth

8 (JBE Vol. 23, No. 6, November 2018),.,.. 2. (context size),., (hop size)., 4., 4 F1-. 4,, 25 51, Table 4. Acoustic event detection results according to the context and hop size Context size [frames] Hop size [frames] , 10. II, 5 6. F1-80.4%, 0.35, F1-73%, 0.46.,. Children shouting, Dish cleanup Speech,, (Class imbalance). 5. Table 5. Detailed results of detection per sound event in an indoor environment <Indoor> F1-score [%] Error Rate Precision Recall Kettle whistle Children crying Children playing Children shouting Dish cleanup Dish rinse sound Dishwasher Doorbell Drawer sound Drop impact sound Fire alarm sound Footfall Keyboard sound Scream Speech Water flowing Instance-based average

9 Bird singing. 6. Table 6. Detailed results of detection per sound event in an outdoor environment <Outdoor> F1-score [%] Error Rate Precision Recall Bicycle idle horn Bird singing Car crash Car idle horn Car passing Car passing horn Drop impact sound Footfall Motorcycle idle horn Motorcycle passing horn Scream Speech Truck idle horn Truck passing Truck passing horn Wind sound Instance-based average III 2, CNN., III 2 DCASE. 7, F1-19.7%, 0.72, F1-3.0%, 0.01., F1-7.2%, 0.16, F1-3.0%, CNN, CNN (w/o aux) CNN (w/ aux)., CNN,. 7. Table 7. Performance test for sound event detection systems F1-score [%] Indoor Error Rate F1-score [%] Outdoor Error Rate GMM MLP CNN (w/o aux.) Proposed (w/ aux.) V., CNN., DCASE TUT,. 13 9, 254. DCASE , 1.7. CNN,.

10 (JBE Vol. 23, No. 6, November 2018) 0 1 CNN. DCASE, II., DCASE , F1-.,.,,.,. (References) [1] A. Temko et al., CLEAR evaluation of acoustic event detection and classification systems, Lecture Notes in Computer Science, vol.4122, pp , [2] D. Stowell et al., Detection and classification of acoustic scenes and events, IEEE Transactions on Multimedia, vol.17, no.10, pp , [3] DCASE Community, [4] J. Portêlo et al., Non-Speech Audio Event Detection, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), [5] DCASE 2016 Task3 Sound event detection in real life audio, [6] DCASE 2017 Task3 Sound event detection in real life audio, [7] A. Mesaros et al., DCASE 2017 Challenge Setup: Tasks, Datasets and Baseline System, Detection and Classification of Acoustic Scenes and Events (DCASE), [8] A. Mesaros, T. Heittola, and T. Virtanen, TUT Database for Acoustic Scene Classification and Sound Event Detection, 24th European Signal Processing Conference (EUSIPCO), pp , [9] S. Adavanne, G. Parascandolo, P. Pertila, T. Heittola, and T. Virtanen, Sound event detection in multichannel audio using spatial and harmonic features, Detection and Classification of Acoustic Scenes and Events (DCASE), [10] I. Jeong, S. Lee, Y. Han, and K. Lee, Audio event detection using multiple-input convolutional neural network, Detection and Classification of Acoustic Scenes and Events (DCASE), [11] S. Adavanne, and T. Virtanen, A report on sound event detection with different binaural features, Detection and Classification of Acoustic Scenes and Events (DCASE), [12] K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Patter Recognition (CVPR), [13] Large Scale Visual Recognition Challenge (LSVRC), [14] ImageNet, [15] Y. Jung, S. Seo, W. Lim, and H. Kim, Design and construction of Acoustic Database for developing Sound Event Detection technique, IEIE Summer General Conference, June, 2018 [16] D. P. Kingma, and J. Ba, Adam: A method for stochastic optimization, Proceedings of the 3rd International Conference on Learning Representations (ICLR), [17] TensorFlow, [18] Keras, [19] Metrics For sound event detection tasks, dcase2017/challenge/metrics

- 2015 : - 2015 ~ : ETRI AV - ORCID : https://orcid.

- :, - 1992 : - 1994 : - 2006 : - 2011 ~ 2017 : (UST) - 1994 ~ :

org/0000-0001-9552-8593 - :,, - 2014 : - 2002 ~ 2003 : Tokyo

2004 : - 2003 ~ 2005 : - 2006 ~ 2010 : (UST) - 2013 ~ 2014 : Univ.

11 : ~ : ETRI AV - ORCID : - :, : : ~ : ETRI AV - :, : : : ~ 2017 : (UST) ~ : ETRI AV - ORCID : - :,, : ~ 2003 : Tokyo Denki University, ~ : ETRI AV / - :,, : : : ~ 2005 : ~ 2010 : (UST) ~ 2014 : Univ. Southern California(USC) ~ : ETRI AV - ORCID : - : /,, UHD/3D/HDR/VR

09권오설_ok.hwp

09권오설_ok.hwp (JBE Vol. 19, No. 5, September 2014) (Regular Paper) 19 5, 2014 9 (JBE Vol. 19, No. 5, September 2014) http://dx.doi.org/10.5909/jbe.2014.19.5.656 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) a) Reduction