(JBE Vol. 24, No. 4, July 2019) (Special Paper) 24 4, 2019 7 (JBE Vol. 24, No. 4, July 2019) https://doi.org/10.5909/jbe.2019.24.4.564 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) a), a) Integral Regression Network for Facial Landmark Detection Do Yeop Kim a) and Ju Yong Chang a).,,... Abstract With the development of deep learning, the performance of facial landmark detection methods has been greatly improved. The heat map regression method, which is a representative facial landmark detection method, is widely used as an efficient and robust method. However, the landmark coordinates cannot be directly obtained through a single network, and the accuracy is reduced in determining the landmark coordinates from the heat map. To solve these problems, we propose to combine integral regression with the existing heat map regression method. Through experiments using various datasets, we show that the proposed integral regression network significantly improves the performance of facial landmark detection. Keyword : Face Alignment, Facial Landmark Detection, Deep Learning a) (Department of Electronics and Communications Engineering, Kwangwoon University) Corresponding Author : (Ju Yong Chang) E-mail: jychang@kw.ac.kr Tel: +82-2-940-5136 ORCID: https://orcid.org/0000-0003-3710-7314 IPIU 2019. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(msit) (No. 2018-0-00735, Media Interaction Technology based on Human Reaction and Intention to Content in UHD Broadcasting Environment) The present Research has been conducted by the Research Grant of Kwangwoon University in 2019. Manuscript received May 7, 2019; Revised July 5, 2019; Accepted July 5, 2019. Copyright 2016 Korean Institute of Broadcast and Media Engineers. All rights reserved. This is an Open-Access article distributed under the terms of the Creative Commons BY-NC-ND (http://creativecommons.org/licenses/by-nc-nd/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited and not altered.
1: (Do Yeop Kim et al.: Integral Regression Network for Facial Landmark Detection). (facial landmark detection). (facial landmarks),,.,, (face morphing). (convolutional neural network; CNN).. CNN (heatmap) (heatmap regression) [1].. (convolutional layer), (over- fitting).,.,. argmax, argmax.., argmax. (integral regression) [2]., argmax (expectation)., (Gaussian) (mean). argmax, argmax...,, 1. (normalization). ReLU 0 (sum-normalization).,... (deep learning) CNN,. CNN. [3] CNN CNN 3 cascaded CNN. [4] 4 cascaded CNN coarse-to-fine 50. cascaded CNN, CNN
(JBE Vol. 24, No. 4, July 2019). Multi-task learning task. task task. [5], (head pose estimation),,,,. [6] ResNet-101 [7],,,. Multi-task learning [5],[6]. task.,. [1] Hourglass [17] Stacked Hourglass Network. [1] 2D 3D., [1] Hourglass.. 1. Table 1. Detail of the proposed network module name heat map regression module integral regression module layer name output size operation conv1 128 128 7 7, 64, stride 2 3 3 max pool, stride 2 conv2_x 64 64 conv3_x 32 32 conv4_x 16 16 conv5_x 8 8 deconv1 16 16 stride deconv2 32 32 stride deconv3 64 64 stride regression 64 64 normalization 64 64 expectation 2 1. Fig. 1. Overview of proposed facial landmark detection network
1: (Do Yeop Kim et al.: Integral Regression Network for Facial Landmark Detection) RGB 68. 1, 1. 1 operation., conv2_x operation 3 stride 2 max pooling, 1 1 64, 3 3 64, 1 1 256 3 (bottleneck architecture) 3 [7]. Integral regression module. 1. RGB. ResNet-50 [7],, pooling linear (fully convolutional network; FCN), 3 deconvolution. 256 256 ResNet-50 8 8, 3 deconvolution 64 64. 2..,.. sum-normalization. ReLU 0 2. ReLU (activation function). (1) 0 1, 1. (1). 3. (loss function). (mean squared error; MSE) L 1.., 1.0, 1.0.. (5) (cost function).
(JBE Vol. 24, No. 4, July 2019) 1.. ResNet-50, deconvolution,. ResNet-50 ImageNet [8]. ResNet-50 1,000 2, deconvolution.. 2.. 300W-LP [9], 300W [10], Menpo [11], [12]. 300W-LP (large-pose) [9] 300W 61,225. large-pose. 300W 300-W Challenge [10] Indoor Outdoor 600. Menpo FDDB [13] AFLW [14]. 68, 39. 39 68. 9,000 68 6,679. [12] 114 114 218,595, 68. 114 64 50. 1, 2, 3 3. 300W-LP 300-W, Menpo,. 1, 2, 3 cat1, 300- VW cat2, cat3. 3. Adam [15], 32. (learning rate) 30. PyTorch [16], GTX1080Ti GPU. (normalized mean error; NME) [1]. NME. NME,. NME (area under curve; AUC). AUC NME. baseline. [1] Facial Alignment Network (FAN) baseline. FAN,. 4.
1: (Do Yeop Kim et al.: Integral Regression Network for Facial Landmark Detection) 300W, Menpo, cat1, cat2, cat3. [1] FAN-Trained, [1] FAN, (ResNet50), (ResNet50-Int). FAN FAN-Int. 2 NME. FAN FAN-Trained., ResNet50 FAN 2. (ResNet50-Int) NME Fig. 2. NME curves of proposed method (ResNet50-Int) and baseline models
570 방송공학회논문지 제24권 제4호, 2019년 7월 (JBE Vol. 24, No. 4, July 2019) 표 2. 각 모델에 대한 AUC(%) 표 3. 각 모델의 프레임 당 처리 시간 (ms) Table 2. AUC (%) of each model Table 3. Processing time per frame (ms) of each model 300W Menpo cat1 cat2 cat3 ResNet50 65.61 65.36 61.46 61.28 55.54 ResNet50-Int 71.71 73.62 72.31 74.96 64.61 Dataset Model Dataset 300W MENPO Cat1 Cat2 Cat3 ResNet50 6.88 6.62 6.39 6.64 6.46 ResNet50-Int 7.20 7.51 6.66 6.49 6.36 FAN-Trained 19.68 20.17 19.26 19.08 18.77 Model FAN-Trained 63.39 63.66 62.10 60.97 48.61 FAN 66.68 65.97 68.99 61.24 55.79 FAN 18.78 19.35 18.94 18.73 18.84 FAN-Int 72.60 74.54 73.01 75.61 67.38 FAN-Int 20.16 20.01 20.35 20.31 21.30 준 듈 성능을 보여 다. 그러나 적분 회귀 모 이 추가된 ResNet 델 준 50-Int 모 은 FAN보다 더 좋은 성능을 보여 다. 한편, FAN-Int의 성능이 FAN의 결과에 비해 크게 향상된 모습을 체 모델 중 가장 뛰어난 성능을 보인다. 표 2는 위 그래프의 성능 차이를 AUC를 사용하여 정량적으로 보 여준다. 제안하는 모델인 ResNet50-Int는 FAN-Int를 제외 한 모든 모델보다 뛰어난 성능을 보인다. ResNet50-Int와 보이며, 전 FAN-Int의 실험 결과로부터 히트맵 회귀 기반 네트워크에 듈 적분 회귀 모 을 적용하는 것이 얼굴 특징점 검출의 성능 킴 증명되었다. 다음으로 우리는 제안하는 모델의 복잡도를 평가하기 위 해서 각 모델들의 수행속도를 측정하였다. 그 결과가 위의 표 3에 나타나 있다. ResNet50 기반의 방법이 FAN에 비해 빠르다는 것을 확인할 수 있다. 이는 FAN 모델의 경우 Hourglass 모듈 이 4개 반복된 구조로서 기본적으로 계산 량이 많기 때문이다. 그리고, 히트맵 회귀 네트워크인 Res Net50과 적분 회귀 모듈을 추가한 ResNet50-Int의 처리 시 을 크게 향상시 이 간의 차이는 모든 테스트 데이터셋에서 1ms 이내이며 결과적 으로 적분 회귀 모듈이 전체 복잡도에 미치는 영향은 매우 작다는 것을 알 수 있다. FAN과 FAN-Int의 처리 시간의 차 이 또한 3ms 이내로 모델의 복잡도에 큰 영향을 주지 않는다. 마지막으로 그림 3은 제안하는 방법을 통해 얻은 얼굴 특징점 결과를 정성적으로 보여준다. 그림 3(a)와 (b)는 각 각 특징점을 성공적으로 검출한 경우와 검출에 실패한 경 우를 보여준다. 성공한 경우 참값에 매우 가까운 특징점을 검출하였으며 턱수염, large-pose, 가리워짐 등에 강인하게 동작한다는 것을 확인할 수 있다. 한편 얼굴이 다른 사물에 의해 매우 많이 가리워진 경우와 얼굴이 크게 회전된 경우 검출에 실패하였다. Ⅴ. 결 론 [17] 본 논문에서는 히트맵 회귀 기반의 얼굴 특징점 검출 방법 그림 3. 제안하는 모델의 얼굴 특징점 검출 결과 (a) 성공적으로 검출한 경우 (b) 검출에 실패한 경우 Fig. 3. Examples of facial landmarks detected by proposed method (a) success cases (b) failure cases
1: (Do Yeop Kim et al.: Integral Regression Network for Facial Landmark Detection).,, state-of-theart. 3. (References) [1] A. Bulat and G. Tzimiropoulos, How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks), IEEE International Conference on Computer Vision, pp. 1021-1030, 2017. [2] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei, Integral human pose regression, European Conference on Computer Vision, pp. 529-545, 2018. [3] Y. Sun, X. Wang, and X. Tang, Deep convolutional network cascade for facial point detection, IEEE Conference on Computer Vision and Pattern Recognition, 2013. [4] E. Zhou, H. Fan, Z. Cao, Y. Jiang, and Q. Yin, Extensive facial landmark localization with coarse-to-fine convolutional network cascade, IEEE International Conference on Computer Vision Workshops, 2013. [5] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, Facial landmark detection by deep multi-task learning, European Conference on Computer Vision, 2014. [6] R. Ranjan, V. M. Patel, and R. Chellappa, Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 1, pp. 121-135. 2019. [7] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. [8] A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems, pp. 1097-1105, 2012. [9] X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, Face alignment across large poses: A 3d solution, IEEE Conference on Computer Vision and Pattern Recognition, pp. 146-155, 2016. [10] C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, 300 faces in-the-wild challenge: Database and results, Image and Vision Computing, vol. 47, pp. 3-18, 2016. [11] S. Zaferiou, The menpo facial landmark localisation challenge, IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017. [12] J. Shen, S. Zafeiriou, G. G. Chrysos, J. Kossaifi, G. Tzimiropoulos, and M. Pantic, The first facial landmark tracking in-the-wild challenge: Benchmark and results, IEEE International Conference on Computer Vision Workshops, 2015. [13] V. Jain and E. Learned-Miller, Fddb: A benchmark for face detection in unconstrained settings, UMass Amherst Technical Report, 2010. [14] M. Köstinger, P. Wohlhart, P. M. Roth, and H. Bischof, Annotated facial landmarks in the wild: A large-scale, real world database for facial landmark localization, IEEE International Conference on Computer Vision Workshops, 2011. [15] D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, International Conference on Learning Representations, 2015. [16] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, Automatic differentiation in pytorch, Advances in Neural Information Processing Systems Workshops, 2017. [17] A. Newell, K. Yang, and J. Deng, Stacked hourglass network for human pose estimation, European Conference on Computer Vision, pp. 483-499, 2016. - 2019 2 : - 2019 3 ~ : - ORCID : https://orcid.org/0000-0002-0624-5469 - : (Face Landmark Detection, 3D Shape Reconstruction)
(JBE Vol. 24, No. 4, July 2019) - 2001 2 : - 2008 2 : - 2008 2 ~ 2009 1 : Mitsubishi Electric Research Laboratories (MERL) Postdoctoral Researcher - 2009 4 ~ 2011 1 : DMC - 2011 4 ~ 2012 2 : BK - 2012 3 ~ 2017 2 : - 2017 3 ~ : - ORCID : https://orcid.org/0000-0003-3710-7314 - :,