(Regular Paper) 20 3, 2015 5 (JBE Vol. 20, No. 3, May 2015) http://dx.doi.org/10.5909/jbe.2015.20.3.449 ISSN 2287-9137 (Online) ISSN 1226-7953 (Print) HEVC a), a), a) Object Tracking in HEVC Bitstreams Dongmin Park a), Dongkyu Lee a), and Seoung-Jun Oh a),,,,. HEVC., (MV : Motion Vector) Spatio-Temporal Markov Random Fields (ST-MRF). ST-MRF. 86.4%, 79.8%, F-measure 81.1% F-measure 0.2% 9%. 5.4ms. Abstract Video object tracking is important for variety of applications, such as security, video indexing and retrieval, video surveillance, communication, and compression. This paper proposes an object tracking method in HEVC bitstreams. Without pixel reconstruction, motion vector (MV) and size of prediction unit in the bitstream are employed in an Spatio-Temporal Markov Random Fields (ST-MRF) model which represents the spatial and temporal aspects of the object s motion. Coefficient-based object shape adjustment is proposed to solve the over-segmentation and the error propagation problems caused in other methods. In the experimental results, the proposed method provides on average precision of 86.4%, recall of 79.8% and F-measure of 81.1%. The proposed method achieves an F-measure improvement of up to 9% for over-segmented results in the other method even though it provides only average F-measure improvement of 0.2% with respect to the other method. The total processing time is 5.4ms per frame, allowing the algorithm to be applied in real-time applications. Keyword : Object tracking, Markov random fields, HEVC, compressed domain a) (Dept. of Electronic Engineering, Kwangwoon university) Corresponding Author : (Seoung-Jun Oh) E-mail: sjoh@kw.ac.kr Tel: +82-2-940-5102 ORCID: http://orcid.org/0000-0002-5036-3761 2015 () (No.R0101-15-293, ) 2013. Manuscript received March 23, 2015; revised May 12, 2015; accepted May 12, 2015.
(JBE Vol. 20, No. 3, May 2015). (Video Object Tracking : VOT),,,,. VOT.. (bitstream), H.264/ Advanced Video Coding(H.264/AVC) [1] High Efficiency Video Coding(HEVC) [2]. (Motion Vector : MV),,..,.. (Global Motion Estimation : GME) (macroblock rejection) H.264/AVC [3].. Kas Nicolas H.264/AVC Scalable Video Coding (SVC) [4]., (temporal filtering). Timed motion history images. H.264/AVC (partition size) mean shift clustering [5]. - (median filter) (Global Motion Compensation : GMC), mean shift, mean shift. (Markov Random Fields : MRF). Treetasanatavorn - (Gibbs-MRF theory) (Bayesian estimation) [6]. (stochastic motion coherence model) [7], (affine model) (projection)., (label). Zeng (classification) [8].,,., (maximum a posteriori probability : MAP). Chen coarse-to-fine [9]. MRF (edge) region growing. Mak RANSAC(RANdom SAmple consensus) MRF [10]. RANSAC,,, 4. Khatoonabadi - (Spatial- Temporal Markov Random Fields : ST-MRF) [11]. ST-MRF MRF (over-segmentation)
. (error propagation). HEVC. Khatoonabadi ST-MRF [11] HEVC. HEVC,, 3.. (Global Motion : GM) [12]. ST-MRF... ST-MRF...., 2.., ST-MRF Iterated Conditional Modes (ICM) [13]. Khatoonabadi 1.. ST-MRF Khatoonabadi ST-MRF [11]. ST-MRF MRF (motion coherence), (spatial compactness), (temporal continuity) H.264/AVC. 1. ST-MRF Fig. 1. Flow chart of the ST-MRF model based object tracking method,
(JBE Vol. 20, No. 3, May 2015) (over-segmentation) (error propagation)..... HEVC.. HEVC,, 3.,. (Coding Unit : CU).., [12]. ST-MRF. Khatoonabadi (object shape adjustment).. 2. 2. Fig. 2. Flow chart of the proposed object tracking
1. ST-MRF ST-MRF (rigid object) [11]. ST-MRF., (4x4 ) (labeling).,.. (posterior probability). inter-frame likelihood, intra-frame likelihood, (a priori probability) (Bayesian framework) (1). ㆍ ㆍ (2). argmax ㆍ ㆍ. (2) (3). argmin log log log, Hammersley-Clifford theorem [14] (3) (Gibbs) (4~6). (energy function), Z (normalizing constant). exp exp exp (4)~(6),, (temporal continuity), (motion coherence), (compactness).,, (scaling factor).,,,,.. (degree of overlap).. flat-texture (outlier). Modified Trimmed Mean [11]., PVM (Polar Vector Median) [11].. (compact).. 8-adjacency (weighted sum).
(JBE Vol. 20, No. 3, May 2015) (7) Stochastic Relaxation(SR) [15] ICM [13]. SR ICM. SR ICM. ICM.., ICM (7).... 2. m (10) (10) (11). V H ㆍ m ㆍ (11), v. m (least squares solution) (12)... M-estimator [16] 6 (Affine) [17]. m, (8). v (9). v m H T ㆍ W ㆍ H ㆍ H T ㆍ W ㆍ V W..,.,,. Arvanitidou 8x8 [16].,,., argmin ㆍ ㆍ ㆍ
. W (13). W..,. [12]. 3. 3 (c),. 3. (13).. v m v (14). v.. M-estimator (iterative algorithm). W m ST-MRF. PVM [11]. PVM 4-neighborhood. 4x4. 4. 4 (15). 3.. (a). (b). (c) Fig. 3. GMC. (a) Motion vector before GMC. (b) Motion vector after GMC. (c) Motion vector after ROI-based GMC
(JBE Vol. 20, No. 3, May 2015) (17). argmin v v v. PVM v p (18)~(19). 4. Fig. 4. Motion vertor assignment for intra-coded block v p median v v p median v v v v v v v v v v v v v 4x4 v v v, v. (polar coordinate). v. (16) 5 PVM. PVM v p 4x4. 4. Khatoonabadi 6 5. PVM. (a). (b) : PVM. (c). : PVM. Fig. 5. PVM. (a) Input vectors. (b) Lengths of input vectors. Red line : representative length. (c) Angles of vectors. Red vector : representative angle
...,...,.., log dynamic range.,. 3.0. 7.,,, 4..,. (correlation).. 6.. (a). (b). Fig. 6. Result of over-segmentation. (a) Tracking result. (b) Motion vectors.. 4x4., (outlier) 2x2 7. Fig. 7. Flow chart of object shape adjustment algorithm 8. 8 (a). (b). (c),. (d).
(JBE Vol. 20, No. 3, May 2015) 8.. (a). (b). (c). (d) Fig. 8. Shape adjustment algorithm process. (a) Object estimation. (b) Object boundary extraction. (c) Search a correlation with transform coefficient. (d) Boundary adjustment. 1. Intel i7-2600 3.40GHz, 14.0GB Microsoft Visual Studio 2010 C++. MPEG SIF (352x240), CIF (352x288). HEVC test Model (HM) 14.0 low-delay P configuration Common Test Conditions (CTC). 1, Quantization Parameter (QP) 22.,,. ST-MRF,, 1, 2/3, 0.25 Khatoonabadi [11].,,.,. ground truth. ground truth ground truth true positive, ground truth false positive, ground truth false negative. (precision), (recall), F-measure, (20), TP true positive, FP false positive, FN false negative.
2. Khatoonabadi 1. Khatoonabadi 6.1%.., 5%... F-measure 0.2% FlowerGarden. 9 FlowerGarden. Khatoonabadi. 1. Khatoonabadi Table 1. Tracking performance comparison of the proposed method with respect to the Khatoonabadi s method Sequence Frame Precision (%) Recall (%) F-measure (%) Pro Ref Pro Ref Pro Ref City (CIF) 100 91.68 94.5 95.39 95.4 93.46 94.9 FlowerGarden (SIF) 50 76.49 59.3 84.77 94.0 80.15 71.2 TableTennis (SIF) 25 91.80 96.3 92.14 89.0 91.91 92.4 Stefan (CIF) 90 95.72 85.4 44.79 59.2 60.10 68.7 HallMonitor (CIF) 60 89.30 70.9 68.82 80.9 77.60 75.4 MobileCalendar (CIF) 100 84.05 77.8 82.54 83.1 83.00 80.1 Coastguard (CIF) 100 64.41 62.8 89.75 90.1 74.61 73.4 Foreman (CIF) 100 97.86 95.5 80.03 85.9 87.62 90.2 Average 86.41 80.3 79.78 84.7 81.06 80.8 * Pro, Ref Khatoonabadi. Reference method Frame #2 Frame #14 Frame #26 Frame #38 Frame #50 Proposed method Frame #2 Frame #14 Frame #26 Frame #38 Frame #50 9. FlowerGarden Fig. 9. Tracking results for FlowerGarden
(JBE Vol. 20, No. 3, May 2015). 10 MobileCalendar 200. Khatoonabadi,.,....,, Reference method Frame #2 Frame #20 Frame #90 Frame #150 Frame #200 Proposed method Frame #2 Frame #20 Frame #90 Frame #150 Frame #200 10. MobileCalendar Fig. 10. Tracking results for MobileCalendar Reference method Frame #2 Frame #15 Frame #50 Frame #70 Frame #90 Proposed method Frame #2 Frame #15 Frame #50 Frame #70 Frame #90 11. Stefan Fig. 11. Tracking results for Stefan
. 11 Stefan.. Stefan. 12 Stefan. Khatoonabadi 15 50.. F-measure.... 2. Khatoonabadi 10 MATLAB Khatoonabadi C++. 2. 12. Stefan Fig. 12. Tracking performance for Stefan 2. Table 2. Processing time of each module per frame Sequence Pre-processing (ms) GMC (ms) ST-MRF tracking (ms) Post-processing (ms) Total (ms) Pro Ref Pro Ref Pro Ref Pro Ref Pro Ref City (CIF) 0.01 0.5 7.2 52.9 1.9 7.3 0.3 0 9.4 60.7 FlowerGarden (SIF) 0.01 0.4 5.3 45.5 1.3 7.0 0.2 0 6.9 52.9 TableTennis (SIF) 0.02 0.5 3.3 38.5 1.6 6.2 0.3 0 5.2 45.2 Stefan (CIF) 0.04 1.3 4.1 50.8 0.6 3.3 0.1 0 4.8 55.4 HallMonitor (CIF) 0.04 1.8 1.9 45.9 0.6 3.6 0.1 0 2.7 51.2 MobileCalendar (CIF) 0.01 0.5 2.6 49.5 0.4 2.7 0.1 0 3.2 52.7 Coastguard (CIF) 0.01 0.6 2.8 48.6 0.4 2.7 0.2 0 3.4 51.9 Foreman (CIF) 0.04 1.5 4.9 48.1 2.7 9.3 0.3 0 8.0 58.9 Average 0.02 0.9 4.0 47.5 1.2 5.3 0.2 0 5.4 53.6 * Pro, Ref Khatoonabadi.
(JBE Vol. 20, No. 3, May 2015)... ST-MRF.,. Foreman Coastguard 6... HEVC.,. ST-MRF,. HEVC 3.,. PVM.. ST-MRF -.,,..... 6.1% 5.0% F-measure 0.2% FlowerGarden F-measure 9%. MobileCalendar. (References) [1] ITU-T, Advanced Video Coding for Generic Audiovisual Services, Rec. H.264/ISO IEC 14996-10 AVC, 2003. [2] B. Bross, W-J. Han, J-R. Ohm, G. J. Sullivan, Y-K. Wang, and T. Wiegand, High Efficiency Video Coding (HEVC) Text Specification Draft 10, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, JCTVC-L1003, Geneva, CH, Jan. 2013. [3] V. Mezaris, I. Kompatsiaris, N. V. Boulgouris, and M. G. Strintzis, Real-time compressed-domain spatiotemporal segmentation and ontologies for video indexing and retrieval, IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 5, pp. 606621, May 2004. [4] C. Käs and H. Nicolas, An approach to trajectory estimation of moving objects in the H.264 compressed domain, in Proc. 3rd Pacific Rim Symp. Adv. Image Video Technol., pp. 318329, 2009. [5] W. Fei and S. Zhu, Mean shift clustering-based moving object segmentation in the H.264 compressed domain, IET Image Process., vol. 4, no. 1, pp. 1118, Feb. 2010. [6] S. Treetasanatavorn, U. Rauschenbach, J. Heuer, and A. Kaup, Bayesian method for motion segmentation and tracking in compressed videos, in Proc. 27th DAGM Conf. Pattern Recognit., pp. 277 284, 2005. [7] S. Treetasanatavorn, U. Rauschenbach, J. Heuer, and A. Kaup, Stochastic motion coherency analysis for motion vector field segmentation on compressed video sequences, in Proc. IEEE Workshop
Image Anal. Multimedia Interact. Services, pp. 14, Apr. 2005. [8] W. Zeng, J. Du, W. Gao, and Q. Huang, Robust moving object segmentation on H.264/AVC compressed video using the block-based MRF model, Real-Time Imaging, vol. 11, no. 4, pp. 290299, Aug. 2005. [9] Y. M. Chen, and I. V. Baji c, Moving region segmentation from compressed video using global motion estimation and markov random fields, IEEE Transactions on Multimedia, vol. 13, no. 3, June 2011. [10] C. M. Mak, and W. K. Cham, Real-time video object segmentation in H.264 compressed domain, IET image Processing, vol. 3, lss. 5, Oct, 2009. [11] S. H. Khatoonabadi and I. V. Baji c, Video object tracking in the compressed domain using spatio-temporal Markov Random Fields, IEEE Transactions on Image Processing, vol. 22, no. 1, pp.300-313, Jan. 2013. [12] D. M. Park, D. K. Lee, S. M. Kim, and S. J. Oh Fast ST-MRF tracking using ROI-based GMC, The Korean Society Of Broadcast Engineers, 2014. Nov. [13] J. Besag, On the statistical analysis of dirty pictures, J. Royal Stat. Soc. B, vol. 48, no. 3, pp. 259302, 1986. [14] J. E. Besag, Spatial interaction and the statistical analysis of lattice systems, J. Royal Stat. Soc., Ser. B, vol. 36, no. 2, pp. 192236, 1974. [15] S. Geman and D. Geman, Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images, IEEE Trans. Pattern Anal. Mach. Intell., vol. 6, no. 6, pp. 721741, Nov. 1984. [16] M. G. Arvanitidou, A. Glantz, A. Krutz, T. Sikora, M. Mrak, and A. Kondoz, Global motion estimation using variable block sizes and its application to object segmentation, in Proc. IEEE Workshop Image Anal. Multimedia Interact. Services, London, U.K., pp. 173176, May 2009. [17] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. 2nd ed, Cambridge Univ. Press, Cambridge, U.K., pp. 3944, 2004. - 2014 2 : - 2014 3 ~ : - ORCID:http://orcid.org/0000-0002-7063-4354 - :, - 2012 2 : - 2014 2 : - 2014 3 ~ : - ORCID:http://orcid.org//0000-0003-0713-854X - :, - 1980 2 : - 1982 2 : - 1988 5 : Syracuse University / - 1982 3 ~ 1992 8 : - 1986 7 ~ 1986 8 : NSF Supercomputer Center - 1987 5 ~ 1988 5 : Northeast Parallel Architecture Center - 1992 3 ~ 1992 8 : - 1992 9 ~ : - 2002 3 ~ : SC29-Korea MPEG Forum - ORCID:http://orcid.org//0000-0002-5036-3761 - :,,