Enhanced Film-Grain-Noise Removal Filter for High Fidelity Video Coding In this paper, we propose a novel technique for film grain noise removal, which can be adopted in high fidelity video coding in order to digitize analog films. Film grain noise enhances the natural appearance of high fidelity video, therefore it is should be preserved. However, film grain noise is a burden to typical video compression systems because it has relatively large energy level in the high frequency region. In order to improve the coding performance while preserving film grain noise, the noise removal and synthesis process is used. We propose a film grain noise removal technology in the pre-processing step. In pre-processing step, film grain noise is removed by using temporal and inter-color correlation. Specially, color image denoisng using inter color prediction provides good denoising performance in noise concentrated B plane because film grain noise has inter-color correlation in the RGB domain. The results show that the coding gain of denoised video is higher than for previous works, while the visual quality of the final reconstructed video is well preserved. Keywords: Film grain noise, Video denoising, Inter-color correlation, Bilateral filter, High fidelity video coding
II f t f t N(f t ) n w f t n t (1) f t n t t
N(f t ) f t n w n w f t f t M f t 1 f t 2... f t M p t m p t m p t m p p t m m 1,2,...,M r t m m 1,2,...,M f t p t m f t p t m f t w [w 0,w 1,...,w M ] T d ε E[(f t f t )2 ] ε w 0 ε d 0 w d Cov 1 (g t )1 w T (6) 1 T (g t )1 d w T t r t, (7) f t f t 1 r t n t, (2) f t [f t,p t 1,p t 2,...,p t M ]T 1 [1,1,...,1] T (M 1) r t [0,r t 1,r t 2,...,r t M ] T r t [n t,r p t 1,r p t 2,...,r p t M ]T r p t m m 1,2,...,M n t n t f t E(f t,n t ) E(f t N(f t ))E(n w ) 0 r t [0,r t 1,r t 1,...,r t M ] T Cov 1 (g t ) g t 1 E[n t,g t i ] E[n t (r t i n p t i )] 0, i 1,2,...,M n t g t i Cov (n t,g t i ) 0 i 1,2,...,M E[r t i,n p t j ] E[f t p t i )n p t j ] 0 g t i g t j Cov(g t ) Cov(g t ) diag(σ 2 n t,σ 2 n t 1,σ 2 n t 2,...,σ 2 n t M ) (8) E(p t m n t m ) 0,m 1,2,...,M, (3) w d f 1 t f t 1 g t, (4) σ 2 n 0 w0 M Σ σ 2 g t m σ 2 n t m 1 g t [ n t,g t 1,g t 2,...,g t M ] T g t m r t m n t m f t f t σ 2 g t m wm M Σ σ 2 g t m σ 2 n m 1 t f t w T f t d, (5) M d Σ w m r t m m 1 (9) n t n p t f t p t f t f t n t p t p t n p t g t m z t m f t p t m, m 1,2,...,M g t m r t m n p t m
z t m g t m n t n t n p t m r t m z t m z t m g t m n t σ 2 z t m σ 2 g t m σ 2 n 2E[g t t m n t ] σ g 2 t m σ 2 n t σ z 2 t m σ g 2 t m σ 2 n t σ z 2 t m e t f t f t M σ 2 e t Σ σ 2 r t m σ 2 n, (10) t m 1 f c n c c f c f c n c, c R,G and B, (11) n c N(f c ) n w n w f c p G R p G B p c k p c k n p c k f G f G w G R p G R w G f G w G B p G B d G, (12) w G R w G w G B p G R p G B
σ 2 n G w G σ 2 n G σ 2 n G B σ 2 n G R σ 2 n G w G c σ 2 n G σ 2 n G B σ 2 n G R d G w G B r wg R r, (13) G B G R D 1 D 1 D 1 DΣ {f k (i)f c (i)} { Σ f k (i)}{ Σ f c (i)} i 0 i 0 i 0 a k c, (16) D 1 D 1 D{ Σ f k (i)}2 { Σ f c (i)}2 i 0 i 0 r G c f G p G c σ 2 n G c r G c c p k c a k c f c b k c, (14) a k c b k c 1 D 1 D 1 b k c { Σ f k (i) a k c Σ f c D i 0 i 0 (i)}, (17) D f k f c E min E{(f k a k c f m b k c )2 }, (15) f k f c (f k f c ) (n k n c ), (18)
ρ nk n c f k f c σ 2 (f k f c ) σ 2 (f k f c ) σ 2 n k σ 2 n c 2 ρ nk n c σ nk σ nc, (19) f k f c E σ (f 2 k f c ) σ (f 2 k f c ) σ 2 (f k f c ) TH σ 2 n k σ 2 n c 2 ρ nk n c σ nk σ nc, (20) f t,c f t,c N(f t,c ) n w, (21) t t c f t,c t c f t 1,c f t 2,c... f t M,c f t,c 1 f t,c p t m,c p t,c k p t m,c p t,c k p t m,c p t,c k n p t m,c n p t,c k m 1,2,...,M k 1,2p t m,c p t,c k r t m,c f t,c p t m,c r t,c k f t,c p t,c k m 1,2,...,M k 1,2 f t,c p t m,c p t,c k f t,c f t,c f t,c 1 r t,c n t,c, (22) f t,c [f t,c, p t 1,c, p t 2,c,..., p t M,c, p t,c 1, p t,c 2 ]T 1 [1,1,...,1](M 3) r t,c [0,r t 1,c,r t 2,c,..., r t M,c, p t,c 1, p t,c 2 ] T n t,c [n t,c n p t 1,c,n p t 2,c,..., n p t M,c, n p t,c 1, n p t,c 2 ]T n p t m,c m 1,2,...,M n t,c f t,c f t,c 1 g t,c, (23) g t,c [ n t,c g t 1,c, g t 2,c,..., g t M,c, g t,c 1, g t,c 2 ] T g t m,c r t m,c r t m,c f t,c f t,c f t,c w T f t,c d, (24) w [w 0,w 1,...,w M,w M 1,w M 2 ] T d ε E[(f t,c f t,c )2 ] ε w 0 ε d 0 w d Cov 1 (g t,c )1 w T (25) 1 T (g t,c )1
d w T r t,c, (26) r t,c [0,r t 1,c,r t 2,c,..., r t M,c, r t,c 1, r t,c 2 ] T e t,c f t,c f t,c M 2 σ 2 e t,c Σ σ 2 r t m,c Σ σ 2 m 1 k 1 r t,c k σ 2 n t,c, (27) III
IV [1] N. Salvaggio, Basic photographic materials and processes 3rd edition, Focal Press, 2009. [2] C. Gomila and A. Kobilansky, "SEI message for film grain noise," Proc. 8th Meeting Joint Video Team, Doc. JVT-H022, Geneva, Switzerland, May 2003, pp. 1-14. [3] C. Gomila, "SEI message for film grain encoding: syntax and results," Proc. 9th Meeting Joint Video Team, Doc. JVT-I013r2, San Diego, California, USA, Sep. 2003, pp. 1-13. [4] "Film grain technology -Specifications for H.264 MPEG-4 AVC bit-streams," Society of Motion
Picture and Television Engineers (SMPTE) RDD 5-2006, Mar. 2006. [5] B. T. Oh, C.-C. J. Kuo, S. Sun, and S.-M. Lei, "Film grain noise modeling in advanced video coding," IEEE Trans. Circuits Syst. Video Technol., Vol. 19, No. 12, Dec. 2009, pp. 1717-1729. [6] B. T. Oh, S.-M. Lei, and C.-C. J. Kuo, "Advanced Film Grain Noise Extraction and Synthesis for High- Definition Video Coding," Proc. Visual Commun. Image Process., San Jose, CA, Vol. 6058, Jan. 2007. [7] A. Buades, B. Coll, and J. M. Morel, "A review of image denoising algorithms, with a new one," MULTISCALE MODEL. SIMUL. (SIAM Interdisc. J.), Vol. 4, No. 2, Jul. 2005, pp. 490-530. [8] M. K. Ozkan, M. I. Sezan, and A. M. Tekalp, "Adaptive motion-compensated filtering of noisy image sequences," IEEE Trans. Circuits Syst. Video Technol., Vol. 3, No. 4, Aug. 1993, pp. 277-290. [9] R. Dugad and N. Ahuja, "Adaptive motioncompensated filtering of noisy image sequences," Proc. IEEE Int. Conf. Image Process., Vol. 4, 1999, pp. 152-156. [10] H.-Y. Cheong, A.M. Tourapis, J. Llach, and J. Boyce, "Adaptive spatio-temporal filtering for video denoising," Proc. IEEE Int. Conf. Image Process., Vol. 2, 2004, pp. 965-968. [11] V. Zlokolica, A. Pizurica, and W. Philips, "Wavelet- Domain Video Denoising Based on Reliability Measures," IEEE Trans. Circuits Syst. Video Technol., Vol. 16, No. 8, 2006, pp. 993-1007. [12] J. C. K. Yan and D. Hatzinakos, "Signal-dependent film grain noise removal and generation based on higherorder statistics," Proc. IEEE Signal Processing Workshop on Higher-Order Statistics, pp. 77-81, 1997. [13] J. C. K. Yan, P. Campisi, and D. Hatzinakos, "Film grain noise removal and generation for color images," Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Vol. 5, 1998, pp. 2957-2960. [14] K. J. Boo and N. K. Bose, "A motion-compensated spatio-temporal filter for image sequences with signaldependent noise," IEEE Trans. Circuits Syst. Video Technol., Vol. 8, No. 3, Jun. 1998, pp. 287-298. [15] O. K. Al-Shaykh and R. M. Mersereau, "Denoising Archival Films using a Learned Bayesian Model," IEEE Trans. Image Process., Vol. 7, No. 12, Dec. 1998, pp. 1641-1652. [16] T. M. Moldovan, S. Roth, and M. J. Black, "Adaptive motion-compensated filtering of noisy image sequences," Proc. IEEE Int. Conf. Image Process., 2006. [17] J. Dai, O. C. Au, C. Pang, W. Yang, and F. Zou, "Film grain noise removal and synthesis in video coding," Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2010, pp. 890-893. [18] L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms," Phys. D: Nonlinear Phenom., Vol. 60, No. 1-4, Nov. 1992, pp. 259-268. [19] L. Guo, O. C. Au, M. Ma, and Z. Liang, "Temporal Video Denoising Based on Multihypothesis Motion Compensation," IEEE Trans. Circuits Syst. Video Technol., Vol. 17, No. 10, Oct. 2007, pp. 1423-1429. [20] "Advanced video coding for generic audiovisual services," ITU-T Rec. H.264 ISO/IEC 14496-10 AVC, Nov. 2007. [21] X. Li, "Demosaicing by successive approximation," IEEE Trans. Image Process., Vol. 14, No. 3, Mar. 2005, pp. 370-379. [22] M. T. Orchard, and G. J. Sullivan, "Overlapped block motion compensation: an estimation-theoretic approach," IEEE Trans. Image Process., Vol. 3, No. 5, Sep. 1994, pp. 693-699. [23] Otus, "A threshold selection method from gray-level histograms," Systems, Man and Cybernetics, IEEE Transactions on, Vol. 9, No. 1, Jan. 1979, pp. 62-66.