Advanced
Robust Motion Compensated Frame Interpolation Using Weight-Overlapped Block Motion Compensation with Variable Block Sizes to Reduce LCD Motion Blurs
Robust Motion Compensated Frame Interpolation Using Weight-Overlapped Block Motion Compensation with Variable Block Sizes to Reduce LCD Motion Blurs
Journal of the Optical Society of Korea. 2015. Oct, 19(5): 537-543
Copyright © 2015, Optical Society of Korea
  • Received : April 04, 2015
  • Accepted : September 09, 2015
  • Published : October 25, 2015
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Jichan Lee
Department of Electronics and Radio Engineering, Kyung Hee University, Yongin 17104, Korea
Jin Hyuk Choi
Humanitas College, Kyung Hee University, Yongin 17104, Korea
*
Humanitas College, Kyung Hee University, Yongin 17104, Korea
Abstract
Liquid crystal displays (LCDs) have slow responses, so motion blurs are often perceived in fast moving scenes. To reduce this motion blur, we propose a novel method of robust motion compensated frame interpolation (MCFI) based on bidirectional motion estimation (BME) and weight-overlapped block motion compensation (WOBMC) with variable block sizes. In most MCFI methods, a static block size is used, so some block artefacts and motion blurs are observed. However, the proposed method adjusts motion block sizes and search ranges by comparing matching scores, so the precise motion vectors can be estimated in accordance with motions. In the MCFI, overlapping ranges for WOBMC are also determined by adjusted block sizes, so the accurate MCFI can be performed. In the experimental results, the proposed method strongly reduced motion blurs arisen from large motions, and yielded interpolated images with high visual performance and peak signal-to-noise ratio (PSNR).
Keywords
I. INTRODUCTION
Liquid crystal displays (LCDs) have slow responses as compared with cathode ray tube (CRT) displays because of LCD hold time [1 - 3] , so ghost phenomenon or motion blurs are often perceived around fast moving objects in LCDs. To reduce the motion blurs, backlight flashing [4] , signal overdrive [5] , or frame rate up-conversion (FRUC) [6 - 12] can be used. However, luminance fluctuation may be seen when the flashing rate is not high enough in backlight flashing, and signal overdrive does not consider image contents [1]
To increase video frame rates for high temporal resolution, frame repetition, back/gray frame insertion and linear interpolation may be used; however, these methods cannot remove motion blurs and judders because they do not use motion information [13] . To produce unaffected frames without block artefacts and motion blurs, motion compensated frame interpolation (MCFI) must be used, where motion vectors (MVs) must be correctly estimated by means of various motion estimation (ME) methods.
The conventional MCFI utilizes unidirectional ME (UME); forward ME, backward ME and bilateral ME [14 - 17] . The critical drawback of the UME is shown in Fig. 1 . Forward and backward MEs cannot estimate all pixels’ MVs, so holes and overlaps result from multiple MVs and no MVs. Meanwhile, bilateral ME may not estimate correct MVs when a matching score of a block including moving objects is lower than matching scores of other blocks. In the result of MCFI using bilateral ME of Fig. 1 , the block including a moving ball is not compensated [18] .
PPT Slide
Lager Image
Various UME methods for MCFI.
For the outstanding interpolation results, bidirectional ME (BME) and overlapped block motion compensation (OBMC) are used [19 - 22] . The results of BME + OBMC are very sensitive to the sizes of blocks and search regions for block matching as shown in Fig. 2 . When the sizes are small, the large motions are not estimated, while the detailed motions are not compensated and huge processing time is needed when large sizes are utilized.
PPT Slide
Lager Image
BME + OBMC using various block sizes and search regions for FRUC.
Various methods have been proposed to improve the performance or computation of FRUC [23 - 25] . Regions overlapping with large or complex motions are not, however, correctly compensated. To solve this problem, variable block sizes and overlapping ranges are needed. A method using variable block sizes was presented [26] ; however, this used block splitting by quad scales (coarse-grained block size is 16×16, and fine-grained block size is 4×4), and refined motion estimation is always performed for all blocks whether or not a block is located at an object boundary.
This paper is aimed at developing a novel FRUC method that robustly compensates regions overlapped with large or complex motions. Accordingly, the proposed method uses variable block sizes and overlapping ranges. The block sizes are enlarged by checking a matching score, and the weighted overlapping ranges (WOBMC, weight-OBMC) are also enlarged by the ratio of block enlargement. Therefore, the block sizes and the weighted overlapping ranges can be suitably determined according to the movement and the size of objects, so the proposed method can interpolate frames with high visual and objective performance.
II. MCFI USING BME AND WOBMC WITH VARIABLE BLOCK SIZES
To estimate forward MVs, the proposed method uses a block matching algorithm (BMA) using mean of absolute differences (MAD), which can directly compare matching scores for blocks with variable sizes in contrast to sum of absolute differences (SAD) [27] , in the direction from the current frame ( It ) to the previous frame ( I t−1 ) as follows:
PPT Slide
Lager Image
where the block size is b × b , b is an odd positive integer, and i and j are integers, where b /2 is the largest integer that is less than or equal to the real value of b /2. The block sizes to estimate MVs are normally used as even windows; however, we use odd windows to estimate pixel’s MVs instead of block’s MVs. Therefore, the MV of It ( x, y ) can be determined by
PPT Slide
Lager Image
where u ∈[ x r /2, x + r /2] and v ∈[ y r /2, y + r /2] so the search range is r × r and r is an odd integer.
To estimate an MV at pixel It ( x, y ) with variable block sizes, an initial block size is set to b 0 × b 0 , and its matching block is found within the initial search range r 0 × r 0 . If the matching score S 0 , which is the reciprocal of the MAD, is less than τm , the block size and the search range are set to α 1 b 0 and α 1 r 0 × α 1 r 0 , respectively ( α >1); otherwise the MV computed in the initial size and the initial expansion rate ( α 0 =1) are stored without expansion. If the matching score S 1 using the enlarged block and search range is less than τm (matching score threshold for expansion of block sizes), the block size and the search region are enlarged by α 2 to compute large MVs; otherwise the MV and the expansion rate are stored and the iterative expansion is terminated. When the expansion rate αk is greater than τα (expansion limit), the iterative adjustment is terminated and the MV and the αk with the greatest matching score are selected. The ME using variable block sizes is shown in Fig. 3 . Backward MVs are estimated in the same manner as forward MVs, and are found by using the direction from the previous frame to the current frame.
PPT Slide
Lager Image
Concept of motion estimation using variable block sizes.
Using the estimated forward and backward MVs, v f and v b , WOBMC is performed to interpolate the intermediate frame I t−0.5 as shown in Fig. 4 . In the conventional OBMC, motion blocks are enlarged by predetermined sizes, and then overlapped region are estimated by the weights. Whereas the weight for the motion compensation (MC) is the 2-dimensional Gaussian kernel, and pixels within the overlapping range (all pixels within the enlarged block) are compensated in proposed WOBMC. The overlapped range is assigned as αkO0 , where O0 is the initial overlapping range for b 0 .
PPT Slide
Lager Image
WOBMC with variable block sizes.
Finally, the brightness of the interpolated frame may be reduced owing to the OBMC; the method compensates for this by using the difference with the interpolated image using the averaging M t−0.5 = ( It + I t−1 ) / 2. The compensated brightness is
PPT Slide
Lager Image
where
PPT Slide
Lager Image
denotes the low-pass filtering of f and the windows size of the median filter is large for large motion regions. In our method, the median filter is used for
PPT Slide
Lager Image
, because the median filter can ignore irregular variations.
III. EXPERIMENTAL RESULTS
To evaluate the proposed method, the proposed method tested 12 image sequences; CIF (common intermediate format, 352×288) with 30 Hz, full HD (high definition, 1920×1080) with 50 Hz, and used 51 odd frames and produced 50 intermediate frames. The 50 images were compared with the 50 real even frames to evaluate their peak signal-to-noise ratio (PSNR) as shown in Fig. 5 . The initial sizes of the block, search range and overlapping range are set to 5×5, 9×9 and 7×7, respectively; other parameters were selected as follows:
PPT Slide
Lager Image
is the typical value for upscaling of images, and τα =7, and τm = 0.05 were experimentally selected to reduce computation complexities and to find optimal block sizes. Some test results with variable parameter values are shown in Table 1 . Although the PSNR results with
PPT Slide
Lager Image
, τα =7 and τm = 0.05 are not the highest, we used these parameters to consider the computation complexities. Therefore, the parameter values can be changed for the variable environments of FRUC such as characteristics of image sequences, hardware performance and network environments; however, we used
PPT Slide
Lager Image
, τα =7 and τm = 0.05 in these experiments to objectively compare the proposed method. In addition, one MV is estimated for 2×2 pixels to reduce complexities.
PPT Slide
Lager Image
Quality comparison.
Comparison of Variable Parameter values (PSNR [DB] and processing time [ms])
PPT Slide
Lager Image
Comparison of Variable Parameter values (PSNR [DB] and processing time [ms])
Visual comparison of the results of OBMC, WOBMC and the proposed method is shown in Fig. 6 . The result images interpolated by WOBMC have fewer block artefacts than the images interpolated OBMC in almost all regions because WOBMC uses weights of all pixels. The proposed method, moreover, removed most block artefacts and motion blurs, and was able to produce unaffected intermediate frames owing to the variable block sizes
PPT Slide
Lager Image
Visual comparison of result images (‘foreman’ sequence); (a) Results of OBMC, (b) Results of WOBMC, and (c) Results of the proposed method.
Some result images are shown in Fig. 7 ; many regions of motion blurs in averaging images ( Fig. 7(a) ) are observed, small motion regions are compensated in the results by static block size ( Fig. 7(b) , green boxes), and most motion regions are more clearly interpolated by the proposed method ( Fig. 7(c) , red boxes). In the results of our method, the face region is preserved in the ‘foreman’ sequence, and the shape of the boat and the river bank regions is clearly interpolated in the ‘coastguard’ sequence. The spectators are clearly separated, the text of the billboard is distinctly shown, and the motion blur of the play is reduced in the ‘stefan’ sequence. Especially, the table tennis ball is shown with one object in the ‘tennis’ sequence.
PPT Slide
Lager Image
Sample result images (from top to bottom: ‘foreman’, ‘coastguard’ ‘stefan’ and ‘tennis’ sequences); (a) Images interpolated by averaging, (b) Images interpolated by static block size, and (c) Results of the proposed method.
For objective comparison, the proposed method was compared with other recent methods [23 - 26] as shown in Table 2 . In most test sequences, the PSNRs are higher than those of other methods. Especially, in sequences having large motions, such as ‘football’, ‘tennis’ and ‘stefan’, the performance of the proposed method is improved. In Fig. 8 , the resultant images of the proposed method and other conventional methods [23 - 26] are shown (significant motion blurs and block artifacts are marked as red regions). Even though variable block size method [26] can reduce some motion blurs, large motion blurs are not corrected; however, the proposed method reduced most motion blurs and block artifacts.
PSNR Comparison with Other Methods (PSNR [DB])
PPT Slide
Lager Image
PSNR Comparison with Other Methods (PSNR [DB])
PPT Slide
Lager Image
Sample result images (from top to bottom: replica [23], replica [24], replica [25], replica [26] and the proposed method); (a) ‘flower’, (b) ‘football’, and (c) ‘stefan’ sequences.
In the test of full HD image sequences, the results of our method are very similar to the original frames as shown in Figs. 9 and 10 , where difference values between the results of the proposed method and the original images are very low values.
PPT Slide
Lager Image
Sample result images (‘into tree’ Full HD, from top to bottom: interpolated images and difference images with an original image); (a) Averaging (PSNR: 30.95dB) and (b) Proposed method (PSNR: 35.83dB).
PPT Slide
Lager Image
Sample result images (‘crowd run’ Full HD, from top to bottom: interpolated images and difference images with an original image); (a) Averaging (PSNR: 25.21dB) and (b) Proposed method (PSNR: 29.82dB).
To evaluate the complexities of the proposed method, we compared the proposed method with a symmetric motion estimation method with variable search ranges [28] , which was implemented by software. The Lim’s method [28] uses variable search ranges, while our method uses variable MV sizes and variable search ranges. The average PSNR of the proposed method is a little higher than the Lim’s method, while our method can speed up about 144 times on average. In addition, we compared our method and the replica of Lee’s method [26] as shown in Table 3 , where we tested the proposed method with two parameter sets; one set is the same parameters of the above experiments, and the other set (say speed-up parameters) is 7×7 and 5×5 for initial search range and overlapping range, respectively, and one MV is estimated for 5×5 pixels. The proposed method has the most complexity; however, the result with speed-up parameters has the least complexity, and has higher PSNR than the result of the replica of Lee’s method [26] .
Comparison with replica[26](PSNR [DB] and processing time [ms])
PPT Slide
Lager Image
Comparison with replica [26] (PSNR [DB] and processing time [ms])
The average processing time of the proposed method for 10 CIF sequences is 4,746 ms; however, this is the processing time under software implementation. The processing time of variable block size ME realized by dedicated hardware for HD images is fast enough for real-time processing [29 , 30] , so the proposed method can be used for real-time FRUC processing by hardware implementation.
IV. CONCLUSIONS
A FRUC method using WOBMC with variable block sizes has been proposed in this paper; in the proposed method, bidirectional MVs are estimated using variable block sizes and search ranges comparing matching scores. The overlapping range for WOBMC is also enlarged by using the ME expansion rate. In addition, brightness compensation is utilized by comparing with the median filtered image of averaging interpolation. Therefore, the proposed method can provide high-performance FRUC regardless of the magnitude and complexity of motions to reduce LCD motion blurs.
Acknowledgements
This work was supported by a grant from the Kyung Hee University in 2012 (KHU-20120577).
References
Chan S. H. , Nguyen T. Q. 2011 “LCD motion blur: modeling, analysis, and algorithm,” IEEE Trans. Imag. Proc. 20 2352 - 2365
Liu Y. , Tian F. , Liu R. 2011 “A novel approach to frame rate up-conversion based on morphing,” J. Comput. Inf. Syst. 7 5219 - 5226
Hong S. J. , Kwon O. K. 2014 “An RGB to RGBY color conversion algorithm for liquid crystal display using RGW pixel with two-field sequential driving method,” J. Opt. Soc. Korea 18 777 - 782
Becker M. 2008 “LCD response time evaluation in the presence of backlight modulations,” SID Symp. Tech. Dig. Papers 39 24 - 27
Zhao H. X. , Chao M. L. , Ni F. C. 2008 “Overdrive LUT optimization for LCD by box motion blur measurement and gamma-based thresholding method,” SID Symp. Tech. Dig. Papers 39 117 - 120
Wang T. S. , Choi K. S. , Jang H. S. , Morales A. W. , Ko S. J. 2010 “Enhanced frame rate up-conversion method for UHD video,” IEEE Trans. Consum. Electron. 56 1108 - 1114
Liu H. , Xiong R. , Zhao D. , Ma S. , Gao W. 2012 “Multiple hypotheses Bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations,” IEEE Trans. Curc. Syst. Vid. Techn. 22 1188 - 1198
Haavisto P. , Juhola J. , Neuvo Y. 1989 “Franctional frame rate up conversion using weighted median filter,” IEEE Trans. Consum. Electron. 35 272 - 278
Wang D. , Lauzon D. 2000 “Hybrid algorithm for estimating true motion fields,” Opt. Eng. 39 2876 - 2881
Dane G. , Nguyen T. Q. 2006 “Optimal temporal interpolation filter for motion-compensated frame rate up conversion,” IEEE Trans. Imag. Proc. 15 978 - 991
Choi B. D. , Han J. W. , Kim C. S. , Ko S. J. 2006 “Frame rate up-conversion using perspective transform,” IEEE Trans. Consum. Electron. 52 975 - 982
Lee S. H. , Yang S. 2002 “Adaptive motion-compensated frame rate up-conversion,” Electron. Lett. 38 451 - 452
Han R. , Men A. 2013 “Frame rate up-conversion for high-definition video applications,” IEEE Trans. Consum. Electron. 59 229 - 236
Wang D. , Vincent A. , Blanchfield P. , Klepko R. 2010 “Motion-compensated frame rate up-conversion-part II: new algorithms for frame interpolation,” IEEE Trans. Broadc. 56 142 - 149
Juang C. L. , Chai T. T. 1996 “Motion-compensated interpolation for scan rate up-conversion,” Opt. Eng. 35 166 - 176
Liu S. , Kuo C. C. J. , Kim J. W. 2003 “Hybrid global-local motion compensated frame interpolation for low bit rate video coding,” J. Vis. Com. Imag. Repres. 14 58 - 76
Kang S. J. , Cho K. R. , Kim Y. H. 2007 “Motion compensated frame rate up-conversion using extended bilateral motion estimation,” IEEE Trans. Consum. Electron. 53 1759 - 1767
Jung H. S. , Kim U. S. , Sunwoo M. H. 2012 “Simplified frame rate up-conversion algorithm with low computational complexity,” Proc. 20th European Signal Processing Conference Bucharest, Romania 385 - 389
Orchard M. T. , Sullivan G. J. 1994 “Overlapped block motion compensation: an estimation theoretic approach,” IEEE Trans. Imag. Proc. 3 693 - 699
Zhai J. , Yu K. , Ki J. , Li S. 2005 “A low complexity motion compensated frame interpolation method,” Proc. Int. Symp. Circuits and Systems International Conference Center, Kobe, Japan 4927 - 4930
Girod B. 2000 “Efficiency analysis of multihypothesis motion-compensated prediction for video coding,” IEEE Trans. Imag. Proc. 9 173 - 183
Choi B. D. , Han J. W. , Kim C. S. , Ko S. J. 2007 “Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation,” IEEE Trans. Circ. Syst. Video. Technol. 17 407 - 416
Wang C. , Zhang L. , He Y. , Tan Y. P. 2010 “Frame rate up conversion using trilateral filtering,” IEEE Trans. Circ. Syst. Video. Technol. 20 886 - 893
Kang S. J. , Yoo S. , Kim Y. H. 2010 “Dual motion estimation for frame rate up-conversion,” IEEE Trans. Circ. Syst. Video. Technol. 20 1909 - 1914
Kim U. S. , Sunwoo M. H. 2014 “New frame rate up-conversion algorithm with low computational complexity,” IEEE Trans. Circ. Syst. Video. Technol. 24 384 - 393
Lee G. G. , Chen C. F. , Hsiao C. J. , Wu J. C. 2014 “Bi-directional trajectory tracking with variable block-size motion estimation for frame rate up-conversion,” IEEE J. Emerg. Select. Top. Circ. Syst. 4 29 - 42
Choi I. Y. , Kang Y. J. , Hong K. M. , Kim S. J. , Lee G. D. 2014 “Study on the improvement of the image analysis speed in the digital image correlation measurement system for the 3-point bend test,” J. Opt. Soc. Korea 18 523 - 530
Lim H. , Park H. W. 2011 “A symmetric motion estimation method for motion-compensated frame interpolation,” IEEE Trans. Imag. Proc. 20 3653 - 3658
Deng L. , Gao W. , Hu M. Z. , Ji Z. Z. 2005 “An efficient hardware implementation for motion estimation of AVC standard,” IEEE Trans. Cons. Electr. 51 1360 - 1366
Atitallah A. B. , Arous S. , Loukil H. , Masmoudi N. 2012 “Hardware implementation and validation of the fast variable block size motion estimation architecture for H.264/AVC,” Int. J. Electr. Commun. 66 701 - 710