Advanced
Edge-preserving demosaicing method for digital cameras with Bayer-like W-RGB color filter array
Edge-preserving demosaicing method for digital cameras with Bayer-like W-RGB color filter array
KSII Transactions on Internet and Information Systems (TIIS). 2014. Mar, 8(3): 1011-1025
Copyright © 2014, Korean Society For Internet Information
  • Received : August 30, 2013
  • Accepted : February 19, 2014
  • Published : March 28, 2014
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Jongjoo Park
Jongwha Chong

Abstract
A demosaicing method for a Bayer-like W-RGB color filter array (CFA) is proposed. When reproducing images from a W-RGB CFA, conventional color separation methods for W-RGB CFA are likely to cause blurring near the edges due to rough averaging using a color ratio of neighboring pixels. Moreover, these methods cannot be applied to real-life digital cameras with W-RGB CFA because the methods were proposed under an ideal situation, W = R + G + B , not a real-life situation, W R + G + B . To improve edge performance, we propose a method of constant color difference assumption with inversed weight, which uses information from all edge directions for interpolating all missing color channels. The proposed method calculates the correlation between W, R, G, and B to enable its application to real-life digital cameras with W-RGB CFA. Simulations were performed to evaluate the proposed method using images captured from a real-life digital camera with W-RGB CFA. Simulation results shows that we can demosaic by using the proposed algorithm compared with the conventional one in about +34.79% SNR, +11.43% PSNR, +1.54% SSIM and 14.02% S-CIELAB error. Thus, the proposed method demosaics better than the conventional methods.
Keywords
1. Introduction
w hile the resolution of consumer digital cameras is increasing, the size of complementary metal-oxide semiconductor (CMOS) image sensors remains limited. As a result of this trend, CMOS image sensors now contain a huge number of pixels. However, the size of each pixel on the sensor is becoming increasingly smaller, and the amount of light absorbed in the same period by each unit pixel is therefore significantly decreased. This degradation of light sensitivity by each unit pixel can lead to deterioration in image quality, especially in low-light conditions.
Images taken by digital cameras that use the above-mentioned CMOS sensors in low-light conditions can have significant blurring and noise because a long exposure time is required for the aperture to gather enough light to produce acceptable images. To overcome this defect, a sensor with a higher sensitivity is required. To address this need, a White-RGB (W-RGB) color filter array (CFA), which has a greater sensitivity than a Bayer color filter array [1] , was developed [2 6] .
Examples of CFA patterns are shown in Fig. 1 . Examples of the Bayer CFA pattern and a Bayer-like W-RGB CFA pattern are shown in Figs. 1 (a) and 1 (b), respectively. A Bayer-like W-RGB CFA substitutes one green pixel on every 2×2 matrix of the Bayer CFA with a white pixel (W 2x,x ) which is made by transparent resin films. To increase sensitivity, the white pixel can be penetrated by all visible light with wavelengths of 450 to 700 nm. Unlike the Bayer CFA, W pixels on the W-RGB CFA enable penetration of all color strengths, while other pixels do not. Therefore, sensitivity can be improved because more information exists in the W pixel locations of the W-RGB CFA than in G pixel locations of the Bayer CFA. Consequently, the existence of the W pixel itself improves sensitivity of the W-RGB sensors.
PPT Slide
Lager Image
Examples of CFA patterns: (a) Bayer pattern, and (b) Bayer-like W-RGB pattern.
A demosaicing method for W-RGB CFA is required to reproduce full-color images from a W-RGB CFA. Therefore a new method that performs demosaicing from Bayer-like W-RGB CFA to full-color image is proposed in this paper.
2. Background
Two conventional methods were proposed by Hiroto Honda et al. [4 , 5] . Their methods assume an ideal situation, W2,1 = R2,2 + G1,2 + B1,1 , which is referred to in Fig. 1 .(b). These methods estimate R, G, and B values at the pixel location W based on the color ratio of neighboring pixels using Eq. 1.
PPT Slide
Lager Image
Raverage , Gaverage , and Baverage represent the average values of R, G, and B located near the target pixel W. RW , GW and BW represent the estimated R, G, and B values located at target pixel W. The method assumes an ideal situation ( W2,1 = R2,2 + G1,2 + B1,1 ), where R, G, and B have the same weight, 1. Their second method [5] considers edge directions for avoiding the blur problem of the method [4] by estimating RW , GW , and BW values only if an edge is not detected at pixel location W. If it is not detected, only GW at location W is estimated using Eq. 1. Performance near edges is poor even though the method [5] considers edge existence.
Ideally, W pixels on the W-RGB CFA can be penetrated by all R, G, and B elements of light; therefore, the method assumes the equation W = R + G + B is satisfied. However, the equation of W = R + G + B is not satisfied in a real-life situation. Consequently, the methods of [4] and [5] cannot be properly applied to real-life digital cameras with W-RGB CFA. In this paper, we propose a method that improves edge performance of conventional methods through a constant color difference assumption with inversed weight, which uses information from all edge directions for interpolating all missing color channels. The proposed method calculates the correlation between W, R, G, and B to enable its application to real-life digital cameras with W-RGB CFA.
3. Demosaicing method for Bayer-like W-RGB CFA
Demosaicing method interpolates missing color channels for reproducing full-color images from mosaiced images using the information of reference pixels near target pixels. It is obvious that averaging reference pixels which lie across an edge structure can cause serious blur defect. Therefore, in order to preserve edge structure well when estimating missing color channels, it is important to avoid averaging between reference pixels which lie across the edge structures. In other words, it is important to utilize the information of edge structure properly for preserving edge structure when estimating missing color channels. To that end, the proposed demosaicing method starts with extracting the information of edge structure at every pixel location.
The proposed method for demosaicing the W-RGB CFA consists of four steps: (1) building an edge direction map, (2) estimating correlations among W, R, G, and B, (3) estimating missing color channels using the constant color difference assumption, and (4) updating the missing color channels previously estimated in an adaptive way. As with the conventional methods mentioned above [4 , 5] , the proposed method assumes a real-life situation ( W2,1 R2,2 + G1,2 + B1,1 ), which estimates the value of G within a 5×5 reference matrix.
- 3.1 Edge Direction Map Building
Sharpness near the object edge is a key factor for the human visual experience. To prevent the appearance of blurring near the edge, it is important to avoid averaging across it. To that end, edge information should be calculated in advance. A commonly used method of expressing edge information utilizes the differences in luminance between adjacent pixels. This method, however, is not reasonable for use in the W-RGB CFA case where only one of three color channels is available for each pixel, including the W pixel; therefore a mosaiced image which is shown in Fig. 1 .(b) does not have complete luminance information at every pixel. Therefore, an edge strength filter [7] for the mosaiced image, Eq. 2, is adopted for expressing edge information.
PPT Slide
Lager Image
By using an edge strength filter, an edge direction map can be built using Eq. 3, where Si,j is the approximate edge strength at pixel location (i, j) , and D _ Xi,j is the total cost toward X degree at pixel location (i, j) . By comparing the adjacent edge strength costs themselves, the target pixel (i, j) is labeled 45° or 90° if D _ 45i,j or D _ 90i,j is larger than D _ 135i,j or D _ 180i,j and vice versa. Two edge direction maps are made by this method. One map has information for 45° or 135° while the other has information for 90° or 180°.
PPT Slide
Lager Image
- 3.2 Estimation of Pixel Correlations
The conventional method [4 , 5] assumes the ideal situation, W = R + G + B ; however, the ideal situation and the real-life situation are different. We assume W = αR + βG + γB to estimate the correlation among W, R, G, and B in a 2×2 matrix. To find the best matched solution of α , β and γ , we use the method of least squares (LS) [8] in a non-singular situation. We can write the non-singular LS expression as Eq. 4:
PPT Slide
Lager Image
In other words, we can express Eq. 4 as follows:
PPT Slide
Lager Image
where
PPT Slide
Lager Image
After further mathematical manipulation of Eqs. 5 and 6, we can obtain a solution as follows:
PPT Slide
Lager Image
where inv(∙) indicates the function of inverse calculation. Candidates for calculating correlation information among W, R, G, and B using Eq. 4 should not be saturated for precise calculation. Namely the candidate values of the W, R, G, and B should be in the range of 0 to 255. Each time we take a picture, circumstances (scene, amount of light absorbed, etc.) change; therefore, correlation coefficients α , β , and γ are renewed when each picture is taken.
- 3.3 Estimation of Missing Color Channels
In order to interpolate missing color channels from W-RGB mosaiced input image, utilization of the existing raw pixels near target locations is the most important part for preserving edge structure. Therefore, the W locations are the best target locations for interpolating missing G channel first because the G pixels lie on all diagonal directions from the target W locations in the center, referred in Fig. 1 (b), raw Bayer-like W-RGB image. Estimation of missing color channels consists of two steps: (1) green color channel estimation at pixel location W, and (2) missing color channel estimation at every pixel location.
- 3.3.1 Green Color Channel Estimation at Pixel Location W
Using the edge direction maps and correlation coefficients previously obtained, the W pixel values are estimated into G values using the weighted color ratio with correlation coefficients, Eq. 8, and the constant color difference assumption [9] , Eq. 9:
PPT Slide
Lager Image
PPT Slide
Lager Image
where
PPT Slide
Lager Image
is the temporal directional estimation result of the edge labeled X in degrees for constant color difference assumption and
PPT Slide
Lager Image
is the resulting value of G at location W using the constant color difference assumption. Using Eqs. 8 and 9, G color channels on pixel location W can be estimated As a result, a W-RGB patterned image which is shown in Fig. 1 .(b) will be converted into an image like standard Bayer pattern which is shown in Fig. 1 .(a). With the temporal result from the section 3.3.1, the conventional demosaicing method for Bayer CFA can be adopted for reproducing full-color images; however, we will interpolate missing color channels keeping the same principles, constant color difference assumption with edge information, for preserving edge structure.
- 3.3.2 Missing Color Channel Estimation at Every Pixel Location
The approach to estimating the missing color channels of R, G, and B at every pixel location differs somewhat from those outlined in section 3.3.1. In the case of green channel estimation at W pixel locations, a weighted color ratio with pre-calculated correlation coefficients and a constant color difference assumption are used to separate W pixel values into R, G, and B pixel values. However, in the case of estimation at R, G, and B, other kinds of directional estimation and constant color difference assumption are used. Eqs. 10 and 11 are for estimation at B pixel values at R pixel locations.
PPT Slide
Lager Image
PPT Slide
Lager Image
PPT Slide
Lager Image
and
PPT Slide
Lager Image
are temporal directional estimations used for the constant color difference assumption and
PPT Slide
Lager Image
is the estimation result of the blue pixel value using the constant color difference assumption. Eqs. 10 and 11 use the information of edge direction toward 45 and 135 degrees because R pixels and B pixels are situated diagonally, like R2,2 , B 1,1 , B1,3 , B3,1 and B3,3 that are referred to in Fig. 1 .(a). Missing pixels that lie diagonally can be estimated by replacing variables of Eqs. 10 and 11; however missing pixels lying vertically and horizontally can be estimated by using Eqs. 12 and 13 which are modified versions of Eqs. 10 and 11, respectively.
PPT Slide
Lager Image
PPT Slide
Lager Image
- 3.4 Update of Estimated Missing Color Channels
The next step is a color update using the correlation weight for more precise estimation near the edge. Two pixels located across a strong edge influence each other less. Therefore, the difference in edge strength will be large if the pixels are located across a strong edge. That means that pixels are inversely correlated; therefore, we utilize an inverse weight to improve performance near an edge. The final value of G at pixel location W can be obtained with Eq. 14 using pre-estimated values and inverse weights:
PPT Slide
Lager Image
where
PPT Slide
Lager Image
is the updated final G pixel values at W pixel locations in an adaptive way using inversed weights, and Dx and Wx are edge strength toward x in degrees and inversed weights of x in degrees, respectively. All estimated missing colors at 3.3.1 and 3.3.2 can be updated by replacing the variables and directional suffixes. With the proposed final equation, Eq. 14, all missing channels of R, G, and B can be effectively estimated while avoiding blur; that is, images from W-RGB CFA can be demosaiced to full-color images with good results.
4. Simulation Results
To simulate the proposed method, images captured by a real-life digital camera with W-RGB CFA were used as input images as shown in Fig. 2 . The ground truth images are shown in Fig. 3 , and the simulation results of the conventional method and the proposed method are shown in Figs. 4 and 5 , respectively. We can recognize the subjective edge performance by comparing the three sets of figures, Fig. 3 , 4 and 5 . Two types of objective simulations were performed to evaluate the accuracy of the proposed method.
PPT Slide
Lager Image
Bayer-like W-RGB patterned images used for the performance evaluation.
PPT Slide
Lager Image
The set of ground truth images for comparison.
PPT Slide
Lager Image
Resulting images from the conventional method [5].
PPT Slide
Lager Image
Resulting images from the proposed method.
The first type of simulation evaluates the accuracy of the demosaicing method by comparing the resulting images of the proposed method with ground truth images. The input images of the proposed method and the ground truth images are of the same scene captured under the same circumstances. SNR, PSNR, and SSIM were measured against ground truth images; the results are shown in Tables 1 , 2 and 3 , respectively. To prove the performance of the proposed method, measuring SNR, PSNR, and SSIM is insufficient; therefore, we adopted the method of the S-CIELAB color difference [10] metric for the second simulations.
Comparisons of SNR for different methods.
PPT Slide
Lager Image
Comparisons of SNR for different methods.
Comparisons of PSNR for different methods.
PPT Slide
Lager Image
Comparisons of PSNR for different methods.
Comparisons of SSIM for different methods.
PPT Slide
Lager Image
Comparisons of SSIM for different methods.
The S-CIELAB color difference between two colors becomes smaller when the two colors have similar elements in L*a*b* space. Namely, the smaller the S-CIELAB color difference becomes, the better the result of the proposed method. The S-CIELAB color difference metric returns 1 for any just-noticeable error; 10 for very high error. The proposed method has the better overall averages of SNR, PSNR, SSIM and S-CIELAB color difference than those of the method [5] , outperforming the conventional method by 4.652 dB, 4.061 dB, 0.015 and −0.045, respectively. Also, we can also figure out that the performance of the proposed method is improved by 34.787%, 11.432%, 1.543% and 14.024% on average for SNR, PSNR, SSIM and S-CIELAB color difference measuring factors, respectively.
Comparisons of average S-CIELAB distance for different methods.
PPT Slide
Lager Image
Comparisons of average S-CIELAB distance for different methods.
In addition, we measure the percentage of pixels exceeding 3, 5, and 10 of the S-CIELAB color differences, as shown in Table 5 . With data from Table 5 , we illustrate the spatial distribution of the errors in Fig. 6 using white indicator when the S-CIELAB color difference exceeds 1 or larger, and green indicator when it exceeds 10 or larger. Each subfigure in Fig. 6 shows the distribution of the errors of the conventional method (left part) and the proposed method (right part), respectively. For every subfigures in Fig. 6 , we can see that the left images have more green and white indicators than the right images; therefore, we can confirm that the proposed method is superior to the conventional one in the aspect of S-CIELAB color difference.
Subjective simulation results of S-CIELAB color difference.
PPT Slide
Lager Image
Subjective simulation results of S-CIELAB color difference.
PPT Slide
Lager Image
Spatial distribution of the errors by using S-CIELAB color difference.
As shown in all simulation results based on measuring SNR, PSNR, SSIM and S-CIELAB difference, the proposed method outperforms the conventional method.
5. Conclusion
In this paper, we proposed a new method of demosaicing for W-RGB CFA. The proposed method can be applied effectively to real-life devices which have CMOS image sensors with W-RGB CFA using information of edge structure, the correlation information among W, R, G, and B, the constant color difference assumption, and adaptive weighted estimation. With the proposed method, images captured by a real-life digital camera with Bayer-like W-RGB CFA can be demosaiced to full-color images with less blur than conventional methods.
BIO
Jongjoo Park received the B.S., and M.S. degrees in Electronics and Computer Engineering from Hanyang University, Seoul, Korea, in 2011, and 2013 respectively. Currently, he is working toward the Ph.D. degree with the Department of Electronics and Computer Engineering, Hanyang University. His current research interests include demosaicing for W-RGB CFA, denoising and deblurring under low-light condition
Jongwha Chong received the B.S. and the M.S. degree in Electronics Engineering from Hanyang University, Seoul, Korea, in 1975, and 1979 respectively and the Ph.D. degree in Electronics & Communication Engineering from Waseda University, Japan, in 1981. Since 1981, he has been a professor of the Department of Electronics Engineering, Hanyang University. From 1979 to 1980, he was a researcher in C&C Research Center of Nippon Electronic Company. From 1983 to 1984, he was a visiting researcher in the Korean Institute of Electronics & Technology. In 1986 and 2008 respectively, he was a visiting professor at the University of California, Berkeley, USA. He was the chairman of CAD & VLSI society in 1993 at Institute of the Electronic Engineers of Korea. He was the president of IEEK in 2007 and the president of KIEEE from 2009 to 2010, respectively. He is currently the Chairman of Fusion SoC Forum. His current research interests are of SoC design methodology including memory centric design and physical design automation of 3D-IC, indoor wireless communication SoC design for ranging and location, video system and power IT system.
References
Bayer B. 1976 “Color imaging array” United States Patent, No. 3, 971,065 Article (CrossRef Link).
Egawa Y. , Tanaka N. , Kawai N. , Seki H. , Nakao A. , Honda H. , Iida Y , Monoi A. 2008 "A White-RGB CFA-Patterned CMOS Image Sensor with Wide Dynamic Range" IEEE International in Proc. of Solid-State Circuits Conference, 2008. ISSCC 2008. Digest of Technical Papers. February 3-7 Article (CrossRef Link).
Getman A. , Kim J. , Kim T. 2010 "Imaging system having White-RGB color filter array" in Proc. of Image Processing, 2010 17th IEEE International Conference on September 26-29 Article (CrossRef Link). 569 - 572
Honda H. , Iida Y. , Itoh G. , Egawa Y. , Seki H. 2007 "A novel Bayer-like WRGB color filter array for CMOS image sensors" in Proc. of SPIE 6492, Human Vision and Electronic Imaging XII Article (CrossRef Link). 64921J 02 -
Honda H. , Iida Y. , Egawa Y. 2007 “High sensitivity color CMOS image sensor with WRGB color filter array and color separation process using edge detection” in Proc. of International Image Sensor Society Workshop Article (CrossRef Link)
Shun K. , Rihito K. , Shigetoshi S. 2013 “Color Reproductivity Improvement with Additional Virtual Color Filters for WRGB Image Sensor” in Proc. of SPIE Color Imaging XVIII: Displaying, Processing, Hardcopy and Applications 1-7, February Article (CrossRef Link)
Pekkucuksen I. , Altunbasak Y. 2012 "Edge Strength Filter Based Color Filter Array Interpolation" Image Processing, IEEE Transactions on Article (CrossRef Link) 21 (1) 393 - 397    DOI : 10.1109/TIP.2011.2155073
Wolberg J. 2010 “Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments” Springer Article (CrossRef Link) 44 - 50
Pekkucuksen I. , Altunbasak Y. 2011 "Edge oriented directional Color Filter Array interpolation" in Proc. of Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on May 22-27 Article (CrossRef Link) 993 - 996
Johnson G. , Fairchild M. 2003 “Measuring images: differences, quality, and appearance” in Proc. SPIE5007 Article (CrossRef Link). 147 - 160