Edge-preserving demosaicing method for digital cameras with Bayer-like W-RGB color filter array

KSII Transactions on Internet and Information Systems (TIIS).
2014.
Mar,
8(3):
1011-1025

- Received : August 30, 2013
- Accepted : February 19, 2014
- Published : March 31, 2014

Download

PDF

e-PUB

PubReader

PPT

Export by style

Article

Metrics

Cited by

TagCloud

A demosaicing method for a Bayer-like W-RGB color filter array (CFA) is proposed. When reproducing images from a W-RGB CFA, conventional color separation methods for W-RGB CFA are likely to cause blurring near the edges due to rough averaging using a color ratio of neighboring pixels. Moreover, these methods cannot be applied to real-life digital cameras with W-RGB CFA because the methods were proposed under an ideal situation,
W
=
R
+
G
+
B
, not a real-life situation,
W
≠
R
+
G
+
B
. To improve edge performance, we propose a method of constant color difference assumption with inversed weight, which uses information from all edge directions for interpolating all missing color channels. The proposed method calculates the correlation between W, R, G, and B to enable its application to real-life digital cameras with W-RGB CFA. Simulations were performed to evaluate the proposed method using images captured from a real-life digital camera with W-RGB CFA. Simulation results shows that we can demosaic by using the proposed algorithm compared with the conventional one in about +34.79% SNR, +11.43% PSNR, +1.54% SSIM and 14.02% S-CIELAB error. Thus, the proposed method demosaics better than the conventional methods.
w
hile the resolution of consumer digital cameras is increasing, the size of complementary metal-oxide semiconductor (CMOS) image sensors remains limited. As a result of this trend, CMOS image sensors now contain a huge number of pixels. However, the size of each pixel on the sensor is becoming increasingly smaller, and the amount of light absorbed in the same period by each unit pixel is therefore significantly decreased. This degradation of light sensitivity by each unit pixel can lead to deterioration in image quality, especially in low-light conditions.
Images taken by digital cameras that use the above-mentioned CMOS sensors in low-light conditions can have significant blurring and noise because a long exposure time is required for the aperture to gather enough light to produce acceptable images. To overcome this defect, a sensor with a higher sensitivity is required. To address this need, a White-RGB (W-RGB) color filter array (CFA), which has a greater sensitivity than a Bayer color filter array
[1]
, was developed
[2
–
6]
.
Examples of CFA patterns are shown in
Fig. 1
. Examples of the Bayer CFA pattern and a Bayer-like W-RGB CFA pattern are shown in
Figs. 1
(a) and
1
(b), respectively. A Bayer-like W-RGB CFA substitutes one green pixel on every 2×2 matrix of the Bayer CFA with a white pixel (W
_{2x,x}
) which is made by transparent resin films. To increase sensitivity, the white pixel can be penetrated by all visible light with wavelengths of 450 to 700 nm. Unlike the Bayer CFA, W pixels on the W-RGB CFA enable penetration of all color strengths, while other pixels do not. Therefore, sensitivity can be improved because more information exists in the W pixel locations of the W-RGB CFA than in G pixel locations of the Bayer CFA. Consequently, the existence of the W pixel itself improves sensitivity of the W-RGB sensors.
Examples of CFA patterns: (a) Bayer pattern, and (b) Bayer-like W-RGB pattern.
A demosaicing method for W-RGB CFA is required to reproduce full-color images from a W-RGB CFA. Therefore a new method that performs demosaicing from Bayer-like W-RGB CFA to full-color image is proposed in this paper.
W_{2,1}
=
R_{2,2}
+
G_{1,2}
+
B_{1,1}
, which is referred to in
Fig. 1
.(b). These methods estimate R, G, and B values at the pixel location W based on the color ratio of neighboring pixels using Eq. 1.
R_{average}
,
G_{average}
, and
B_{average}
represent the average values of R, G, and B located near the target pixel W.
R_{W}
,
G_{W}
and
B_{W}
represent the estimated R, G, and B values located at target pixel W. The method assumes an ideal situation (
W_{2,1}
=
R_{2,2}
+
G_{1,2}
+
B_{1,1}
), where R, G, and B have the same weight, 1. Their second method
[5]
considers edge directions for avoiding the blur problem of the method
[4]
by estimating
R_{W}
,
G_{W}
, and
B_{W}
values only if an edge is not detected at pixel location W. If it is not detected, only
G_{W}
at location W is estimated using Eq. 1. Performance near edges is poor even though the method
[5]
considers edge existence.
Ideally, W pixels on the W-RGB CFA can be penetrated by all R, G, and B elements of light; therefore, the method assumes the equation
W
=
R
+
G
+
B
is satisfied. However, the equation of
W
=
R
+
G
+
B
is not satisfied in a real-life situation. Consequently, the methods of
[4]
and
[5]
cannot be properly applied to real-life digital cameras with W-RGB CFA. In this paper, we propose a method that improves edge performance of conventional methods through a constant color difference assumption with inversed weight, which uses information from all edge directions for interpolating all missing color channels. The proposed method calculates the correlation between W, R, G, and B to enable its application to real-life digital cameras with W-RGB CFA.
W_{2,1}
≠
R_{2,2}
+
G_{1,2}
+
B_{1,1}
), which estimates the value of G within a 5×5 reference matrix.
By using an edge strength filter, an edge direction map can be built using Eq. 3, where
S_{i,j}
is the approximate edge strength at pixel location
(i, j)
, and
D
_
X_{i,j}
is the total cost toward
X
degree at pixel location
(i, j)
. By comparing the adjacent edge strength costs themselves, the target pixel
(i, j)
is labeled 45° or 90° if
D
_
45_{i,j}
or
D
_
90_{i,j}
is larger than
D
_
135_{i,j}
or
D
_
180_{i,j}
and vice versa. Two edge direction maps are made by this method. One map has information for 45° or 135° while the other has information for 90° or 180°.
W
=
R
+
G
+
B
; however, the ideal situation and the real-life situation are different. We assume
W
=
αR
+
βG
+
γB
to estimate the correlation among W, R, G, and B in a 2×2 matrix. To find the best matched solution of
α
,
β
and
γ
, we use the method of least squares (LS)
[8]
in a non-singular situation. We can write the non-singular LS expression as Eq. 4:
In other words, we can express Eq. 4 as follows:
where
After further mathematical manipulation of Eqs. 5 and 6, we can obtain a solution as follows:
where inv(∙) indicates the function of inverse calculation. Candidates for calculating correlation information among W, R, G, and B using Eq. 4 should not be saturated for precise calculation. Namely the candidate values of the W, R, G, and B should be in the range of 0 to 255. Each time we take a picture, circumstances (scene, amount of light absorbed, etc.) change; therefore, correlation coefficients
α
,
β
, and
γ
are renewed when each picture is taken.
where
is the temporal directional estimation result of the edge labeled X in degrees for constant color difference assumption and
is the resulting value of G at location W using the constant color difference assumption. Using Eqs. 8 and 9, G color channels on pixel location W can be estimated As a result, a W-RGB patterned image which is shown in
Fig. 1
.(b) will be converted into an image like standard Bayer pattern which is shown in
Fig. 1
.(a). With the temporal result from the section 3.3.1, the conventional demosaicing method for Bayer CFA can be adopted for reproducing full-color images; however, we will interpolate missing color channels keeping the same principles, constant color difference assumption with edge information, for preserving edge structure.
and
are temporal directional estimations used for the constant color difference assumption and
is the estimation result of the blue pixel value using the constant color difference assumption. Eqs. 10 and 11 use the information of edge direction toward 45 and 135 degrees because R pixels and B pixels are situated diagonally, like
R_{2,2}
, B
_{1,1}
,
B_{1,3}
,
B_{3,1}
and
B_{3,3}
that are referred to in
Fig. 1
.(a). Missing pixels that lie diagonally can be estimated by replacing variables of Eqs. 10 and 11; however missing pixels lying vertically and horizontally can be estimated by using Eqs. 12 and 13 which are modified versions of Eqs. 10 and 11, respectively.
where
is the updated final G pixel values at W pixel locations in an adaptive way using inversed weights, and
D_{x}
and
W_{x}
are edge strength toward
x
in degrees and inversed weights of
x
in degrees, respectively. All estimated missing colors at 3.3.1 and 3.3.2 can be updated by replacing the variables and directional suffixes. With the proposed final equation, Eq. 14, all missing channels of R, G, and B can be effectively estimated while avoiding blur; that is, images from W-RGB CFA can be demosaiced to full-color images with good results.
Bayer-like W-RGB patterned images used for the performance evaluation.
The set of ground truth images for comparison.
Resulting images from the conventional method [5] .
Resulting images from the proposed method.
The first type of simulation evaluates the accuracy of the demosaicing method by comparing the resulting images of the proposed method with ground truth images. The input images of the proposed method and the ground truth images are of the same scene captured under the same circumstances. SNR, PSNR, and SSIM were measured against ground truth images; the results are shown in
Tables 1
,
2
and
3
, respectively. To prove the performance of the proposed method, measuring SNR, PSNR, and SSIM is insufficient; therefore, we adopted the method of the S-CIELAB color difference
[10]
metric for the second simulations.
Comparisons of SNR for different methods.
Comparisons of PSNR for different methods.
Comparisons of SSIM for different methods.
The S-CIELAB color difference between two colors becomes smaller when the two colors have similar elements in L*a*b* space. Namely, the smaller the S-CIELAB color difference becomes, the better the result of the proposed method. The S-CIELAB color difference metric returns 1 for any just-noticeable error; 10 for very high error. The proposed method has the better overall averages of SNR, PSNR, SSIM and S-CIELAB color difference than those of the method
[5]
, outperforming the conventional method by 4.652 dB, 4.061 dB, 0.015 and −0.045, respectively. Also, we can also figure out that the performance of the proposed method is improved by 34.787%, 11.432%, 1.543% and 14.024% on average for SNR, PSNR, SSIM and S-CIELAB color difference measuring factors, respectively.
Comparisons of average S-CIELAB distance for different methods.
In addition, we measure the percentage of pixels exceeding 3, 5, and 10 of the S-CIELAB color differences, as shown in
Table 5
. With data from
Table 5
, we illustrate the spatial distribution of the errors in
Fig. 6
using white indicator when the S-CIELAB color difference exceeds 1 or larger, and green indicator when it exceeds 10 or larger. Each subfigure in
Fig. 6
shows the distribution of the errors of the conventional method (left part) and the proposed method (right part), respectively. For every subfigures in
Fig. 6
, we can see that the left images have more green and white indicators than the right images; therefore, we can confirm that the proposed method is superior to the conventional one in the aspect of S-CIELAB color difference.
Subjective simulation results of S-CIELAB color difference.
Spatial distribution of the errors by using S-CIELAB color difference.
As shown in all simulation results based on measuring SNR, PSNR, SSIM and S-CIELAB difference, the proposed method outperforms the conventional method.
Jongjoo Park received the B.S., and M.S. degrees in Electronics and Computer Engineering from Hanyang University, Seoul, Korea, in 2011, and 2013 respectively. Currently, he is working toward the Ph.D. degree with the Department of Electronics and Computer Engineering, Hanyang University. His current research interests include demosaicing for W-RGB CFA, denoising and deblurring under low-light condition
Jongwha Chong received the B.S. and the M.S. degree in Electronics Engineering from Hanyang University, Seoul, Korea, in 1975, and 1979 respectively and the Ph.D. degree in Electronics & Communication Engineering from Waseda University, Japan, in 1981. Since 1981, he has been a professor of the Department of Electronics Engineering, Hanyang University. From 1979 to 1980, he was a researcher in C&C Research Center of Nippon Electronic Company. From 1983 to 1984, he was a visiting researcher in the Korean Institute of Electronics & Technology. In 1986 and 2008 respectively, he was a visiting professor at the University of California, Berkeley, USA. He was the chairman of CAD & VLSI society in 1993 at Institute of the Electronic Engineers of Korea. He was the president of IEEK in 2007 and the president of KIEEE from 2009 to 2010, respectively. He is currently the Chairman of Fusion SoC Forum. His current research interests are of SoC design methodology including memory centric design and physical design automation of 3D-IC, indoor wireless communication SoC design for ranging and location, video system and power IT system.

1. Introduction

PPT Slide

Lager Image

2. Background

Two conventional methods were proposed by Hiroto Honda et al.
[4
,
5]
. Their methods assume an ideal situation,
PPT Slide

Lager Image

3. Demosaicing method for Bayer-like W-RGB CFA

Demosaicing method interpolates missing color channels for reproducing full-color images from mosaiced images using the information of reference pixels near target pixels. It is obvious that averaging reference pixels which lie across an edge structure can cause serious blur defect. Therefore, in order to preserve edge structure well when estimating missing color channels, it is important to avoid averaging between reference pixels which lie across the edge structures. In other words, it is important to utilize the information of edge structure properly for preserving edge structure when estimating missing color channels. To that end, the proposed demosaicing method starts with extracting the information of edge structure at every pixel location.
The proposed method for demosaicing the W-RGB CFA consists of four steps: (1) building an edge direction map, (2) estimating correlations among W, R, G, and B, (3) estimating missing color channels using the constant color difference assumption, and (4) updating the missing color channels previously estimated in an adaptive way. As with the conventional methods mentioned above
[4
,
5]
, the proposed method assumes a real-life situation (
- 3.1 Edge Direction Map Building

Sharpness near the object edge is a key factor for the human visual experience. To prevent the appearance of blurring near the edge, it is important to avoid averaging across it. To that end, edge information should be calculated in advance. A commonly used method of expressing edge information utilizes the differences in luminance between adjacent pixels. This method, however, is not reasonable for use in the W-RGB CFA case where only one of three color channels is available for each pixel, including the W pixel; therefore a mosaiced image which is shown in
Fig. 1
.(b) does not have complete luminance information at every pixel. Therefore, an edge strength filter
[7]
for the mosaiced image, Eq. 2, is adopted for expressing edge information.
PPT Slide

Lager Image

PPT Slide

Lager Image

- 3.2 Estimation of Pixel Correlations

The conventional method
[4
,
5]
assumes the ideal situation,
PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

- 3.3 Estimation of Missing Color Channels

In order to interpolate missing color channels from W-RGB mosaiced input image, utilization of the existing raw pixels near target locations is the most important part for preserving edge structure. Therefore, the W locations are the best target locations for interpolating missing G channel first because the G pixels lie on all diagonal directions from the target W locations in the center, referred in
Fig. 1
(b), raw Bayer-like W-RGB image. Estimation of missing color channels consists of two steps: (1) green color channel estimation at pixel location W, and (2) missing color channel estimation at every pixel location.
- 3.3.1 Green Color Channel Estimation at Pixel Location W

Using the edge direction maps and correlation coefficients previously obtained, the W pixel values are estimated into G values using the weighted color ratio with correlation coefficients, Eq. 8, and the constant color difference assumption
[9]
, Eq. 9:
PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

- 3.3.2 Missing Color Channel Estimation at Every Pixel Location

The approach to estimating the missing color channels of R, G, and B at every pixel location differs somewhat from those outlined in section 3.3.1. In the case of green channel estimation at W pixel locations, a weighted color ratio with pre-calculated correlation coefficients and a constant color difference assumption are used to separate W pixel values into R, G, and B pixel values. However, in the case of estimation at R, G, and B, other kinds of directional estimation and constant color difference assumption are used. Eqs. 10 and 11 are for estimation at B pixel values at R pixel locations.
PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

- 3.4 Update of Estimated Missing Color Channels

The next step is a color update using the correlation weight for more precise estimation near the edge. Two pixels located across a strong edge influence each other less. Therefore, the difference in edge strength will be large if the pixels are located across a strong edge. That means that pixels are inversely correlated; therefore, we utilize an inverse weight to improve performance near an edge. The final value of G at pixel location W can be obtained with Eq. 14 using pre-estimated values and inverse weights:
PPT Slide

Lager Image

PPT Slide

Lager Image

4. Simulation Results

To simulate the proposed method, images captured by a real-life digital camera with W-RGB CFA were used as input images as shown in
Fig. 2
. The ground truth images are shown in
Fig. 3
, and the simulation results of the conventional method and the proposed method are shown in
Figs. 4
and
5
, respectively. We can recognize the subjective edge performance by comparing the three sets of figures,
Fig. 3
,
4
and
5
. Two types of objective simulations were performed to evaluate the accuracy of the proposed method.
PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

PPT Slide

Lager Image

Comparisons of SNR for different methods.

PPT Slide

Lager Image

Comparisons of PSNR for different methods.

PPT Slide

Lager Image

Comparisons of SSIM for different methods.

PPT Slide

Lager Image

Comparisons of average S-CIELAB distance for different methods.

PPT Slide

Lager Image

Subjective simulation results of S-CIELAB color difference.

PPT Slide

Lager Image

PPT Slide

Lager Image

5. Conclusion

In this paper, we proposed a new method of demosaicing for W-RGB CFA. The proposed method can be applied effectively to real-life devices which have CMOS image sensors with W-RGB CFA using information of edge structure, the correlation information among W, R, G, and B, the constant color difference assumption, and adaptive weighted estimation. With the proposed method, images captured by a real-life digital camera with Bayer-like W-RGB CFA can be demosaiced to full-color images with less blur than conventional methods.
BIO

Bayer B.
1976
“Color imaging array”
United States Patent, No. 3, 971,065
Article (CrossRef Link).

Egawa Y.
,
Tanaka N.
,
Kawai N.
,
Seki H.
,
Nakao A.
,
Honda H.
,
Iida Y
,
Monoi A.
2008
"A White-RGB CFA-Patterned CMOS Image Sensor with Wide Dynamic Range"
IEEE International
in Proc. of Solid-State Circuits Conference, 2008. ISSCC 2008. Digest of Technical Papers.
February 3-7
Article (CrossRef Link).

Getman A.
,
Kim J.
,
Kim T.
2010
"Imaging system having White-RGB color filter array"
in Proc. of Image Processing, 2010 17th IEEE International Conference on
September 26-29
Article (CrossRef Link).
569 -
572

Honda H.
,
Iida Y.
,
Itoh G.
,
Egawa Y.
,
Seki H.
2007
"A novel Bayer-like WRGB color filter array for CMOS image sensors"
in Proc. of SPIE 6492, Human Vision and Electronic Imaging XII
Article (CrossRef Link).
64921J 02 -

Honda H.
,
Iida Y.
,
Egawa Y.
2007
“High sensitivity color CMOS image sensor with WRGB color filter array and color separation process using edge detection”
in Proc. of International Image Sensor Society Workshop
Article (CrossRef Link)

Shun K.
,
Rihito K.
,
Shigetoshi S.
2013
“Color Reproductivity Improvement with Additional Virtual Color Filters for WRGB Image Sensor”
in Proc. of SPIE Color Imaging XVIII: Displaying, Processing, Hardcopy and Applications
1-7, February
Article (CrossRef Link)

Pekkucuksen I.
,
Altunbasak Y.
2012
"Edge Strength Filter Based Color Filter Array Interpolation"
Image Processing, IEEE Transactions on
Article (CrossRef Link)
21
(1)
393 -
397
** DOI : 10.1109/TIP.2011.2155073**

Wolberg J.
2010
“Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments”
Springer
Article (CrossRef Link)
44 -
50

Pekkucuksen I.
,
Altunbasak Y.
2011
"Edge oriented directional Color Filter Array interpolation"
in Proc. of Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on
May 22-27
Article (CrossRef Link)
993 -
996

Johnson G.
,
Fairchild M.
2003
“Measuring images: differences, quality, and appearance”
in Proc. SPIE5007
Article (CrossRef Link).
147 -
160

Citing 'Edge-preserving demosaicing method for digital cameras with Bayer-like W-RGB color filter array
'

@article{ E1KOBZ_2014_v8n3_1011}
,title={Edge-preserving demosaicing method for digital cameras with Bayer-like W-RGB color filter array}
,volume={3}
, url={http://dx.doi.org/10.3837/tiis.2014.03.017}, DOI={10.3837/tiis.2014.03.017}
, number= {3}
, journal={KSII Transactions on Internet and Information Systems (TIIS)}
, publisher={Korean Society for Internet Information}
, author={Park, Jongjoo
and
Chong, Jongwha}
, year={2014}
, month={Mar}