Advanced
Demosaicing Method for Digital Cameras with White-RGB Color Filter Array
Demosaicing Method for Digital Cameras with White-RGB Color Filter Array
ETRI Journal. 2016. Mar, 38(1): 164-173
Copyright © 2016, Electronics and Telecommunications Research Institute (ETRI)
  • Received : November 27, 2014
  • Accepted : October 14, 2015
  • Published : March 01, 2016
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Jongjoo Park
Euee Seon Jang
Jong-Wha Chong

Abstract
Demosaicing, or color filter array (CFA) interpolation, estimates missing color channels of raw mosaiced images from a CFA to reproduce full-color images. It is an essential process for single-sensor digital cameras with CFAs. In this paper, a new demosaicing method for digital cameras with Bayer-like W-RGB CFAs is proposed. To preserve the edge structure when reproducing full-color images, we propose an edge direction–adaptive method using color difference estimation between different channels, which can be applied to practical digital camera use. To evaluate the performance of the proposed method in terms of CPSNR, FSIM, and S-CIELAB color distance measures, we perform simulations on sets of mosaiced images captured by an actual prototype digital camera with a Bayer-like W-RGB CFA. The simulation results show that the proposed method demosaics better than a conventional one by approximately +22.4% CPSNR, +0.9% FSIM, and +36.7% S-CIELAB distance.
Keywords
I. Introduction
While the resolutions of digital cameras continue to increase, the size of complementary metal–oxide–semiconductor (CMOS) image sensors remains limited. As a result, CMOS image sensors now contain a very large number of pixels; however, the sizes of sensor elements within a color filter array (CFA) [1] are becoming increasingly smaller, and the amount of light absorbed in a given period by each element is therefore significantly decreased. This degradation of light sensitivity can lead to deterioration in image quality — something that is particularly evident in images shot in low-light conditions.
Images captured in low-light conditions by a digital camera that uses a CMOS sensor can be affected by significant blurring and noise because a long exposure time is required for the aperture to gather enough light to produce acceptable images. To overcome this defect, a sensor with a higher sensitivity is required. To address this need, a White-RGB (W-RGB) CFA, which has greater sensitivity than a conventional Bayer CFA, was developed [2] .
Examples of CFA patterns are shown in Fig. 1 . Figures 1(a) and 1(b) are examples of a conventional Bayer CFA pattern and a Bayer-like W-RGB CFA pattern, respectively. A Bayer-like W-RGB CFA substitutes one green filter array in every 2 × 2 matrix of a Bayer CFA with a white filter array ( W 2x,x ), which is made from transparent resin films. These white filter arrays can be penetrated by all visible light with wavelengths of 450 nm to 700 nm. In other words, the white filter arrays in a W-RGB CFA enable penetration of all color intensities (red, green, and blue), while other pixels do not. Consequently, the existence of the white filter arrays themselves improves the sensitivity of the WRGB sensors [2] , [3] . This fact is verified in Fig. 2 , which describes the transmittance (sensitivity) of each filter array in the Bayer-like W-RGB CFA used in the simulation section. To compare the sensitivity of the W-RGB CFA with that of the Bayer, we depict a graph of average sensitivity as shown in Fig. 3 . The average transmittance was improved by +51.04% by replacing one green filter array with a white filter array in a unit 2 × 2 matrix. As a result, we can say that the sensitivity of white filter arrays is higher than that of other filter arrays. The outstanding sensitivity of the W-RGB CFA was proven using the figures described above. Now, we will turn our attention to the utilization of the W-RGB CFA.
PPT Slide
Lager Image
Examples of CFA patterns: (a) Bayer pattern and (b) Bayer-like W-RGB pattern.
PPT Slide
Lager Image
Spectral transmittance (sensitivity) of each filter array (white, red, green, and blue).
PPT Slide
Lager Image
Average spectral transmittance of Bayer-like W-RGB CFA and Bayer CFA.
To reproduce full-color images by estimating missing color channels of raw W-RGB mosaiced images from a highly sensitive W-RGB CFA, a new demosaicing method is required. Therefore, a new method that performs demosaicing with a Bayer-like W-RGB CFA is proposed herein.
The remainder of this paper is organized as follows. Related works that address W-RGB are introduced in Section II. In Section III, the details of the proposed demosaicing method are described. Some simulation results of the proposed method are presented in Section IV, and our conclusions are reported in Section V.
II. Related Works
Many demosaicing methods for the conventional Bayer CFA have been proposed to date; however, few methodologies for handling W-RGB have been suggested, because W-RGB research is still in its early stages. Conventional methods for W-RGB CFA can be classified into three categories — color separation, training, and virtual color filter. The color separation method proposed by Hiroto Honda and others estimates missing red, green, and blue channels at white sample locations by using the color ratios of neighboring pixels [3] . They first assume that the value of a white raw sample is the same as the sum of the red, green, and blue sample values at the same pixel location; that is, for ( i , j ), we have W i, j = R i, j + G i, j + B i, j . Using this, they also suppose that the value of a raw white sample is similar to the sum of the neighboring red, green, and blue sample values. For example, an equation such as W 2,1 R 2,2 + G 1,2 + B 1,1 , will also be satisfied. With this supposition, they estimate missing red, green, and blue channels at white sample locations from the following equations:
(1) { R W =W× R average ( R average + G average + B average ) , G W =W× G average ( R average + G average + B average ) , B W =W× B average ( R average + G average + B average ) ,
where R average , G average , and B average represent the average intensities of red, green, and blue located near the white target pixels, respectively; and R W , G W , and B W represent the estimated red, green, and blue channels at the white locations, respectively. This method is ineffective when edges occur near white target pixels; therefore, an edge-adaptive method was proposed for avoiding blur defects near target white pixels [4] . The edge-adaptive color separation method interpolates three channels at white target locations if edges are not detected; otherwise, the method interpolates only green channels at the white samples to avoid a serious blur effect. Although the edge-adaptive method verifies the existence of edges to avoid blur, it remains insufficient for avoiding blur. In addition, the assumption that W i, j = R i, j + G i, j + B i, j is not satisfied in real-life situations.
The training method predefines logical functions for estimating missing red, green, and blue channels at white locations [5] . By comparing the resulting values from the predefined functions with the resulting measured values from a real device, the pixel values of the missing red, green, and blue can be estimated. However, the means of defining logical functions can be challenging.
The virtual color filter method extracts missing color channels from the white spectrum in the wavelengths of 400 nm to 700 nm, the visible light region, using predefined virtual color filter models [6] . To build these models, an image of a reference color spectrum board, which is captured by a W-RGB CFA, is required. This method is the simplest method for W-RGB CFA; however, it is found to be inadequate for application to a real device, because it requires extra steps to set up parameters for every camera.
Conventional methods that deal with W-RGB CFA remain poorly understood, because as mentioned, related research is still in its early stages. Therefore, in this paper, we propose a detailed method for demosaicing W-RGB images that preserves edge structures better than conventional methods.
III. Proposed Demosaicing Method
It is obvious that averaging pixels that lie across an edge structure can cause a blur defect. Therefore, when edge-distribution information is effectively utilized to avoid averaging across edges, the edge performance of demosaicing can significantly increase. The proposed CFA interpolation method will be described using information regarding edge distribution in the following sections. The method consists of the following three main steps: (a) extraction of edge information; (b) estimation of the correlation among white, red, green, and blue samples; and (c) interpolation of missing color channels using pre-extracted edge and correlation information.
The notation of a pixel at ( i , j ) in a Bayer-like W-RGB image is represented by either ( W i, j , r i, j , g i, j , b i, j ), ( R i, j , g i, j , b i, j , w i, j ), ( r i, j , G i, j , b i, j , w i, j ), or ( r i, j , g i, j , B i, j , w i, j ), where capitalized W i, j , R i, j , G i, j , and B i, j denote the already known white, red, green, and blue samples from the Bayer-like W-RGB. Notations written in lowercase letters, w i, j , r i, j , g i, j , and b i, j , denote unknown values of corresponding color channels in the image. The temporal estimates of w i, j , r i, j , g i, j , and b i, j are denoted as
( w ^ i,j , r ^ i,j , g ^ i,j , b ^ i,j )
and
( w ⌣ i,j , r ⌣ i,j , g ⌣ i,j , b ⌣ i,j );
the final estimates of r i, j , g i, j , and b i, j are denoted as
r ˜ i,j , g ˜ i,j ,
and
b ˜ i,j ,
respectively, for clear explanation. In addition, we predefine the directional representations used throughout the paper. To interpolate avoiding averaging across edges, we use twelve directional representatives, as illustrated in Fig. 4 — north (N), south (S), east (E), west (W), northeast (NE), northwest (NW), southeast (SE), southwest (SW), diagonal 1 (D1), diagonal 2 (D2), vertical (V), and horizontal (H).
PPT Slide
Lager Image
Twelve directional representations used for proposed method.
- 1. Extraction of Edge Information
Sharpness near object edges plays an important role in the human visual experience when watching and capturing photographs. As mentioned above, to prevent the appearance of blur near the edges when reproducing a full-color image from a mosaiced image, it is important to avoid averaging across edge structures. To that end, edge information should be calculated in advance. A commonly used method of expressing edge information utilizes the differences in luminance between adjacent pixels. This method, however, is not reasonable for use in the W-RGB case in which only one of four color channels is available for each pixel location, including the white sample. Namely, a mosaiced image does not have complete luminance information at every pixel; therefore, we should utilize the information in the available color channels. To express edge information in this situation, an edge strength filter for a mosaiced image is used [7] . The edge strength at pixel location (2, 2), referenced in Fig. 1(b) , for example, can be calculated using the following equation:
(2) S 2,2 =| W 1,1 − W 3,3 |+| W 1,3 − W 3,1 |+| R 1,2 − R 3,2 |+| B 2,1 − B 2,3 |,
where S 2,2 denotes the edge strength at pixel location (2, 2). This edge strength filter is reasonable for mosaiced images because it only compares the pixel values that belong to the same color channel for extracting edge strength information. The edge strengths at the other locations can be calculated in the same way by replacing W , R , and B according to the structure of the W-RGB pattern. The resulting map using (2) contains only the edge strength information at every pixel location and does not contain the edge direction information. Therefore, to extract the edge direction information from a pre-calculated edge strength map, we use the following equations:
(3)          E D(i,j) ={ D1    if  D D1(i,j) > D D2(i,j) , D2    otherwise,          E P(i,j) ={ V   if  D V(i,j) > D H(i,j) , H   otherwise, [ D D1(i,j) = ∑ m=−2 1 ( ∑ n=−1 2 ( S i+m−1,j+n − S i+m,j+n−1 ) ) , D D2(i,j) = ∑ m=−2 1 ( ∑ n=−2 1 ( S i+m,j+n − S i+m−1,j+n+1 ) )  , [ D V(i,j) = ∑ m=−2 2 ( ∑ n=−2 1 ( S i+m,j+n − S i+m,j+n+1 ) ), D H(i,j) = ∑ m=−2 1 ( ∑ n=−2 2 ( S i+m,j+n − S i+m+1,j+n ) ),
where E D(i, j) and E P(i, j) denote the edge directions at location ( i , j ), and D x(i, j) is the total cost toward the x -direction at location ( i , j ). By comparing the adjacent edge strength costs that lie on the corresponding direction, the target location ( i , j ) is labeled D1 or V if D D1(i, j) or D V(i, j) is larger than D D2(i, j) or D H(i, j) , and vice versa. Note that the two types of edge direction maps ( E D(i, j) and E P(i, j) ) are built by (3); one of the map types, E D(i, j) , has the edge direction information in diagonal directions (Dl or D2) at ( i , j ), while the other, E P(i, j) , has it in perpendicular directions (H or V).
- 2. Estimation of Correlation among Raw Color Channels
As mentioned in Section II, conventional color separation methods assume the ideal situation, W i, j = R i, j + G i, j + B i, j at ( i , j ); however, in real-life situations, this assumption is not satisfied. Therefore, we assume that white, red, green, and blue samples have a linear relationship, W = αR + βG + γB , in a 2 × 2 unit matrix. To find the optimal solution for α , β , and γ , we use the method of least squares (LS) in a non-singular situation [8] . We can simply write a non-singular LS expression as follows:
(4) arg  min α,β,γ { ∑ W​,R,G,B ( W−(αR+βG+γB) ) 2 }.
In other words, we can simply rewrite (4) as follows:
(5) Ax=b,
where
(6) A=[ R 1 G 1 B 1 R 2 G 2 B 2 ⋮ ⋮ ⋮ R N G N B N ], x=[ α β γ ], b=[ W 1 W 2 ⋮ W N ].
After further mathematical manipulation of (5) and (6), we can obtain a simple optimal solution as follows:
(7) Ax=b,  A T Ax= A T b, x=inv ( A T A) AT b,
where inv(∙) indicates an inverse calculation function. Candidates for calculating the correlation among white, red, green, and blue samples for (6) should not be saturated for precise calculation. Specifically, the candidate samples should be in the range of 0 to 255. Each time we take a picture, circumstances such as the scene and amount of light absorbed change; therefore, correlation coefficients α , β , and γ are renewed when each picture is taken.
The edge strength map, the edge direction maps, and correlation coefficients obtained in Sections III-1 and III-2 are used in the following steps. In the following section, we describe the method for interpolating missing color channels.
Figure 5(a) illustrates the workflow of the interpolation method with four sequential processes — (A) missing green channel interpolation at white locations; (B) missing red and blue channel interpolation at blue and red locations; (C) missing green channel interpolation at red and blue locations; and (D) missing red and blue channel interpolation at green and white locations.
PPT Slide
Lager Image
Workflow and spatial arrangement of proposed demosaicing method’s temporal results: (a) workflow of proposed method; (b) input raw (mosaiced) image from Bayer-like W-RGB CFA (Point A); (c) result of green channel interpolation at white location (Point B); (d) result of green channel interpolation at red and blue locations (Point C); (e) result of red and blue channel interpolation at blue and red locations (Point D); and (f) final result of red and blue channel interpolation at green and white locations (Point E).
- 3. Interpolation of Missing Color Channels
The most important aspect of interpolating missing color channels from a mosaiced input image is the utilization of the existing samples near target locations. Therefore, the white sample locations represent the best target locations for first interpolating the green channel because the green samples lie in the two opposite directions (D1 and D2) from the target white sample locations in the center, referenced in Fig. 1(b) , of a raw Bayer-like W-RGB image.
A. Green Channels Interpolation at White Samples Locations
The first step of the proposed demosaicing method for missing green channels at white sample locations starts with estimating the color differences between different channels at a target location ( i , j ). The estimation of the color channel differences of the white–green diagonal can be achieved as follows:
(8) { Δ C g,w,(i,j) D1 = g ⌢ i,j D1 − W i,j         if  E D(i,j) =D1, Δ C g,w,(i,j) D2 = g ⌢ i,j D2 − W i,j         if  E D(i,j) =D2,
where
Δ C g,w,(i,j) D1
and
Δ C g,w,(i,j) D2
denote the color channel differences between green and white values at white sample locations, ( i , j ). At the target locations, we already know the values of the white samples from the raw mosaiced input image, whereas for the green channels we do not know the values; therefore, we use the already known term of W i, j and the unknown term of g i, j . In (8), it is required to estimate the unknown
g ^ i,j x ,
the green channel estimator, at white target locations to calculate the color channel difference. Therefore, we first use a Laplacian interpolation filter [9] to estimate green channels at white locations. By using a Laplacian interpolation filter, green channels at white sample locations can be achieved as follows:
(9) { g ⌢ i,j D1 = G i−1,j+1  +  G i+1,j−1 2 + 2× W i,j  −  W i−2,j+2  − W i+2,j−2 4    if   E D(i,j)  =D1, g ⌢ i,j D2 = G i−1,j−1  +  G i+1,j+1 2 + 2× W i,j  −  W i−2,j−2  −  W i+2,j+2 4    if   E D(i,j)  =D2,
where
g ^ i,j D1
and
g ^ i,j D2
denote the green channel estimators at ( i , j ) when the target pixel is labeled D1 and D2, respectively. In this case, we consider only the diagonal directions, D1 and D2, because green color filter elements lie diagonally from the white color filter elements in the center, as referenced in Fig. 1(b) .
Equation (8) limits the scope to only pixel location ( i , j ); therefore, we can extend the equations for including adjacent pixels along the same direction. In this way, (8) can be modified as follows:
(10) { Δ C ′ g,w,(i,j) D1  = Δ C g,w,(i−1,j+1) D1 N 1 + Δ C g,w,(i,j) D1 N 2 + Δ C g,w,(i+1,j−1) D1 N 3 if  E D(i,j)  =D1, Δ C ′ g,w,(i,j) D2  = Δ C g,w,(i−1,j−1) D2 N 1 + Δ C g,w,(i,j) D2 N 2 + Δ C g,w,(i+1,j+1) D2 N 3 if  E D(i,j)  =D2,
where
ΔC ' g,w,(i,j) x
denotes the extended color channel difference estimation at white sample locations, and Nx denotes a normalizing factor. Using existing color samples and estimators and by applying a Laplacian interpolation filter, we can rewrite (10) as follows:
(11) { Δ C g,w,(i,j) D1  = G i−1,j+1 − w ⌢ i−1,j+1 D1 4 + g ⌢ i,j D1 − W i,j 2 + G i+1,j−1 − w ⌢ i+1,j−1 D1 4 if  E D(i,j)  = D1, Δ C g,w,(i,j) D2  = G i−1,j−1 − w ⌢ i−1,j−1 D2 4 + g ⌢ i,j D2 − W i,j 2 + G i+1,j+1 − w ⌢ i+1,j+1 D2 4 if  E D(i,j)  = D2.
Each element in (11) is normalized by the distance between their operands. The extended equations of the color channel differences estimation, shown in (11), are required to estimate
w ^ i,j x
, which is a white channel estimator at an adjacent green pixel location near a target white sample location; this is despite the fact that white channels are unnecessary for a full-color image using the RGB color space. This white channel estimator is only used for temporal reasons. When we estimate the white channel estimator,
w ^ i,j x
, at green sample locations, the red and blue channels at those locations are missing while the green samples are already known. Therefore, we must initially set up an equation for a white channel estimator at a green location by using the relationship among the adjacent existing red, blue, and green samples. The equation for a white channel estimator at a target green sample location ( i , j ) using the pre-calculated correlation coefficients, α , β , and γ , can be written as follows:
(12) w ⌢ i,j =α( R i,j−1 + R i,j+1 2 )+β( G i,j )+γ( B i−1,j + B i+1,j 2 ).
By transposing the equations of the color difference estimation as shown in (8), the missing green channels at the white locations can be estimated using the following equation:
(13) g ⌣ i,j = W i,j +Δ C g,w,(i,j) ′ .
Combining (9), (11), (12), and (13), we can acquire equations for an initially interpolated green channel at a white pixel location ( i , j ) as follows:
(14) { g ⌣ i,j D1 = W i,j  + G i−1,j+1 − w ⌢ i−1,j+1 4 + g ⌢ i,j D1 − W i,j 2 + G i+1,j−1 − w ⌢ i+1,j−1 4 if  E D(i,j)  = D1, g ⌣ i,j D2 = W i,j  + G i−1,j−1 − w ⌢ i−1,j−1 4 + g ⌢ i,j D2 − W i,j 2 + G i+1,j+1 − w ⌢ i+1,j+1 4 if  E D(i,j)  = D2.
In (14), we do not apply direction-adaptive weights. The performance of the interpolation can be improved when we apply directional weights from all directions — N, S, E, W, NE, NW, SE, and SW. To build direction-adaptive weights, we utilize the fact that two pixels located across a strong edge influence each other less. This fact indicates that the difference in edge strength will be large when the pixels are located across a strong edge. In other words, the pixels are inversely correlated; therefore, we build inverse weights to improve the performance near edge structures. Finally, the equations for interpolating missing green channels at white pixel locations,
g ˜ i,j
, can be modified using direction-adaptive weights as follows:
(15) Δ C ˜ g,w,(i,j)  =  Δ C ′ g,w,(i,j)  × A +( I NE Δ C g,w,(i+1,j−1) D1  +  I NW Δ C g,w,(i−1,j−1) D2 + I SW Δ C g,w,(i−1,j+1) D1 + I SE Δ C g,w,(i+1,j+1) D2 ) ×( (1−A) ( I NE + I NW + I SW + I SE ) ),
(16) g ˜ i,j = W i,j +Δ C ˜ g,w,(i,j) = W i,j +( g ⌣ i,j − W i,j )×A+ [ I NE ( G i+1,j−1 − w ⌣ i+1,j−1 )     + I NW ( G i−1,j−1 − w ⌣ i−1,j−1 )+ I SW ( G i−1,j+1 − w ⌣ i−1,j+1 )     + I SE ( G i+1,j+1 − w ⌣ i+1,j+1 ) ]×[ (1−A) ( I NE + I NW + I SW + I SE ) ],
where coefficient A denotes an interpolation contribution at ( i , j ). The green and white channel estimators at white and green sample locations can be calculated by
(17) g ⌣ i,j = I NE G i+1,j−1 + I NW G i−1,j−1 + I SW G i−1,j+1 + I SE G i+1,j+1 ( I NE + I NW + I SW + I SE ) + I NE ( W i,j − W i+2,j−2 )+ I NW ( W i,j − W i−2,j−2 ) 2( I NE + I NW + I SW + I SE ) + I SW ( W i,j − W i−2,j+2 )+ I SE ( W i,j − W i+2,j+2 ) 2( I NE + I NW + I SW + I SE ) , w ⌣ i,j  = α { I N R i−1,j  +  I S R i+1,j ( I N + I S ) }+ β( G i,j ) +γ { I W B i,j−1  +  I E B i,j+1 ( I W + I E ) ​ }.
The direction-adaptive weights can be calculated as follows:
(18) { K NE =  | S i,j − S i+1,j−1 |+| S i+1,j−1 − S i+2,j−2 | K NW =  | S i,j − S i−1,j−1 |+| S i−1,j−1 − S i−2,j−2 | K SW =  | S i,j − S i−1,j+1 |+| S i−1,j+1 − S i−2,j+2 | K SE =  | S i,j − S i+1,j+1 |+| S i+1,j+1 − S i+2,j+2 | ,    { K N =  | S i,j − S i,j−1 |+| S i,j−1 − S i,j−2 | K S =  | S i,j − S i,j+1 |+| S i,j+1 − S i,j+2 | K E =  | S i,j − S i+1,j |+| S i+1,j − S i+2,j | K W =  | S i,j − S i−1,j |+| S i−1,j − S i−2,j | ,
where
{ I NE = K NW × K SW × K SE I NW = K NE × K SW × K SE I SW = K NE × K NW × K SE I SE = K NE × K NW × K SW   ,                 { I N = K S × K E × K W I S = K N × K E × K W I E = K N × K S × K W I W = K N × K S × K E .
The inversely correlated weight I NE , for example, can be achieved by multiplying three temporal weights, K NW , K SW , and K SE , which have different directional subscripts to that of I NE . The temporal weight Kx can be calculated by adding edge strength differences over a local window within the corresponding directions. Using the final equations, (15), (16), and (17), we can interpolate the green channel at the white sample locations.
B. Red and Blue Channel Interpolation at Blue and Red Sample Locations
The results of the previous section, the interpolation of the missing green channel at white sample locations, are illustrated in Fig. 5(c) . As shown in the corresponding figures of the temporal arrangement, the missing green channels at white sample locations are all filled; therefore, we can operate a similar process to that outlined in Section III-3-A by replacing the corresponding color notations and directional superscripts for interpolating missing red and blue channels at blue and red sample locations. We choose the step of red and blue channel interpolation at blue and red locations for the second step depending on the structure of the temporal arrangement. In other words, red and blue samples from the temporal arrangement lie in both opposite directions (D1 and D2) from the target blue and red sample locations in the center. The process of interpolating missing red channels at blue sample locations, as a representative example, begins with equations similar to (8) as follows:
(19) { Δ C r,b,(i,j) D1 = r ⌢ i,j D1 − B i,j         if  E D(i,j) =D1, Δ C r,b,(i,j) D2 = r ⌢ i,j D2 − B i,j         if  E D(i,j) =D2.
By applying the same principles of the previous section, the missing red channel interpolation at blue sample locations can be achieved using the following equations:
(20) r ˜ i,j   = B i,j +Δ C ˜ r,b,(i,j) = B i,j +( r ⌣ i,j − B i,j )A+ [ I NE ( R i+1,j−1 − b ⌣ i+1,j−1 )     +  I NW ( R i−1,j−1  −  b ⌣ i−1,j−1 )+ I SW ( R i−1,j+1  − b ⌣ i−1,j+1 )     +  I SE ( R i+1,j+1  −  b ⌣ i+1,j+1 ) ]×[ (1 − A) ( I NE  + I NW  +  I SW  +  I SE ) ],
where
(21) r ⌣ i,j = I NE R i+1,j−1 + I NW R i−1,j−1 + I SW R i−1,j+1 + I SE R i+1,j+1 I NE + I NW + I SW + I SE + I NE ( B i,j  − B i+2,j−2 )+ I NW ( B i,j  − B i−2,j−2 )  2( I NE + I NW + I SW + I SE ) +   I SW ( B i,j  − B i−2,j+2 ) +  I SE ( B i,j  − B i+2,j+2 ) 2( I NE + I NW + I SW + I SE ) , b ⌣ i,j = I NE B i+1,j−1 + I NW B i−1,j−1 + I SW B i−1,j+1 + I SE B i+1,j+1 I NE + I NW + I SW + I SE + I NE ( R i,j  − R i+2,j−2 )+ I NW ( R i,j  − R i−2,j−2 )  2( I NE + I NW + I SW + I SE ) +   I SW ( R i,j  − R i−2,j+2 ) +  I SE ( R i,j  − R i+2,j+2 ) 2( I NE + I NW + I SW + I SE ) .
Missing blue channels can be interpolated in the same way by replacing the role of red and blue notations. With the equations described above, missing red and blue channels at blue and red locations can be interpolated while edges are well preserved by striving to avoid direct averaging across edges using a direction-adaptive approach.
C. Interpolation of Remaining Missing Channels
After carrying out the process outlined in Section III-3-B, the temporal arrangement, which is shown in Fig. 5(d) , enables the interpolation of missing green channels at red and blue locations. We choose the step of green channel interpolation at blue and red locations for the third step because the temporal arrangement of the red samples does not lie in the two directions opposite (only H) from the green pixels in the center, even though the green samples lie in both opposite directions (V and H) after carrying out the steps of Section III-3-A, as shown in Fig. 5(c) . Therefore, it is impossible to estimate the green and red estimators,
g ⌣ i,j
and
r ⌣ i,j
, which are used as temporal estimators when performing the interpolation process. Interpolation of the green missing channels at the red and blue locations can also be performed using the same process outlined in Sections III-3-A and III-3-B by replacing the corresponding color notations. The process can be accomplished using the following:
(22) g ˜ i,j = R i,j +Δ C ˜ g,r,(i,j) = R i,j +( g ⌣ i,j − R i,j )×A+ [ I N ( G i,j−1 − r ⌣ i,j−1 )    + I S ​( G i,j+1  − r ⌣ i,j+1 )+  I E ​( G i+1,j  − r ⌣ i+1,j )+  I W ​( G i−1,j  − r ⌣ i−1,j ) ]      ×[ (1−A) ( I N + I S + I E + I W ) ],
where
(23) g ⌣ i,j = I N G i,j−1 + I S G i,j+1 + I E G i+1,j + I W G i−1,j ( I N + I S + I E + I W ) + I N ( R i,j  −  R i,j−2 )+  I S ( R i,j  −  R i,j+2 )+  I E ( R i,j  − R i+2,j )+  I W ( R i,j  −  R i−2,j ) 2( I N + I S + I E + I W ) , r ⌣ i,j = I N R i,j−1 + I S R i,j+1 + I E R i+1,j + I W R i−1,j ( I N + I S + I E + I W ) + I N ( G i,j  −  G i,j−2 )+  I S ( G i,j  − G i,j+2 )+  I E ( G i,j  − G i+2,j )+  I W ( G i,j  −  G i−2,j ) 2( I N + I S + I E + I W ) .
By using the same principles, color difference assumption, described above, equations for the remaining missing channel can be induced.
By the end of the remaining step, all of the missing channels are interpolated; therefore, a Bayer-like W-RGB mosaiced input image is now considered to be a full-color image having all red, green, and blue channels with good performance at the edge regions.
IV. Simulation Results
The common method for evaluating the performance of demosaicing methods for Bayer CFAs uses test images made by removing specific color channels at every pixel by force to form images with a Bayer pattern. This method is reasonable for demosaicing a Bayer-mosaiced image. However, for demosaicing W-RGB images, we cannot build input images with a W-RGB pattern from a full-color image, because we cannot extract the exact values of those white samples that do not satisfy the equation W i, j = R i, j + G i, j + B i, j . Therefore, we use an actual prototype digital camera with Bayer-like W-RGB CFA, as shown in Fig. 6 , to capture raw test images with a Bayer-like pattern. The conditions for capturing images are described in Table 1 .
PPT Slide
Lager Image
Prototype digital camera used for capturing W-RGB mosaiced input images.
Image capturing conditions.
Conditions Conditions
Sensor type CMOS image sensor
Number of pixels 5,111,680
Pixel size 1.4 µm × 1.4 µm
Exposure time 0.23 fps
White gain 16
Red gain 199
Green gain 16
Blue gain 35
Measuring the quality of demosaicing is not a simple task, because it is not easy to obtain a ground-truth full image using the RGB space for comparison; therefore, we downscale the resolution of the original W-RGB mosaiced images with a resolution of 2,608 × 1,960 to half size to acquire full-color ground-truth images with a resolution of 1,304 × 980. The input mosaiced images are shown in Fig. 7 . As mentioned in Section II, research addressing W-RGB is still in its early stages; therefore, all conventional methods remain poorly understood, because their proposed approaches to W-RGB are concept-based. As a result, we can only compare the proposed method to the color separation method [3] using (1), which shows complete equations for estimating missing red, green, and blue channels at white sample locations. In this case, however, only three channels at white sample locations can be interpolated; therefore, bilinear interpolation is adopted for reproducing a full-color image with the rest of the color channels.
PPT Slide
Lager Image
Set of raw W-RGB mosaiced images for simulations from real-life prototype digital camera with W-RGB CFA.
To compare the performance between the conventional and proposed methods, CPSNR, FSIM [10] , and the S-CIELAB color distance [11] were measured against ground-truth images. These metrics are widely used for evaluating the demosaicing method; FSIM and S-CIELAB can achieve high consistency for subjective evaluation by the human visual system. For simulation, we set the contribution parameter A to 7/10 to avoid edges requiring too much strengthening. The objective results of measuring CPSNRs and FSIMs and the average S-CIELAB color distances are shown in Tables 2 , 3 , and 4 , respectively.
Comparison of CPSNR according to different methods.
Method [3] Proposed method Improvement rate (%)
(a) 32.5041 38.8946 19.6605
(b) 35.3032 41.1113 16.4520
(c) 34.2445 39.7609 16.1082
(d) 29.5779 36.8052 24.4347
(e) 28.1584 38.2309 35.7708
(f) 36.9981 42.4348 14.6945
(g) 34.1023 42.5074 24.6467
(h) 35.5693 42.0681 18.2708
(i) 33.9160 40.1355 18.3379
(j) 32.6829 40.3919 23.5872
(k) 30.2155 39.2370 29.8571
(l) 28.7731 38.1270 32.5091
Avg. 32.6704 39.9753 22.3594
Comparison of FSIM according to different methods.
Method [3] Proposed method Improvement rate (%)
(a) 0.9799 0.9905 1.0817
(b) 0.9842 0.9921 0.8027
(c) 0.9834 0.9920 0.8745
(d) 0.9812 0.9920 1.1007
(e) 0.9878 0.9947 0.6985
(f) 0.9896 0.9954 0.5861
(g) 0.9896 0.9962 0.6669
(h) 0.9881 0.9949 0.6882
(i) 0.9867 0.9956 0.9020
(j) 0.9851 0.9938 0.8832
(k) 0.9879 0.9954 0.7592
(l) 0.9805 0.9923 1.2035
Avg. 0.9853 0.9937 0.8533
Comparison of S-CIELAB distance according to different methods.
Method [3] Proposed method Improvement rate (%)
(a) 0.4526 0.3378 25.3646
(b) 0.1222 0.0975 20.2128
(c) 0.2275 0.2113 7.12090
(d) 1.2052 0.8677 28.0037
(e) 1.0160 0.4677 53.9665
(f) 0.1391 0.1285 7.62040
(g) 0.3773 0.2151 42.9897
(h) 0.2343 0.1912 18.3952
(i) 0.4221 0.3614 14.3805
(j) 0.5383 0.3982 26.0264
(k) 1.0365 0.4966 52.0888
(l) 1.4434 0.7905 45.2335
Avg. 0.6012 0.3803 36.7454
Figure 8 shows the spatial distribution of the errors in white when the S-CIELAB color difference exceeds one or more, and green when it exceeds ten or more. Each subfigure in the figures shows the spatial distribution of the errors of the conventional method (left part) and proposed method (right part), respectively.
PPT Slide
Lager Image
Spatial distribution of errors by using S-CIELAB color distance metric.
Figure 9 shows the regions of the resulting images using the conventional and proposed demosaicing methods using the input images of (b), (h), (j), and (l) in Fig. 7 . These enlarged output images enable us to check aliasing artifacts.
PPT Slide
Lager Image
Result of demosaicing of input images (b), (h), (j), and (l) in Fig. 7 for subjective evaluation of aliasing artifacts. From top to bottom: whole image and specific image. From left to right: ground-truth images, resulting images using method [3], and resulting image using proposed method.
As shown in the objective results in Tables 2 , 3 , and 4 , the proposed method has the better overall averages of CPSNR, FSIM, and the S-CIELAB distance than those of method [3] , outperforming the conventional method by 7.305 dB, 0.008 dB and −0.220 dB, respectively. In addition, we can also determine that the performance of the proposed method is improved by an average of 22.359%, 0.853%, and 36.745% for CPSNR, FSIM, and S-CIELAB distance measuring factors, respectively. Furthermore, in the subjective evaluation, the proposed method can preserve edges better than the conventional method with fewer aliasing artifacts. As shown in all of the experimental results, the proposed method outperforms the conventional methods.
V. Conclusion
In this paper, a new demosaicing method for W-RGB CFA was presented. Our proposed demosaicing method can be effectively applied to real-life situations based on color differences estimation using the information of edge structure, correlation among raw samples, and directional adaptive weights. We analyzed the CPSNR, FSIM, and S-CIELAB distance performance of our proposed method and compared it with those of the conventional method. We found that with our proposed demosaicing method, the images captured by a real-life digital camera with Bayer-like W-RGB CFA can be demosaiced to full-color images with less blur and fewer artifacts than with conventional methods.
SK hynix provided the research facilities for this study. This research was supported by the MSIP (Ministry of Science, ICT & Future Planning), Rep. of Korea, under the ITRC (Information Technology Research Center) support program (IITP-2015-H8501-15-1005) supervised by the IITP (Institute for Information & Communications Technology Promotion).
BIO
herojongjoo@gmail.com
Jongjoo Park received his BS and MS degrees from the Department of Electronics and Computer Engineering, Hanyang University, Seoul, Rep. of Korea, in 2011 and 2013, respectively. Since 2013, he has been working toward his PhD degree in electronics and computer engineering. His main research interests include demosaicing for W-RGB CFA, denoising, and deblurring under low-light conditions.
esjang@hanyang.ac.kr
Euee Seon Jang received his BS degree in electronics engineering from Jeonbuk National University, Jeonju, Rep. of Korea, in 1991 and his PhD degree in electrical and computer engineering from The State University of New York at Buffalo, USA, in 1996. He is currently a professor with the Department of Computer Science & Engineering, Hanyang University, Seoul, Rep. of Korea. He has authored more than 150 MPEG contribution papers, more than 30 journal or conference papers, 35 pending or accepted patents, and two book chapters. He has received three ISO/IEC certificates of appreciation for contributions to MPEG-4 development. He received the presidential award from the Korean government for his contribution to MPEG standardization. His research interests include image/video coding, reconfigurable video coding, and computer graphics objects.
Corresponding Author jchong@hanyang.ac.kr
Jong-Wha Chong received his BS and MS degrees in electronics engineering from Hanyang University, Seoul, Rep. of Korea, in 1975 and 1979, respectively, and his PhD degree in electronics and communication engineering from Waseda University, Tokyo, Japan, in 1981. Since 1981, he has been with the Department of Electronic Engineering, Hanyang University, where he is now a professor. His main research interests are SoC design methodology; memory-centric design and physical design automation of 3D-ICs; indoor wireless communication SoC design for ranging and location; and video systems and power IT systems.
References
Bayer B.E. 1975 Color Imaging Array US Patent 3,971,065 issued July 20, 1976
Egawa Y. “A White-RGB CFA-Patterned CMOS Image Sensor with Wide Dynamic Range,” IEEE Int. Solid-State Circuits Conf. San Francisco, CA, USA Feb. 3–7, 2008 52 - 59
Honda H. “A Novel Bayer-like WRGB Color Filter Array for CMOS Image Sensors,” Proc. SPIE, Human Vis. Electron. Imag. XII San Jose, CA, USA Feb. 12, 2007 6492 1 - 10
Honda H. “High Sensitivity Color CMOS Image Sensor with WRGB Color Filter Array and Color Separation Process Using Edge Detection,” Int. Image Sensor Soc. Workshop Ogunquit, ME, USA June 6–10, 2007 263 - 266
Getman A. , Kim J. , Kim T.-C. “Imaging System Having White-RGB Color Filter Array,” IEEE Int. Conf. Image Process. Hong Kong, China Sept. 26–29, 2010 569 - 572
Shun K. “Color Reproductivity Improvement with Additional Virtual Color Filters for WRGB Image Sensor,” SPIE, Color Imag. XVIII: Displaying, Process., Hardcopy Appl. CA, USA Feb. 4, 2013 8652 1 - 7
Pekkucuksen I. , Altunbasak Y. 2012 “Edge Strength Filter Based Color Filter Array Interpolation,” IEEE Trans. Image Process. 21 (1) 393 - 397    DOI : 10.1109/TIP.2011.2155073
Wolberg J. 2010 “Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments,” Springer-Verlag Berlin, Germany 44 - 50
Adams J.E. , Hamilton J.F. 1995 Adaptive Color Plane Interpolation in Single Sensor Color Electronic Camera US Patent 5,629,734 issued May 13, 1997
Zhang L. 2011 “FSIM: A Feature Similarity Index for Image Quality Assessment,” IEEE Trans. Image Process. 20 (8) 2378 - 2386    DOI : 10.1109/TIP.2011.2109730
Johnson G.M. , Fairchild M.D. “Measuring Images: Differences, Quality, and Appearance,” Conf. SPIE/IS&T Electron. Imag. Santa Clara, CA, USA June 12, 2003 147 - 160