In generally, the resolution of reconstructed threedimensional images can be seriously degraded by undesired occlusions in the integral imaging system, because the undesired information of the occlusion overlap the threedimensional images to be reconstructed. To solve the problem of the undesired occlusion, we present an exemplarbased image restoration method in integral imaging system. In the proposed method, a minimum spanning treebased stereo matching method is used to remove the region of undesired occlusions in each elemental image. After that, the removed occlusion region of each elemental images are reestablished by using the exemplarbased image restoration method. For further improve the performance of the image restoration, the structure tensor is used to solve the filling error cause by discontinuous structures. Finally, the resolution enhanced threedimensional images are reconstructed by using the restored elemental images. The preliminary experiments are presented to demonstrate the feasibility of the proposed method.
I. INTRODUCTION
Integral imaging is a threedimensional imaging technology that can provide full parallax and continuous viewing points
[1]
. More notably, integral imaging provides naturally colored images without the use of any special glasses
[2]
. Since integral imaging was first proposed in 1908, it has drawn a great deal of attention from researchers
[3

7]
. To implement threedimensional imaging and display, integral imaging systems require multiple perspectives on threedimensional objects. These recorded perspective images are called elemental images, each of which plays an important role in integral imaging.
Realworld scenes have complex arrangements of objects with multiple occlusions. Occlusions can be present in all but the most constrained environments. In the digital image processing field, researchers often consider the occluded region to be a missing region and use imaging restoration methods to restore the missing region
[8

15]
. The most representative method in
[8]
is the exemplarbased image restoration algorithm; however, depth information is not considered in this method. In attempts to expand upon previous research, one pixel restoration method was proposed in a computational integral imaging system
[16]
to address this depth problem. In this method, some of the invisible pixels of the occluded target region in an elemental image can be restored by using the corresponding visible pixels from the others due to different perspectives on the elemental images. However, if the foreground object happens to be located very close to the occluded target, it can cause major information loss of the target object in the elemental images. It is difficult to reconstruct the occluded target region with such limited information. Therefore, this pixel restoration method can be affected by the distance between the occluded target object and the foreground object.
In this paper, we propose an image restoration method to overcome the limitation of the distance between the occluded target object and the foreground objects. In the proposed method, a minimum spanning tree (MST) is used to estimate the occluded target region of each elemental image, and these occluded target regions are marked based upon the estimated depth maps. Then the proposed pixel restoration method is used to fill in the region left behind after occlusion removal. Our method combines the exemplarbased image restoration algorithm with the pixel restoration scheme to enhance the visual quality of threedimensional integral imaging reconstruction for partially occluded objects. In Section II, the traditional image restoration methods are reviewed. The proposed method is presented in Section III. In Section IV, we report on carrying out several experiments and confirming the feasibility of our method. Finally, we conclude this paper in Section V.
II. REVIEW OF PIXEL RESTORATION SCHEMES
In previous works, several methods have been suggested to solve the occlusion problem
[16

18]
. Most approaches have attempted to develop specific image processing algorithms based on statistical or contour analysis to alleviate the occlusion problem. Besides the image processing algorithms, another method was proposed in which the occluded target region was directly removed for visibilityenhanced reconstruction
[17]
. However, in this approach, the vacant target area caused by occlusion removal may lead to visual quality degradation of the reconstructed target image. To overcome this problem, Piao et al.
[16]
proposed an effective approach for visibilityenhanced reconstruction of a partially occluded threedimensional scene by using the pixelrestoration method in a computational integral imaging system. This scheme can be expressed as follows:
where,
S_{m,n}
(
x,y
) is the pixel located at the position of (
x, y
) in elemental image
S_{m,n}
(
x, y
).
is the disparity between,
S_{m,n}
(
x, y
) and its corresponding pixel in
S_{i,j}
, which can be obtained from the disparity map in
[19]
. In synthetic aperture integral imaging, the elemental image array is assumed to be
M
×
N
elemental images.
As is described in Section I, the pixel restoration scheme depends on the distance between the occluded target region and the foreground objects. By analyzing whether the distance
d
between the occluded target regions and the foreground objects is less than Δ
d
_{min}
, those target regions cannot be completely reconstructed due to the effect of the foreground objects. Δ
d
_{min}
is defined as follows:
Where
d_{o}
is the distance between the lenslet array and the occluded target region, and
d_{c}
is the distance between the lenslet array and the foreground objects.
l_{c}
is the size of the occluding object,
p
is the pitch of the lenslet, and
n
is an index number defined in our previous work
[16]
.
III. THE PROPOSED METHOD
In synthetic aperture integral imaging, every elemental image represents a slightly different viewpoint on a threedimensional scene. Thus a number of invisible pixels of the occluded region of an elemental image viewed from one viewpoint may be visible in other elemental images due to their viewpoint differences. As is shown in the above description, using a pixel restoration scheme can restore a partially occluded region, but it fails for the region where the distance between the occluded target region and the foreground objects is less than Δ
d
_{min}
. We introduce a new image restoration method in synthetic aperture integral imaging in which all the missing pixels of the occluded region in each elemental image can be restored by finding the best patches from the others.
 A. MSTBased Stereo Matching
In previous work, a depth estimation method using MSTbased stereo matching in integral imaging was proposed to detect occlusions; it is suitable for both simple and complex threedimensional scenes
[19]
.
Zhong et al.
[19]
presents a nonlocal stereo matching method that can produce an accurate disparity map between two elemental images. According to the principle of measurement by triangulation, the disparity map can be transformed to a depth map, and we can use the depth map to detect the occlusions. In
[19]
, the reference elemental image
I_{n}
(
n
= 1,...,
N
) is represented as a connected, undirected graph
G
= (
V, E
), where each node in
V
corresponds to a pixel in
I_{n}
, and each edge in
E
connects a pair of neighboring pixels. In
[19]
, the first elemental image is selected as the reference image and a MST
T
is constructed from
G
. A nonlocal aggregated cost
is computed as follows:
where
C_{d}
is the matching cost,
Dis
(
p, q
) is the sum of the edge weights along the path (that is, the shortest path) between any two pixels
p
and
q
.
σ
is a userspecified parameter for distance adjustment, which is often set to be 0.1. The disparity at pixel
p
could be obtained by finding the minimum matching cost.
Fig. 1
(a) and (b) show the input elemental images used to extract the disparity map,
Fig. 1
(c) shows the disparity map that is generated by the nonlocal aggregation with nonlocal refinement.
Fig. 1
(d) shows the occlusion detection results of the first elemental images, which is replaced with a green color.
(a) The first elemental image. (b) The second elemental image. (c) Disparity map produced from MSTbased stereo matching. (d) Occlusion detection.
 B. Image Restoration Scheme
As shown in
Fig. 2
, five elemental images are selected as the source region. As shown in
Fig. 2
(a), we find the best patch from the source region to restore the missing region, which is enclosed by a solid red rectangle. The principle for selecting the elemental images is defined in Eq. (12).
(a) Five elemental images are selected as exemplar images to restore the elemental image, which is enclosed by a solid rectangle. (b) The best patches are selected from the five elemental images.
 1) Computing Patch Priorities
The filling order is crucial to nonparametric texture synthesis, and designing a fill order that explicitly encourages propagation of linear structure together with texture should produce a better image restoration
[8]
. Criminisi’s work performs this task through a bestfirst filling algorithm that depends entirely on the priority values that are assigned to each patch on the fill border. As shown in
Fig. 3
, we define the priorities for each point to determine the filling order.
Notation diagram. Ω is the removed region and its contour is δΩ, Φis the source region that was not removed. The patch ψ_{p} centered on the point p ∈ δΩ is the region to be filled.
For each point
p
∈
δ
Ω, its priority
P_{i,j}
(
p
) is defined as follows:
where the subscript
i, j
represents the
i^{th}
row and
j^{th}
column of the image in a synthetic aperture integral imaging system.
We call
Conf_{i,j}
(
p
) the confidence term and
D_{i,j}
(
p
) the data term, and they are defined as follows:
where, 
ψ_{i,j P}
 is the area of
ψ_{i,j P}
as shown in
Fig. 3
,
α
is a normalization factor,
n_{p}
is a unit vector orthogonal to the border
δ
Ω going through point
p
, and
is the isophote at point
p
. During initialization, the function
Conf_{i,j}
(
p
) is set to
Conf_{i,j}
(
p
) = 0 ∀
p
∈ Ω
_{i,j}
,
Conf_{i,j}
(
p
) = 1∀
p
∈ Φ
_{i,j}
.
 2) Propagating Texture and Structure
Once all priorities on the fill border have been computed, the patch
with highest priority is found. We use Eq. (9) to find the most similar patch
to fill the patch
where
ψ_{q}
∈ {Φ
_{i,j}
, Φ
_{0,}
_{j}
, Φ
_{i,}
_{0}
, Φ
_{M,j}
, Φ
_{i,N}
},
M
×
N
is the number of the elemental image in the synthetic aperture integral imaging method; in fact, as shown in
Fig. 2
(b), two steps are implemented to find the most similar patch:
In step 1, we find the five best patches {
ψ
_{i,j}
,
ψ
_{0,}
_{j}
,
ψ
_{i,}
_{0}
,
ψ
_{M,j}
,
ψ
_{i,N}
} from the five selected elemental images by using the dissimilarity measurement of the sum of squared intensity differences (SSD):
where
m, n
is the size of the patch
and
ψ_{q}
,
p_{x,y}
and
q_{x,y}
is the corresponding pixel’s value in
and
ψ_{q}
.
In step 2, we find the most similar patch from the five best patches by combining the
SSD
and a gradientbased measure (
GRAD
):
where
ψ_{q}
= {
ψ
_{i,j}
,
ψ
_{0,}
_{j}
,
ψ
_{i,}
_{0}
,
ψ
_{i,}
_{0}
,
ψ
_{i,N}
}, ∇
p_{x,y}
and ∇
q_{x,y}
is the corresponding pixel’s gradient value in
and
ψ_{q}
.
ω
is an equilibrium factor, in our experiment,
ω
= 0.05.
 3) Updating Confidence Values
After the patch
has been filled with new pixel values, the confidence
C_{i,j}
(
p
) is updated in the area delimited by
as follows:
Repeat the above steps, until the whole region is filled. Ultimately, we achieve the image in which the occlusion is removed, as shown in
Fig. 4
.
Image restoration in synthetic aperture integral imaging. (a) The mask marked in green. (b) The occlusion removed from the elemental image.
 4) Filling Order Control Using a Structure Tensor
The exemplarbased image restoration algorithm performs well in filling large missing regions. However, discontinuous structures can be problematic, and these discontinuous structures will influence patch priorities.
In view of this problem, we use the structure tensor to recompute the data term in Eq. (8)
[20]
.The structure tensor is also referred to as the secondmoment matrix. It summarizes the predominant directions of the gradient, and the degree along those directions. The structure tensors
J_{i,j}
are defined as follows:
where
K_{ρ}
is the normal kernel and
ρ
is the variance of
K_{ρ}
, ⊗ represents the cumulative operation, and * represents the Gaussian convolution operation.
Eq. (8) can be replaced as follows:
where
div
is the divergence operator.
We also modify the priority
P_{i,j}
(
p
) in Eq. (6) as follows:
where
α, β
is the adjustment coefficient and
α
+
β
= 1.
IV. EXPERIMENT AND DISCUSSION
In this section, image restoration and threedimensional image reconstruction in a synthetic aperture integral imaging system is implemented. The distance between the building and the pickup devices is approximately 20 m, and a toy human figure is located at approximately 1 m from the pickup devices. The camera in use has an image sensor array of 2400×1600 pixels, and each pixel size is 8.2 μm. In order to improve the operation speed, every elemental image is resized to 480×320 pixels. The moving step of the camera is Δ
d
= 2 mm.
As shown in
Fig. 5
, the occlusion regions are marked as a green circle by using MSTbased stereo matching, the image restoration algorithm used for the marked region. The result of Criminisi’s algorithm is shown in
Fig. 5
(a); we found that some errors occurred, which are marked by green circles. In our experiment, some errors exist in almost every restored element image, and these errors will reduce the visual quality of the reconstructed images.
Fig. 5
(b) shows the result proposed in
[16]
. It can be seen that only partially occluded regions are restored. This is because the distance between the occluded target region and the foreground objects is less than Δ
d
_{min}
.
Fig. 5
(c) shows the restored elemental image by using the proposed method without the structure tensor . Compared to
Fig. 5
(a), there are fewer errors occur in the fill regions, in which improving the visual quality of the reconstructed images significantly.
Fig. 5
(d) shows an improved algorithm for image restoration based on additional structure tensor in synthetic aperture integral imaging. Compared to
Fig. 5
(c), the error fill regions are further reduced. We have demonstrated that the proposed method remains robust when large regions need to be restored.
Restored elemental images using (a) algorithm in [8], (b) algorithm in [16]. (c) The proposed method. (d) The proposed method with a structure tensor.
Fig. 6
illustrates two sets of images reconstructed with the computation integral imaging reconstruction technique at the distance of 500 mm and 760 mm.
Fig. 6
(a) and (d) show the images reconstructed at
Z
= 500 mm and
Z
= 760 mm by using originally captured elemental images, and
Fig. 6
(b) and (e) show the images reconstructed using the occlusion removal method in
[16]
at
Z
= 500 mm and
Z
= 760 mm.
Fig. 6
(c) and (f) show the reconstructed 3D images by using the proposed method.
Reconstructed 3D images at (a)–(c) Z = 500 mm, (d)–(f) Z = 760 mm.
In the comparisons shown in
Fig. 6
(a)–(f), we were easily able to show that the visual quality of the reconstructed images is improved by using the proposed restoration method.
Fig. 6
(f) shows the reconstructed image, in which the restoration region is marked with a green circle. Even though blurring occurs in a small region of the reconstructed image due to missing depth information, the proposed method outperforms the one in
[16]
in terms of visual quality. We are continuing to develop extensions of the proposed method to address the issue of the missing depth information.
V. CONCLUSIONS
In this paper, we proposed an exemplarbased image restoration method to solve the occlusion problem in synthetic aperture integral imaging. Our proposed method successfully overcame the limitation of the distance between the target and the occluding object. Experimental results confirm the feasibility of the restoration method when applied to threedimensional image reconstructions. However, some depth information is missed by use of our method. In future works, we intend to extend the proposed method to address the issue of missing depth information.
Acknowledgements
This work was supported by the National Science Foundation of China (No. 61405022) and the Fundamental Research Funds for the Central Universities (No. DUT15RC (3)046).
BIO
Miao Zhang
received the Ph.D. degree in Electronics Engineering from Kwangwoon University, South Korea, in 2012. She is currently an Associate Professor with the School of Software Technology, Dalian University of Technology (DUT). Her current research interests include 3D imaging, display, and visualization.
Zhaolong Zhong
He received the B.S. degree in Communication and Electronic Information Engineering from Qingdao Technological University, China, in 2014. Currently, he is working toward the M.S. degree in Information and Communication Engineering, Dalian University of Technology. His research interests include 3D pattern recognition, and 2D/3D image processing.
Yongri Piao
He received the B.S. degree in automation engineering from Jilin University, China, in 2003, and the M.S. and Ph.D. degrees in information and communication engineering from Pukyong National University, Republic of Korea, in 2005 and 2008, respectively. From September 2008 to December 2011, he was a Research Professor at the 3D Display Research Center of Kwangwoon University. Since March 2012, he has been an Associate Professor at the School of Information and Communication Engineering, Dalian University of Technology, Dalian, China. His research interests include optical imaging and 3D display, optical and digital encryption, 3D pattern recognition and tracking, and 2D/3D image processing.
Lippmann G.
1908
“La photographic integrale,”
Comptes Rendus de l'Académie des Sciences
146
446 
451
Lee J. J.
,
Lee B. G.
,
Yoo H.
2011
“Depth extraction of threedimensional objects using block matching for slice images in synthetic aperture integral imaging,”
Applied Optics
50
(29)
5624 
5629
DOI : 10.1364/AO.50.005624
Jang J. S.
,
Javidi B.
2002
“Threedimensional synthetic aperture integral imaging,”
Optics Letters
27
(13)
1144 
1146
DOI : 10.1364/OL.27.001144
Stern A.
,
Javidi B.
2003
“3D computational synthetic aperture integral imaging (COMPSAII),”
Optics Express
11
(19)
2446 
2451
DOI : 10.1364/OE.11.002446
Hong S. H.
,
Jang J. S.
,
Javidi B.
2004
“Threedimensional volumetric object reconstruction using computational integral imaging,”
Optics Express
12
(3)
483 
491
DOI : 10.1364/OPEX.12.000483
Jang J. Y.
,
Ser J. I.
,
Cha S.
,
Shin S. H.
2012
“Depth extraction by using the correlation of the periodic function with an elemental image in integral imaging,”
Applied Optics
51
(16)
3279 
3286
DOI : 10.1364/AO.51.003279
Arimoto H.
,
Javidi B.
2001
“Integral threedimensional imaging with digital reconstruction,”
Optics Letters
26
(3)
157 
159
DOI : 10.1364/OL.26.000157
Criminisi A.
,
Perez P.
,
Toyama K.
2004
“Region filling and object removal by exemplarbased image restoration,”
IEEE Transactions on Image Processing
13
(9)
1200 
1212
DOI : 10.1109/TIP.2004.833105
Komodakis N.
“Image completion using global optimization,”
in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition
New York, NY
2006
442 
452
Sun J.
,
Yuan L.
,
Jia J.
,
Shum H. Y.
2005
“Image completion with structure propagation,”
ACM Transactions on Graphics
24
(3)
861 
868
DOI : 10.1145/1073204.1073274
Huang T.
,
Chen S.
,
Liu J.
,
Tang X.
“Image inpainting by global structure and texture propagation,”
in Proceedings of the 15th International Conference on Multimedia
Augsburg, Germany
2007
517 
520
Zhou H.
,
Zheng J.
“Adaptive patch size determination for patch based image completion,”
in Proceedings of 17th IEEE International Conference on Image Processing
Hong Kong
2010
421 
424
Liu Y.
,
Tian X. J.
,
Wang Q.
,
Shao S. X.
,
Sun X. L.
“Image inpainting algorithm based on regional segmentation and adaptive window exemplar,”
in Proceedings of 2010 2nd International Conference on Advanced Computer Control (ICACC)
Shenyang, China
2010
656 
659
Wu J.
,
Ruan Q.
“Object removal by cross isophotes exemplarbased inpainting,”
in Proceedings of 18th International Conference on Pattern Recognition (ICPR)
Hong Kong
2006
810 
813
Telea A.
2004
“An image inpainting technique based on the fast marching method,”
Journal of Graphics Tool
9
(1)
25 
36
Piao Y.
,
Zhang M.
,
Kim E. S.
2012
“Effective reconstruction of a partially occluded 3D target by using a pixel restoration scheme in computational integralimaging,”
Optics and Lasers in Engineering
50
(11)
1602 
1610
DOI : 10.1016/j.optlaseng.2012.05.013
Hong S. H.
,
Javidi B.
2005
“Threedimensional visualization of partially occluded objects using integral imaging,”
Journal of Display Technology
1
(2)
354 
359
DOI : 10.1109/JDT.2005.858879
Piao Y.
,
Kim E. S.
2011
“Performanceenhanced recognition of a far and partially occluded 3D object by use of direct pixelmapping in computational curvingeffective integral imaging,”
Optics Communications
284
(3)
747 
755
DOI : 10.1016/j.optcom.2010.10.002
Zhong Z.
,
Piao Y.
,
Qu H.
,
Zhang M.
“MSTbased occlusion detection in synthetic aperture integral imaging,”
in Proceedings of 2015 OSA Imaging and Applied Optics Congress: Imaging and Applied Optics
Arlington, VA
2015
Liu K.
,
Su B. T.
,
Wang Y. B.
2012
“Improved algorithm of exemplarbased image inpainting,”
Computer Engineering
38
(7)
193 
195