In this paper, we propose a threedimensional (3D) image correlator by use of computational integral imaging reconstruction based on the modified convolution property of periodic functions (CPPF) for recognition of partially occluded objects. In the proposed correlator, elemental images of the reference and target objects are picked up by a lenslet array, and subsequently are transformed to a subimage array which contains different perspectives according to the viewing direction. The modified version of the CPPF is applied to the subimages. This enables us to produce the plane subimage arrays without the magnification and superimposition processes used in the conventional methods. With the modified CPPF and the subimage arrays, we reconstruct the reference and target plane subimage arrays according to the reconstruction plane. 3D object recognition is performed through crosscorrelations between the reference and the target plane subimage arrays. To show the feasibility of the proposed method, some preliminary experiments on the target objects are carried out and the results are presented. Experimental results reveal that the use of plane subimage arrays enables us to improve the correlation performance, compared to the conventional method using the computational integral imaging reconstruction algorithm.
I. INTRODUCTION
Occluded objects have been an interesting research area for the field of threedimensional (3D) object visualization and recognition. Recently, several approaches employed a 3D imaging technique to solve the occlusion problem [19]. Among them, integral imaging can be considered as one of the promising solutions, and thus some techniquesbased on integral imaging have been proposed for 3D recognition of partially occluded objects. Integral imaging is a method of recording and displaying techniques for 3D scenes with the lenslet array. The recorded elemental images consist of several twodimensional (2D) images with different perspectives
[10

17]
. In 2007, a 3D image correlator using the computational integral imaging reconstruction (CIIR) method was proposed
[6]
. In CIIR, the plane images were generated by the magnification and superimposition process of elemental images. Each plane image contains both focused and defocused images for occlusion and target objects. This enables us to recognize target objects with occlusion. However, the CIIR process may produce the addition of blur noises in plane images due to the interpolated images and high computation loads. To overcome this problem, the modified CIIRbased 3D correlator using smart pixel mapping, which can reduce the magnification factor and thus improve the correlation performance, was proposed in 2008
[7]
. However, this method also utilized the magnification process to generate the depthconverted plane images.
Recently, a depth slicing method using a convolution property between periodic functions (CPPF) has been proposed to produce the plane image array, which is similar to plane images from the previous CIIR method
[18
,
19]
. In integral imaging with the lenslet array, an elemental image has periodic property depending on object depth. This method takes advantage of the periodic property of spatial frequency of an elemental image. The CPPF is performed by convolution between the elemental image and the 2D
δ
function array whose spatial period corresponds with target depth. As a result, the targeting depth image can be reconstructed as the structure of the plane image arrays with fast calculation time compared with the previous CIIR method. In addition, CPPF technique can be applied to optical 3D refocusing display with the lenslet array
[20]
. Here, 3D objects with their own perspectives can be reconstructed to be refocused on their depth in the space for viewers.
In this paper, we propose a 3D image correlator using the computational reconstruction based on the modified CPPF for partially occluded object recognition. We introduce a modified CPPF (MCPPF) method for subimages to produce the plane subimage arrays (PSAs) without magnification and superimposition processes used in the conventional methods. In the proposed correlator, elemental images of the reference and target objects are captured by the image sensor through the lenslet array. And, the recorded elemental images are transformed to the subimages, which contain different perspectives according to the viewing direction. The proposed MCPPF method can be applied to these subimage arrays for reconstruction of the 3DPSAs. Only the target PSA is reconstructed on the right plane where the target object was originally located and contains clearly focused perspectives. On the other hand, other target PSAs are reconstructed on the various reconstruction planes where the focused and blurred images are mixed. 3D object recognition is performed through crosscorrelations between the reference and the target PSAs. To show the feasibility of the proposed method, some preliminary experiments on the target objects are carried out and the results are presented.
II. PROPOSED MCPPFBASED 3D IMAGE CORRELATOR
Figure 1
shows the schematic diagram of the proposed method. The proposed method is divided into four processes; (1) pickup, (2) subimage transform, (3) computational reconstruction using MCPPF and (4) recognition processes.
Schematic diagram of the proposed method: (a) pickup, (b) subimage transform, (c) computational reconstruction using MCPPF, and (d) recognition process.
 2.1. Pickup Process
The first process of the proposed method is pickup of 3D objects. This is the same as the conventional pickup process in integral imaging. The lenslet array is used to capture 3D objects. We assume that a reference object
R
(
x_{r}
,
y_{r}
,
z_{r}
) is located at the known distance of
z_{r}
in front of a lenslet array in the pickup system as shown in
Fig. 1(a).
And a target object
O
(
x_{o}
,
y_{o}
,
z_{o}
), to be recognized is partially occluded by the occlusion object and located at an arbitrary distance of
z_{o}
in front of the lenslet array. Then elemental images of the reference and target objects are recorded by the lenslet array and the 2D image sensor. The recorded elemental images of the reference and target objects are stored for the use of the next process.
The conventional pickup process of an integral imaging system by using the direct pickup method is based on ray optics. The geometrical relationship between a point object and its corresponding point images on the elemental image plane is shown in
Fig. 2(a).
In the conventional integral imaging system with planar lenslet array, the geometrical relation of 2D form is given by
Schematic diagram of the computational reconstruction using CPPF.
In the
Fig. 2(a)
and Eq. (1), the origin of the coordinates is the edge of the elemental lens located at the bottom of the lenslet array. The object point (
x_{o}
,
y_{o}
,
z_{o}
) is the position along the
x
,
y
, and
z
axes, respectively.
P
represents the distance between the optical centers of the neighboring elemental lenses and is assumed to be the same as the lateral length of an elemental image.
f
is the focal length of an elemental lens in a planar lenslet array.
x_{Ek}
and
y_{El}
are the image point by the
k
th and
l
th elemental lenses, respectively. The imaging distance of a point object measured from lenslet array is
z_{E}
=
z_{O }
f
/ (
z_{O}
+
f
).
X_{Zo}
is depthdependent spatial period. However, due to the limit of resolution of a pickup device, Eq. (1) should be converted into pixel units. Thus the pixelated version of Eq. (1) is denoted by
where
N
,
P
, and
k
_{max}
are lateral resolution of the elemental image array in onedimensional (1D) condition, the diameter of an elemental image and the lateral number of elemental lenses, respectively.
From the geometrical relationship, the 1D form of spatial period depending on object depth is given by 
x_{E}
_{i}
–
x_{E}
_{(i1)}
, 2 ≤ i≤
k
_{max}
. Then the depth dependent spatial period is to be 
x_{E}
_{i}
–
x_{E}
_{(i1)}
= 
z_{O}P
/ (
z_{O}
+
f
). Hence, the pixelated version is denoted by
Equation (3) implies that the intensity of the elemental image corresponding with an object locating specific depth is imaged with the same interval on the imaging plane as described in
Fig. 2(b).
Thus, a lenslet array in an integral imaging system may be regarded as the convertor which converts depth information of 3D space changes into 2D periodic information and vice versa.
 2.2. Subimage Transform
The recorded elemental images do not have the characteristic of the shift invariance. That is, the projected small images in elemental images are scalechanged if the 3D object is located at different locations. For this reason, we apply the subimage transform to the recorded elemental images. The subimage array can provide both shift invariance and the whole projected image in elemental images
[5]
. The subimage transform can be performed based on either single pixel extraction or multiplepixel extraction. In this paper, our intention is to apply the concept of conventional CPPF to subimages for the proposed 3D image correlator. Therefore, the conventional CPPF technique can be modified for subimages.
We explain the principle of the proposed MCPPF method for subimage array. The lateral resolution of the elemental image array
N
is given by
N
=
n
×
k
_{max}
when an elemental image is composed of
n
pixels. The
x_{CEk}
may be represented as ordered pair (
k
,
m
), where k is index of an elemental image and
m
is given by
m
=
x_{CEk}
– (
k
– 1)
n
.
m
is order of pixel in
k
th elemental image and 1≤
m
≤
n
. The lateral resolution of subimage array is the same as that of the elemental image array,
N
. If the number of subimage array is
j
_{max}
and each subimage is composed of
ξ
pixels, then the
N
can be described by
N
=
ξ
×
j
_{max}
. A pixel in the subimage array may be also represented as ordered pair (
j
,
η
), where
j
is a subimage number and
η
is order of the pixel in
j
th subimage and 1≤
η
≤
ξ
. In this variable correspondence, we can find the relationship as follows
where
c
is the number of pixels to be extracted for subimage transform.
To show the periodic property of subimage array adequately, let us consider the condition
c
=1 as shown in
Fig. 3.
The pixel correspondence between elemental image array and subimage array may be represented as follows
Schematic diagram of subimage transform.
A pixel position on the subimage array can be represented as the ordered pair (
j
,
η
) by
Hence, the 1D form of spatial period in subimage domain is given by 
s
_{i}
–
s
_{(i1)}
, 2 ≤ i≤
j
_{max}
. The detailed version of spatial period is given by
For example, in
Fig. 3
, suppose
n
and
k
_{max}
are 10 pixels and 9, respectively. Then,
ξ
and
j
_{max}
are 9 pixels and 10, respectively. In this condition, if the depth dependent spatial period of the elemental image,
X_{Zo}
, is 11 pixels, then that of the subimage array,
X_{sub}
, is 10 pixels.
In this paper, the multiplepixel extraction method is used to increase the resolution of subimages
[4]
. Since the multiplepixel extraction is based on the block combination from each elemental image, the spatial period of each pixel within the block is the same with that of the single pixel extraction method. Therefore, we can apply Eq. (7) to subimages generated by multiplepixel extraction.
 2.3. Computational Reconstruction using MCPPF
In the third process of the proposed method, we apply the MCPPF technique to the recorded subimages in order to generate 3D PSAs for object recognition. In the transformed subimage array, the subimage has a spatially periodic property, and the spatial period is dependent on the depth of an object.
Mostly, depth extraction or recognition process in various methods is followed in the reverse process of obtaining multiple 2D images from 3D scenes. On the other hand, the MCPPF is to use the superimposing property of convolution between a set of periodic functions and periodic
δ
function array to discriminate target periodic spatial information from the input function. The 3D image reconstruction method using MCPPF is as follows.
The intensity of 2D image may be written as
g
(
x_{E}
) =
f
(
x_{E}
) ⊕
h
(
x_{E}
), where ⊕ denotes the convolution,
x_{E}
represents the x coordinate on the imaging plane,
f
(
x_{E}
) represents the scaled object intensity, and
h
(
x_{E}
) represents the intensity impulse response, respectively. Moreover, depthdependent image intensity of a 3D object can be represented as
g
(
x_{E}
)
_{Zo}
=
f
(
x_{E}
)
_{Zo}
⊕
h
(
x_{E}
)
_{Zo}
. Assuming the geometrical optics condition (
λ
→0), the intensity impulse response
h
(
x_{E}
)
_{Zo}
can be represented by the
δ
function. Thus, the intensity of the subimage array at the corresponding object depth
z_{O}
may be written as
Although the object intensity is continuously distributed along the
z
axis, the intensity impulse response of the subimage array is represented as a discrete spatial period because of the limited resolution of a pickup device sensor. Hence, the subimage image array may be represented as
G
(
x_{E}
)=Σ
g
(
x_{E}
)
_{Zo}
. To discriminate the depth information in a subimage image array, the MCPPF method can perform,
G
(
x_{E}
) ⊕
h
(
x_{E}
)
_{Zo}
, more specifically the convolution property between periodic
δ
function arrays.
The 3D image reconstruction process using MCPPF is illustrated in
Fig. 4.
The pickup condition of the elemental image is as follows. The focal length and the diameter of an elemental lens are 15 mm and 5 mm, respectively. The size of the lenslet array is 30×30. The resolution of the elemental image is 900×900 pixels. The 3D object is located along the
z
axes. The right side of the arrow mark in
Fig. 4
represents the convolution process between the subimage array and a
δ
function array. The pixel extraction factor in Eq. (4) is applied by
c
=5. The resolution of a subimage is 150×150 pixels. The
X_{sub}
is spatial period of
δ
function array, and the spatial period varies depending on target depth. The images arranged in the middle of
Fig. 4
are output results of the MCPPF according to spatial period with 1 pixel interval. To observe the results clearly, the enlarged images are prepared at the bottom of
Fig. 4.
3D image reconstruction process using MCPPF applied in a subimage array.
 2.4. Recognition Process
From the process of computational reconstruction based on MCPPF, we can reconstruct the reference and target PSAs. First, the reference PSA is reconstructed on the known
z_{r}
plane for the reference object. This is called the reference template. Next, for the partially occluded target object, a set of target PSAs are reconstructed on each reconstruction plane by changing the distance
z
from the lenslet array as shown in
Fig. 1(c)
. Among them, the target PSA reconstructed at the
z_{o}
plane of the original position of the target object during the pickup process has highly focused images of the target object. On the other hand, the other target PSAs reconstructed on the
z
planes far away from the
z_{o}
plane become out of focus.
Once a target PSA
P_{O}
(
x
,
y
,
z
) of the target object reconstructed at the arbitrary distance of
z
using the MCPPF technique is generated, the correlation process can be performed between the target PSA of
P_{O}
(
x
,
y
,
z
) and the reference template of
P_{R}
(
x
,
y
,
z_{r}
). Accordingly, we calculate the correlation result using the following correlation operation as given by
where * denotes the complex conjugate. It is seen that the correlation peak
C
(
z
) is a function of distance z and the behavior of
C
(
z
) such as sharpness can be a performance measure for 3D object recognition. This correlation process is repeated for all target PSAs reconstructed on
z
planes by varying the distance
z
within the desired depth range. In fact, since the target PSA with the most highly focused target image is reconstructed at the
z_{o}
plane, a sharp correlation peak may be expected to be occurred on the
z_{o}
plane through the correlations between the known reference template and target PSAs of the target object.
III. EXPERIMENTAL RESULTS AND DISCUSSIONS
To demonstrate the feasibility of our proposed 3D image correlator, the preliminary experiment on the 3D objects in space was performed. In the experiment, a 3D object (car) is employed as reference test object as illustrated in
Fig. 5(a)
. And, the occlusion (a bundle of plants) is shown in
Fig. 5(b)
.
Figure 5(c)
shows the camera view image as the object and the occlusion are arranged inthe object field.
Experimental conditions (a) 3D object to be recognized (b) Occlusion object (c) Observed scene for occlusion and 3D object.
The experimental structure is shown in
Fig. 6.
The 3D object and occlusion are located at 200 mm and 100 mm from the lenslet array, respectively. Here the lenslet array is composed of 30×30 lenslets, and the lenslet diameter and the focal length are 5 mm and 15 mm, respectively.
Experimental setup.
Using the experimental setup of
Fig. 6
, we first recorded the elemental images of reference objects which were located at a known position
z
=200 mm in front of a lenslet array. And, we captured the elemental image array of the unknown target object, which was located at an arbitrary position behind occlusion.
Figure 7(a)
and
(b)
show the captured elemental images for reference object and partially occluded object, respectively. And the corresponding subimages are shown in
Fig. 7(c)
and
(d)
after subimage transform. The resolutions of the elemental images and subimages are 900×900 pixels.
(a) Elemental images for 3D object (b) Elemental images for partially occluded 3D object (c) Subimages for 3D object (d) Subimages for partially occluded 3D object.
Figure 8(a)
shows the reconstructed PSA of reference object using the computational reconstruction based on MCPPF for subimages of
Fig. 7(c)
, which the reconstruction depth was 200 mm. Using the subimages of the partially occluded object shown in
Fig. 7(d)
, we generated their target PSAs according to the distances from 170 mm to 230 mm with the step of 5 mm. Some examples are shown in
Fig. 8(b)
.
Reconstructed images using computational reconstruction based on MCPPF (a) Reference PSA, (b) Target PSAs at different depths.
With the reference PSA and target PSAs, the correlation performances of the proposed method were measured using the correlation operation of Eq. (9). Correlation results were calculated by conducting the correlation between the reference PSA and a series of the target PSAs. The correlation results are shown in
Figs. 9
and
10.
Figure 9
shows the autocorrelation results of the reference object shown in
Fig. 5(a).
The correlation array was obtained due to the image array structure of PSAs. Among them, the maximum correlation peak was measured for object recognition. The curve of correlation peaks from the proposed method along the distance
z
was shown in
Fig. 9(b).
This result indicates that the ‘car’ object is located at
z
=200 mm because the maximum of correlation peaks is obtained at this distance.
(a) Autocorrelation output result of the reference object (b) Graph of maximum correlation peaks according to the different reconstruction distances.
(a) Correlation output result of unknown object (b) Graph of maximum correlation peaks of unknown true and false object according to the different reconstruction distances.
And the correlation result for the partially occluded objects as shown in
Fig. 5(c)
was shown in
Fig. 10(a).
The curve of maximum correlation peaks for two different cars (True car and False car) were calculated and presented in
Fig. 10(b).
In addition, we compared the proposed method with the conventional method based on CIIR
[7]
. The comparison experiments were performed under the same condition. The recorded elemental images were used to generate the CIIR images. The CIIR images for the reference object and the unknown target object were shown in
Fig. 11
. We compared correlation peaks of partially occluded object between proposed and conventional methods as shown in
Fig. 12.
Each correlation peak was normalized using the value of the maximum correlation peak obtained in the proposed method. For the conventional method, the maximum value of correlation peak was approximately 0.94. It is seen that the proposed method provides not only higher correlation peak but also sharper characteristic of correlation peaks than the conventional method. This is because our method can remove the blurring effect due to the magnification and superimposing processes of the CIIR process.
Reconstructed plane images using the conventional CIIR method.
Comparison of correlation peaks of partially occluded object between the proposed and conventional methods.
IV. CONCLUSIONS
In conclusion, we proposed a new 3D image correlator using the computational reconstruction based on MCPPF for partially occluded object recognition. In the proposed correlator, PSAs generated from MCPPF were used for 3D object recognition and thus we can recognize the partially occluded 3D object using the maximum peaks correlated with the reference object. Some preliminary experiments show the usefulness of the proposed method by comparison with the conventional method using CIIR.
Color versions of one or more of the figures in this paper are available online.
Acknowledgements
This work was supported in part by the IT R&D program of MKE/KEIT. [10041682, Development of highdefinition 3D image processing technologies using advanced integral imaging with improved depth range] and this work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Science, ICT and Future Planning) (No. 2013067321).
Stern A.
,
Javidi B.
2006
Threedimensional image sensing visualization and processing using integral imaging
Proc. IEEE
http://dx.doi.org/10.1109/JPROC.2006.870696
94
591 
607
Javidi B.
,
PonceDiaz R.
,
Hong S.H.
2006
Threedimensional recognition of occluded objects by using computational integral imaging
Opt. Lett.
http://dx.doi.org/10.1364/OL.31.001106
31
1106 
1108
DaneshPanahp M.
,
Javidip B.
,
Watson E. A.
2008
Three dimensional imaging with randomly distributed sensors
Opt. Express
http://dx.doi.org/10.1364/OE.16.006368
16
6368 
6377
Shin D.H.
,
Lee B.G.
,
Lee J.J.
2008
Occlusion removal method of partially occluded 3D object using subimage block matching in computational integral imaging
Opt. Express
http://dx.doi.org/10.1364/OE.16.016294
16
16294 
16304
Park J.H.
,
Kim J.
,
Lee B.
2005
Threedimensional optical correlator using a subimage array
Opt. Express
http://dx.doi.org/10.1364/OPEX.13.005116
13
5116 
5126
Park J.S.
,
Hwang D.C.
,
Shin D.H.
,
Kim E.S.
2007
Resolutionenhanced 3D image correlator using computationally reconstructed integral images
Opt. Commun.
http://dx.doi.org/10.1016/j.optcom.2007.04.007
276
72 
79
Zhang M.
,
Piao Y.
,
Kim E.S.
2010
Occlusionremoved scheme using depthreversed method in computational integral imaging
Appl. Opt.
http://dx.doi.org/10.1364/AO.49.002571
49
2571 
2580
Li G.
,
Kwon K.C.
,
Shin G.H.
,
Jeong J.S.
,
Yoo K.H.
,
Kim N.
2012
Simplified integral imaging pickup method for real objects using a depth camera
J. Opt. Soc. Korea
http://dx.doi.org/10.3807/JOSK.2012.16.4.381
16
381 
385
Shin D.
,
Javidi B.
2012
Threedimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing
Opt. Lett.
http://dx.doi.org/10.1364/OL.37.001394
37
1394 
1396
Lippmann G.
1908
La photographic intergrale
C. R. Acad. Sci.
146
446 
451
Okano F.
,
Hoshino H.
,
Arai J.
,
Yuyama I.
1997
Realtime pickup method for a threedimensional image based on integral photography
Appl. Opt.
http://dx.doi.org/10.1364/AO.36.001598
36
1598 
1603
Lee B.
,
Jung S.
,
Park J.H.
2002
Viewingangleenhanced integral imaging by lens switching
Opt. Lett.
http://dx.doi.org/10.1364/OL.27.000818
27
818 
820
Shin D.H.
,
Lee S.H.
,
Kim E.S.
2007
Optical display of true 3D objects in depthpriority integral imaging using an active sensor
Opt. Commun.
http://dx.doi.org/10.1016/j.optcom.2007.03.072
275
330 
334
MartinezCuenca R.
,
Saavedra G.
,
MartinezCorral M.
,
Javidi B.
2009
Progress in 3D multiperspective display by integral imaging
Proc. IEEE
http://dx.doi.org/10.1109/JPROC.2009.2016816
97
1067 
1077
Cho M.
,
Shin D.
2013
3D integral imaging display using axially recorded multiple images
J. Opt. Soc. Korea
http://dx.doi.org/10.3807/JOSK.2013.17.5.410
17
410 
414
Park J.H.
,
Hong K.
,
Lee B.
2009
Recent progress in threedimensional information processing based on integral imaging
Appl. Opt.
http://dx.doi.org/10.1364/AO.48.000H77
48
H77 
H94
Li G.
,
Kim S.C.
,
Kim E.S.
2011
Viewing qualityenhanced reconstruction of 3D object images by using a modified computational integral imaging reconstruction technique
3D Research
3
1 
9
Jang J.Y.
,
Ser J.I.
,
Cha S.
,
Shin S.H.
2012
Depth extraction by using the correlation of the periodic function with an elemental image in integral imaging
Appl. Opt.
http://dx.doi.org/10.1364/AO.51.003279
51
3279 
3286
Jang J.Y.
,
Shin D.
,
Kim E.S.
2014
Improved 3D image reconstruction using convolution property between periodic functions in curved integral imaging
Opt. Lasers Eng.
http://dx.doi.org/10.1016/j.optlaseng.2013.09.011
54
14 
20
Jang J.Y.
,
Shin D.
,
Kim E.S.
2014
Optical threedimensional refocusing from elemental images based on a sifting property of the periodic δfunction array in integralimaging
Opt. Express
http://dx.doi.org/10.1364/OE.22.001533
22
1533 
1550