Photoncounting sensing is a widely used technique for lowlightlevel imaging applications. This paper proposes a distance information extraction method with photoncounting passive sensing under lowlightlevel conditions. The photocounting passive sensing combined with integral imaging generates a photonlimited elemental image array. Maximumlikelihood estimation (MLE) is used to reconstruct the photonlimited image at certain depth levels. The distance information is extracted at the depth level that minimizes the sum of the standard deviation of the corresponding photoevents in the elemental image array. Experimental and simulation results confirm that the proposed method can extract the distance information of the object under lowlightlevel conditions.
I. INTRODUCTION
Distance information extraction has been the subject of research for numerous applications
[1

3]
. For depth information to be extracted, threedimensional (3D) information needs to be acquired and processed. For example, stereo images or a sequence of images are often matched to each other to extract depth information based on pixel disparity
[2
,
3]
.
Integral imaging (II) is primarily a 3D display technique but it has been widely adopted for information processing such as obtaining depth information and object recognition
[4

11]
. During II recording, an elemental image array is generated, that has different views of the object. One advantage of II is that only a single exposure is required to obtain 3D information; no calibration is needed, unlike stereo imaging, and no active illumination is needed, unlike holography or light detection and ranging (LIDAR)
[12
,
13]
. A depth extraction technique using elemental images has been studied in
[8

11]
. Depth is extracted by means of onedimensional elemental image modification and a correlationbased multibaseline stereo algorithm
[10]
. In
[11]
, the depth level of the reconstruction plan is estimated by minimizing the sum of the standard deviations of the corresponding pixels’ intensity.
Photoncounting imaging has been developed for lowlightlevel imaging applications such as night vision, and laser radar, radiological, and stellar imaging
[14

18]
. Advanced photoncounting imaging technology can register a single photoevent at each pixel. In that case, photodetection is carried out in the binary mode generating a binary dotted image. The object recognition with nonlinear matched filtering is proposed in
[19]
. II reconstruction with maximum likelihood estimation (MLE) is proposed in
[20]
. Stereoscopic photoncounting sensing has been proposed for distance information extraction
[21]
.
This paper proposes the use of photoncounting passive sensing combined with integral imaging for distance information extraction under lowlightlevel conditions. Photonlimited imagery is reconstructed with maximum likelihood estimation (MLE). In this paper, the MLE for photonlimited scene reconstruction in 3D space is proposed with the Poisson distribution in
[20]
, while the probability model is modified according to the lowlightlevel conditions. It has been shown that the MLE is merely the average of the photoevents in the elemental image array. Those photoevents are associated with pixels corresponding to a specific point in 3D space. The obtained depth level is the distance that minimizes the sum of the standard deviations of the corresponding photocounts. The sum of the standard deviations represents the uncertainty of the sampled information. There have been efforts to minimize the uncertainty in order to reconstruct the occluded scene in
[1]
and the intensity elemental image array in
[11]
.
Photonlimited elemental images are simulated on a computer while varying the expected total number of photoevents. The performance is evaluated accordingly. We also compare the distance extraction between photonlimited and intensity elemental images. The uncertainty minimization has been applied to both photoevent and intensity cases and consistent results are obtained from both cases. The experimental results confirm that the proposed method can extract distance information under lowlightlevel conditions. To the best of the authors’ knowledge, it is the first report on distance extraction by use of photoncounting passive sensing combined with integral imaging.
The rest of the paper is organized as follows. Section 2 describes the photoevent model and the distance information extraction algorithm. The experimental and simulation results are presented in Section 3. Conclusion follows in Section 4.
II. DISTANCE INFORMATION EXTRACTION WITH PHOTONCOUNTING INTEGRAL IMAGING
The II recording system generates an elemental image array as illustrated in
Fig. 1
. The microlens array is composed of a large number of small convex lenslets, and the ray information captured by each lenslet appears as an elemental image, that has different view of the object.
Under lowlightlevel conditions, a photodetector can register a single photoevent and generate a binary dotted elemental image array. It can be assumed that the probability of a photoevent is proportional to the intensity of the pixel at a lowlight level
[15]
. Thus, the following probability model is valid:
Integral imaging system with, (a) CCD camera, (b) photodetector.
where
y_{i}
indicates a single photoevent at pixel
i
,
N_{p}
is an expected total photocounts in the scene,
x_{i}
is the normalized intensity at pixel
i
, i.e.,
where
N_{t}
is the total number of pixels in the scene. It can be seen that
E
(
y_{i}
) =
n_{i}
and
Let
y_{i}
, where
i
= 1, 2,… ,
K
, is the number of photocounts detected at pixel i, which corresponds to the
x_{i}
; K is the number of lenslets, which captures the point
A
in
Fig. 1
. The joint probability distribution of
y_{i}
,… ,
y_{K}
, since they are independently registered, becomes
It can be assumed that all
n_{i}
’s are equal and proportional to intensity
n_{A}
at the point
A
in the reconstruction plane as illustrated in
Fig. 1
(b), thus Eq. (3) simplifies to
The MLE (maximum likelihood estimation) of
n_{A}
in Eq. (4) is obtained as
which is the average of the photocounts originated from the point
A
.
The sum of the standard deviation of the photocounts over the reconstruction plane is chosen for our metric. It is assumed that the distance level minimizes the sum of standard deviations of the corresponding photocounts. Therefore, the depth
z
to the reconstruction plane is estimated as
where
N_{r}
is the number of voxels in the reconstruction plane,
K_{j}
is the number of the lenslets capturing the voxel
j
, and
y_{j,i}
is the photocounts in the imaging plane associated with the voxel
_{j}
.
Eqs. (6) and (8) are equivalent with Eqs. (9) and (11), which extract distance information in the conventional intensity elemental images as the following
[11]
:
An elemental image array.
Photonlimited elemental image arrays when N_{p} is (a) 1 × 10^{6}, (b) 5 × 10^{6}, (c) 1 × 10^{7}, (d) 5 × 10^{7}.
In the next section, we evaluate the performance of the depth extraction using Eq. (6) with different expected total number of photoevents. The distance extraction derived by intensity images in Eq. (9) is also compared with the photonlimited images.
III. EXPERIMENTAL RESULTS
The II recording system is composed of a microlens array and a pickup camera. The pitch of each lenslet is 1.09 mm, and the focal length of each lenslet is about 3.3 mm. One toy car is used in the experiments.
Figure 2
shows the elemental image array. The size of the elemental image array is 1419×1161 pixels and the number of elemental images is 22×18. Onehundred photonlimited elemental image arrays are generated using a pseudo random number generator on a computer.
Figures 3
(a)(d) show the examples of the photonlimited images while varying photocounts;
N_{p}
varies as 1 × 10
^{6}
, 5 × 10
^{6}
, 1 × 10
^{7}
, and 5 × 10
^{7}
.
Figures 4
is a plot of the sum of the standard deviations obtained from a grayscaled image according to Eq. (9)
[11]
. The sum of the standard deviations is minimized at the depth level of 84 mm.
5
(a)(d) display the average and the errorbar of the standard deviations’sum
Sum of standard deviations (SSD) with a gray scaled elemental image array.
Average and error bar of SSD over 100 photonlimited images when N_{p} is (a) 1 × 10^{6}, (b) 5 × 10^{6}, (c) 1 × 10^{7}, (d) 5 × 10^{7}.
over 100 photon limited images, which are obtained by Eq. (6). As more photoevents are acquired, the photonlimited image starts to resemble the intensity image in
Fig.3
, and the results in
Fig. 5
approach those of
Fig. 4
. In this experiment, the depth level can be extracted when the photocounts exceed 5×10
^{6}
as illustrated in
Fig. 5
(b).
IV. CONCLUSIONS
In this paper, a photoncounting integral imaging method for distance information extraction is proposed. The depth level of the object is determined by the distance at which the sum of the standard deviations of the photocounts is minimized. The method is based on a compact system that requires only a single exposure under passive mode to obtain 3D information. Experimental and simulation results confirm that the proposed method can be used to obtain the distance information to an object at a lowlight level. It was confirmed that the depth level is the same as the intensity images. Further investigation on the distance information extraction of multiple or occluded objects under lowlightlevel conditions remains for future study.
Acknowledgements
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science, and Technology (No. 20110003853).
Schechner Y. Y.
,
Kiryati N.
(1998)
“Depth from defocus vs.stereo: how different really are they?”
Proc. International Conference on Pattern Recognition
1784 
1786
Lee S.W.
,
Kim N.
(2006)
“A method for precise depth detection in stereoscopic display”
J. Opt. Soc. Korea
10
37 
41
Dalmia A. K.
,
Trivedi M.
(1996)
“Depth extraction using a single moving camera: an integration of depth from motion and depth from stereo”
Machine Vision and Applications
9
43 
55
Lippmann G.
(1908)
“La photographie integrale”
C. R. Acad. Sci.
146
446 
451
Son J.Y.
,
Saveljev V. V.
,
Choi Y.J
,
Bahn J.E.
,
Kim S.K
,
Choi H.
(2003)
“Parameters for designing autostereoscopic imaging systems based on lenticular, parallax barrier, and integral photography plates”
Opt. Eng.
42
3326 
3333
Park S.G.
,
Song B.S.
,
Min S.W.
(2010)
“Analysis of image visibility in projectiontype integral imaging system without diffuser,”
J. Opt. Soc. Korea
14
121 
126
Park J.H.
,
Kim J.
,
Lee B.
(2005)
“Threedimensional optical correlator using a subimage array,”
Opt. Express
13
5116 
5126
Hwang D.C.
,
Shin D.H.
,
Kim S.C.
,
Kim E.S.
(2008)
“Depth extraction of threedimensional objects in space by the computational integral imaging reconstruction technique,”
Appl. Opt.
47
D128 
D135
Wu C.
,
McCormick M.
,
Aggoun A.
,
Kung S. Y.
(2008)
“Depth mapping of integral images through viewpoint image extraction with a hybrid disparity analysis algorithm,”
Journal of Display Technology
4
101 
108
Park J.H.
,
Jung S.
,
Choi H.
,
Kim Y.
,
Lee B.
(2004)
“Depth extraction by use of a rectangular lens array and onedimensional elemental image modification,”
Appl. Opt.
43
4882 
4895
Lee D.
,
Yeom S.
,
Kim S.
,
Son J.Y.
(2008)
“Occluded object reconstruction and recognition with computational integral imaging,”
Hankook Kwanghak Hoeji (Korean J. Opt. Photon.)
19
270 
274
Shaked N. T.
,
Katz B.
,
Rosen J.
(2009)
“Review of threedimensional holographic imaging by multipleviewpointprojection based methods,”
Appl. Opt.
48
H120 
H136
Degnan J. J.
2010
“Photon counting lidars for airborne and spacebornetopographic mapping,”
in Proc. Applications of Lasers for Sensing and Free Space Communications (LSC)
San Diego, CA, USA
LSWA4
Hecht E.
2001
Optics
4th ed.
Addison Wesley
San Francisco, USA
Goodman J. W.
1985
Statistical Optics
Johnwiley & Sons, Inc.,
Boston, USA
Refregier Ph.
,
Goudail F.
,
Delyon G.
(2004)
“Photon noise effect on detection in coherent active images,”
Opt. Lett.
29
162 
164
Morris G. M.
(1984)
“Scene matching using photonlimited images,”
J. Opt. Soc. Am. A
1
482 
488
Watson E. A.
,
Morris G. M.
(1990)
“Comparison of infrared upconversion methods for photonlimited imaging,”
J. Appl. Phys.
67
6075 
6084
Yeom S.
,
Javidi B.
,
Watson E.
(2005)
“Photon counting passive 3D image sensing for automatic target recognition,”
Opt. Express
13
9310 
9330
Yeom S.
,
Javidi B.
,
Lee C.W.
,
Watson E.
(2007)
“Photoncounting passive 3D image sensing for reconstruction and recognition of partially occluded objects,”
Opt. Express
15
16189 
16195
Yeom S.
(2011)
“Stereoscopic photon counting passive sensing for extraction of distance information,”
3D Res.
02
03003 