In this paper the object recognition performance of a photon counting integral imaging system is quantitatively compared with that of a conventional gray scale imaging system. For 3D imaging of objects with a small number of photons, the elemental image set of a 3D scene is obtained using the integral imaging set up. We assume that the elemental image detection follows a Poisson distribution. Computational geometrical ray back propagation algorithm and parametric maximum likelihood estimator are applied to the photon counting elemental image set in order to reconstruct the original 3D scene. To evaluate the photon counting object recognition performance, the normalized correlation peaks between the reconstructed 3D scenes are calculated for the varied and fixed total number of photons in the reconstructed sectional image changing the total number of image channels in the integral imaging system. It is quantitatively illustrated that the recognition performance of the photon counting integral imaging system can be similar to that of a conventional gray scale imaging system as the number of image viewing channels in the photon counting integral imaging (PCII) system is increased up to the threshold point. Also, we present experiments to find the threshold point on the total number of image channels in the PCII system which can guarantee a comparable recognition performance with a gray scale imaging system. To the best of our knowledge,this is the first report on comparisons of object recognition performance with 3D photon counting & gray scale images.
I. INTRODUCTION
Threedimensional (3D) integral imaging and its applications have been investigated for 3D sensing, visualization, display,and recognition of objects
[1

12]
. This technique records 2D elemental (or multi view) images of a 3D scene for 3D image reconstruction and depth extraction. Both optical and numerical reconstructions are possible
[10

11
,
13]
. Recently,photon counting integral imaging has been explored for 3D object reconstruction and recognition for photonstarved scenes. A variety of techniques, including maximum likelihood estimator and statistical sampling theory were applied for 3D imaging and recognition of photonstarved objects, respectively,using a fixed number of elemental images
[13

15]
.
In this paper, we quantitatively compare the object recognition performance of a photon counting integral imaging (PCII)system with that of a conventional gray scale imaging system.Elemental image detection is based on a conventional photon counting model generating different Poisson numbers for each elemental image. A computational ray back propagation algorithm and a parametric maximum likelihood estimator
[16]
are applied to the photon limited elemental images in order to reconstruct the 3D scene. The performance of photon counting image recognition is evaluated by measuring the correlation peaks between the reconstructed 3D scenes.The photonlimited 3D reference object is computationally reconstructed to synthesize a matched filter. Normalized correlation peak values are calculated for the varied and fixed total number of photons in the reconstructed sectional image in order to compare the photon counting recognition performance obtained by changing the total number of the image channels in the PCII system. These results are compared with conventional 2D image recognition using gray scale images. We quantitatively illustrate that the object recognition performance of PCII system can be similar to one obtained by using a general gray scale image as the number of the image viewing channels in integral imaging system is increased up to the threshold point. Also, we show that the threshold point on the total number of image channels in the PCII system, which guarantees a comparable recognition performance with gray scale imaging system, can be found using a statistical sampling distribution of the measured average correlation peak values.
II. OBJECT RECOGNITION WITH PCII SYSTEM
Fig. 1
shows a schematic setup of the integral sensing system with multiple image channels. The scene with very low intensity level light generates the photon counting elemental images. For 3D sensing of photon limited object,the system may utilize a single sensor with lenslet array, a sensor array, or a single moving camera in order to record the low intensity level light ray emanating from a 3D object. Each sensor captures its own photon counting twodimensional(2D) elemental (perspective) image, which contains directional
setup of 3D integral imaging system with multipleelemental images (image channels).
information of the 3D object. The irradiance of one voxel on the surface of the photon limited 3D object is recorded on the corresponding pixel position of each photon limited 2D elemental image. Numerical 3D reconstruction of the original object can be performed by applying computational ray back propagation algorithm and parametric maximum likelihood estimator (MLE) to the recorded photon counted elemental images
[13]
. The pixel values on each elemental image corresponding to one voxel of the photon limited 3D object are assumed to be a random variable following a Poisson distribution function
[17]
. It is previously shown that one voxel value of the original 3D object can be retrieved by applying the parametric MLE to the elemental images pixel values as follows
[13]
:
where
C_{v}
is Poisson random number with the mean parameter
□_{v}
at each voxel of the reconstructed image, subscript
v
denotes a voxel index,
□
is the expected number of photons in the photon limited elemental image,
I_{v}
is the normalized irradiance, and Ne is the total number of elemental images
[13]
. Thus, the original 3D object can be estimated by using this reconstruction procedure. It is interesting to note that a single photon at wavelength
λ
=500 nm carries an energy of
E=hc/λ
= 3.97×10
^{19}
Joules, where
h
is Plank’s constant and c is 3×10
^{8}
meters/second so that the received total energy Et in the reconstructed sectional image of the PCII system can be equal to □
;□N_{e}
[13]
. Therefore, the received photon energy becomes proportional to the total number of the image channel in the PCII system,
N_{e}
.In this paper, it should be noted that the Poisson parameter(mean and variance) values for each pixel in the elemental images in the PCII system are very small. In other words,the probability of counting more than one photon per one pixel in the elemental image is very small.
In order to evaluate the object recognition performance of PCII system, a matched spatial filter is applied to the sectional images reconstructed by using the integral imaging technique as follows
[18]
:
where
PII^{ref}
(
x, y ; z=z_{0}
) and
PII^{ief}
(
x, y ; z=z_{0}
) are sectional images for reference object and input objects, respectively,
z_{0}
is the reconstruction distance, and
F
denotes the Fourier transform.
III. EXPERIMENTAL RESULTS
Experiments to evaluate and compare object recognition performance of an imaging system based on a conven tional photon counting model are presented. We recorded 9×9 elemental images for two toy cars by moving a CCD camera transversally in both x and y directions as shown in
Fig. 1
. Two objects denoted as car I and car II are used as two different classes of data for recognition (see
Fig. 2
).
According to the classical photon counting detection model, the photoncounting elemental images of the toy cars were generated from the recorded elemental images,respectively
[13]
. Then, the sectional images of the 3D scenes for toy cars I and car II were reconstructed at a distance of z
_{0}
= 100cm with their corresponding photoncounting elemental images. We vary the expected number of photons in each input scene to test the recognition performance of the PCII system.
Figure 3
shows the sectional images reconstructed from the photoncounting elemental images of car I and car II, respectively.
The matched filter in Eq. (2) was applied to the reconstructed sectional images in order to inspect object recognition performance of PCII system.
Figure 4
(a) shows the correlation plots of the photon counting image of car I in
Fig. 3
(a) used as a reference with the true class photon counting image as an input data. The true class input data was generated with the same expected number of photons (□=1000) and total number of the elemental images (
N_{e}
=1)as the reference but independent Poisson distributions were used in each case.
Figure 4
(b) shows the correlation plots of the photon counting image of the car I in
Fig. 3
(a) as a reference object with that in
Fig. 3
(b) as a false class
Two toy cars used in the recognition experiments.Each elemental image size is 125×125 pixels. (a) Car I. (b)Car II.
Sectional images reconstructed from the photon counting elemental images of cars I and II. The expected number of photons in the photon limited elemental image □ was 1000. The total number of the elemental images N_{e} was varied from 1 or 81. (a) Reconstructed car I with N_{e}=1 (b)reconstructed car II with N_{e}=1 (c) reconstructed car I with N_{e}=81 (d) reconstructed car II with N_{e} =81.
input data. For comparison, both plots are normalized to the same value that is the autocorrelation of the reference object. The measured correlation peak values were 0.21 and 0.12, respectively. It is noted that it is difficult to make a discrimination between car I and II by using only a single photon counting elemental image with the expected number of photons □=1000.
Figure 5
(a) shows the correlation plots of the photon counting image of the car I in
Fig. 3
(c) as a true reference object (true class) with the true class photon counting image as an input data. The true class input data was generated with the same expected number of photons (□=1000) and total number of the elemental images (
N_{e}
=81)as the reference but with independent Poisson distributions in each case.
Figure 5
(b) shows the correlation plots of the
The correlation plots obtained with the sectional images reconstructed from the small number of photonscounted elemental images. The expected number of photons in the photon limited elemental image □ was 1000. The total number of the elemental images (parallax) N_{e} was 81. (a) Correlation plot between the photon counted image of the car I in Fig. 3(c)as a reference and the true class photon counted image as a input data (b) correlation plot between the photon counted image of the car I in Fig. 3(c) as a reference and the false class photon counted image of car II in Fig. 3(d) as a input data.
The correlation plots obtained with the conventionalgray scale images. (a) Autocorrelation plot of the gray scaleintensity image of car I in Fig. 2(a). (b) crosscorrelation plotbetween the gray scale intensity image of the car I in Fig. 2(a)as a reference object and the false class gray scale intensityimage of car II in Fig. 2(b) as an input data.
The correlation plots obtained with the conventionalgray scale images. (a) Autocorrelation plot of the gray scaleintensity image of car I in Fig. 2(a). (b) crosscorrelation plotbetween the gray scale intensity image of the car I in Fig. 2(a)as a reference object and the false class gray scale intensityimage of car II in Fig. 2(b) as an input data.
photon counting image of car I in
Fig. 3
(c) as a reference object with that in
Fig. 3
(d) for car II as a false class input data. Both plots (
Fig. 5
(a) and
5
(b)) were normalized to the same value that is the autocorrelation of the true reference object. The measured correlation peak values were 0.97 and 0.46, respectively. For comparison, we present the autocorrelation and crosscorrelation plots in
Fig. 6
with the gray scaled intensity images of car I and car II in
Fig. 2
(a) and
2
(b), respectively. The gray scale images in
Fig 2
have the same view point as the photon counting images in
Fig 3
. Both plots (
Fig. 6
(a) and
6
(b)) were normalized to the same value that is the autocorrelation of the true reference object. The autocorrelation and crosscorrelation peak values calculated by using the conventional gray scale intensity images were 1.00 and 0.41, respectively. It is noted that when the total number of the image channels in the integral sensing system was increased to 81, the correlation plots between reference (car I) and unknown input photon counted images (car I and car II) become very similar to those between reference and unknown input intensity images in
Fig. 2
.
Figure 7
shows the correlation peak values computed between the photon counting sectional image of reference car I, and the photon counting sectional image of true class car I or false class car II. The correlation peak values were normalized to the same value that is the autocorrelation of the true reference object. The total number of elemental image
N_{e}
was varied as 1, 9, 25, 49 and 81. In order to obtain the statistical parameters (maximum, mean and minimum values in
Fig. 7
) of the computed correlation peak values between reference and unknown input data,the sectional image of the 3D scenes for the reference car I was reconstructed at distance z
_{0}
= 100 cm with its own photoncounting elemental images. Then, the sectional images of the 3D scenes for the true class car I and false class car II were reconstructed at distance z
_{0}
= 100 cm with their own photoncounting elemental images, respectively.The process is repeated 30 times. Each time, the correlation peak values between reference and unknown input data were calculated.
Correlation peak values computed between the photon counting sectional images of reference car I and the photon counting sectional image of true class car I or false class car II where the total number of elemental images(image channels) N_{e} was varied with 1 9 25 49 and 81. (a) □=100 (b) □=1000 (c) □=10000 where □ is the expected number of photons in the photon limited elemental image.
As shown in
Fig. 7
, the average correlation peak value between the photon counting sectional image of reference car I and the photon counting sectional image of true class car I approaches the autocorrelation peak value (=1.00) of the conventional intensity imaging in
Fig. 2
(a) when the total number of the image channel in the system was increased for fixed a number of photons. It is also noted that the average correlation peak value between the photon counting sectional image of reference car I and the photon counting sectional image of false class car II approaches the crosscorrelation peak value (=0.41) calculated of the conventional intensity imaging of the car I and car II in
Fig. 2
(a) and
2
(b) when the total number of the image channels was increased for fixed number of photons.These experimental results in
Fig. 7
demonstrate that the increasing the number of the image viewing channels in PCII system can enable the recognition performance of PCII system to be similar to one of a conventional gray scale imaging system.
As the experiment to find the threshold point on the total number of image channels for the fixed averaged number of photons □ which guarantees a comparable recognition performance with gray scale imaging system,
Table 1
and
2
show the average correlation peak values computed between the photon counting sectional image of reference car I and the photon counting sectional image of true class car I or false class car II changing the number of the image channels, respectively, where the expected number of photons in the photon limited elemental image was fixed as 10, 1000, 2000, 3000, 4000, 5000, 6000,7000, 8000, 9000, or 10000. The correlation peak values in the
Table 1
and
2
were normalized to the same value that is the autocorrelation of the true reference object. The
Average correlation peak values computed between the photon counting sectional images of reference car I and the photoncounting sectional image of true class car I
Average correlation peak values computed between the photon counting sectional images of reference car I and the photoncounting sectional image of true class car I
Average correlation peak values computed between the photon counting sectional images of reference car I and the photoncounting sectional image of false class car II
Average correlation peak values computed between the photon counting sectional images of reference car I and the photoncounting sectional image of false class car II
sectional image tested in the experiments has 125×125 pixels.It can be assumed that an arbitrary cutoff correlation value (for example 0.95 or 0.99) utilized in the conventional gray scale imaging system can be a possible decision criterion to evaluate the recognition performance in the 3D PCII system. Therefore, the threshold point on the total number of image channels for the arbitrary fixed averaged number of photons in the elemental image may be empirically found with the average correlation peak value measured in experiments. For example, if the average correlation peak value as the decision rule for an object recognition is set as 0.9500 or 0.9900 with the fixed averaged number of photons □=6000, the threshold point on the total number of image channels can be 9 and 49,respectively as shown in
Table 1
. It seems very difficult to derive the closedform to describe the relationship between the optical parameters ( □,
N_{e}
, z
_{0}
, and reconstructed sectional image size) in the PCII system in order to find the threshold point on the total number of image channels which guarantees a comparable recognition performance with gray scale image. However, it can be possible to empirically form the statistical distributions of the average correlation peak values for the training objects to classify the object of interest with the given optical parameters in PCII system. Then, the threshold point on the total number of image channels can be found using the statistical distribution of the average correlation peak values for the training objects.
The reconstructed sectional image can be characterized by the photon number per pixel, the total number of elemental images
N_{e}
and the expected number of photons □ in the photon limited elemental image. It is noted that the sectional image tested in the experiments with □ =6000 and
N_{e}
=9 has only 3.456 photons per pixel. It means that the received total energy in the reconstructed sectional image required for obtaining a comparable recognition performance with gray scale imaging system would be approximately 2.14×10
^{14}
Joules with an average wavelength of 500 nm. It is also interesting to note that the average correlation peak value is kept as 0.9500 when the total number of photons in the reconstructed sectional image
N_{p}
is approximately fixed as 50000 (in other words, the received total energy in the reconstructed sectional image is approximately fixed as 2.14×10
^{14}
Joules) even though the expected number of photons □ in the elemental image was decreased. These experimental results quantitatively illustrate that the recognition performance of the PCII system can be similar to one of conventional gray scale imaging system with less power consumption on condition that any noise sources in our analysis/experiments are not included. In addition, it is shown that the threshold point on the total number of image channels in the PCII system,which guarantees a comparable recognition performance with gray scale imaging system, can be found using the statistical distribution of the average correlation peak values measured in the experiments. In our next work, we are planning to analyze the recognition performance in the PCII system with noise sources.
IV. CONCLUSION
In summary, we have quantitatively compared object recognition performance of PCII system with that of object recognition using conventional gray scale images. For visualization of a scene with small number of photons, integral imaging technique and parametric maximum likelihood estimator can be applied to the photon limited elemental image of the object. To evaluate photon counting object recognition performance, normalized correlation peak values between the reconstructed reference object and unknown input objects are computed for the varied total number of photons or the fixed one in the reconstructed sectional image changing the total number of image channels in the PCII system. The results are compared with those measured between the reference object and unknown input images obtained by conventional gray scale imaging system. It is shown that photon counting object recognition performance can be similar to one of conventional gray scale imaging system as the total number of the image channels in PCII system is increased up to the threshold point. Also, it is shown that the threshold point on the total number of image channels in the PCII system can be found using the statistical distribution of the average correlation peak values measured in the experiments. The PCII system to require less power consumption may replace the conventional imaging systems that generate gray scaled images for object recognition purposes.
Acknowledgements
This study was supported by research fund from Chosun University, 2010.
Lippmann G
1908
La photographic integrale
C. R. Acad. Sci.
146
446 
451
Ives H. E
1931
Optical properties of a Lippmann lenticuled sheet
J. Opt. Soc. Am.
21
171 
176
Javidi B
,
Okano F
,
Son J
2008
Threedimensional Imaging Visualization and Display Technologies
Springer
New York USA
Wei T
,
Shin D
,
Lee B
2009
Resolutionenhanced reconstruction of 3D object using depthreversed elemental images for partially occluded object recognition
J. Opt.Soc. Korea
13
139 
145
MartinezCuenca R
,
Saavedra G
,
MartinezCorral M
,
Javidi B
Progress in 3D multiperspective display by integral imaging
Proc. IEEE
97
1067 
1077
Okoshi T
1980
Threedimensional displays
Proc. IEEE
68
548 
564
Okano F
,
Arai J
,
Mitani K
,
Okui M
2006
Realtime integral imaging based on extremely high resolution video system
Proc. IEEE
94
490 
501
Park S
,
Song B
,
Min S
Analysis of image visibility in projectiontype integral imaging system without diffuser
J. Opt. Soc. Korea
14
121 
126
Igarishi Y
,
Murata H
,
Ueda M
1978
3D display system using a computergenerated integral photograph
Jpn. J.Appl. Phys.
17
1683 
1684
Arimoto H
,
Javidi B
2001
Integrate threedimensional imaging with computed reconstruction
Opt. Lett.
26
157 
159
Stern A
,
Javidi B
2006
Threedimensional image sensingvisualization and processing using integral imaging
Proc.IEEE
94
591 
607
Levoy M
2006
Light fields and computational imaging
IEEE Computer Mag.
39
46 
55
Tavakoli B
,
Javidi B
,
Watson E
2008
Three dimensional visualization by photon counting computational integral imaging
Opt. Express
16
4426 
4436
Yeom S
,
Javidi B
,
Watson E
2005
Photon counting passive 3D image sensing for automatic target recognition
Opt.Express
13
9310 
9330
Moon I
,
Javidi B
2009
Threedimensional recognition of photonstarved events using computational integral imaging and statistical sampling
Opt. Lett.
34
731 
733
Guillanume M
,
Melon P
,
Refregier P
1998
Maximumlikelihood estimation of an astronomical image from a sequence at low photon levels
J. Opt. Soc. Am. A
15
2841 
2848
Goodman J. W
1985
Statistical Optics
John Wiley & Sonsinc.
Hoboken USA
Javidi B
,
Tajahuerce E
2000
Threedimensional object recognition by use of digital holography
Opt. Lett.
25
610 
612