A picoprojector is a compact and mobile projector that has an infinite focus. We apply the picoprojector to a projectiontype integral imaging system, which can expand the image depth to form multiple central depth planes. In a projectiontype integral imaging system, the image flipping problem arises because the expanded elemental images pass through a lens array. To solve this problem, we propose the ray tracing of a picoprojector at a central depth plane and compensate the elemental image using a pixelmapping process. Experiments to verify the proposed method are performed, and the results are presented.
I. INTRODUCTION
It has been demonstrated that integral imaging is an autostereoscopic technique that shows a fullcolor, fullparallax, real threedimensional (3D) image
[1]
. Also, realtime 3D pictures can be provided with relatively low eye fatigue by an integral imaging display. However, integral imaging systems have many challenges, such as a narrow viewing angle, the spatial limitation of resolution, and the expressible depth range. It has been reported that these parameters exhibit a tradeoff relationship
[2]
. Since the first proposal by Lippmann, integral imaging systems have been further developed and improved by many research groups
[3

5]
.
A projector can be used as the display device for the elemental images in an integral imaging system, which is called projectiontype integral imaging. Employing a diffuser screen in the elemental image plane, the projectiontype integral imaging system uses the same method as an ordinary integral imaging system
[6

11]
, but, due to the diffuser, the resolution and brightness of the elemental image are restricted in projectiontype integral imaging. To overcome these weaknesses, previous studies have proposed some kinds of projectiontype systems without the diffuser and reported a method for evaluating the visibility of the reconstructed images in terms of the fill factor
[12]
. This system has the strength that spatial and/or temporal multiplexing is feasible. Also, multiple elemental image layers can be used within the depth of field of the projector
[13
,
14]
.
In this paper, we apply a picoprojector to projectiontype integral imaging, without a diffuser, in realdisplay mode. Due to the infinite focus of the picoprojector, the location of central depth plane (CDP) can be controlled in a projectiontype integral imaging system. Using matrix analysis, ray tracing is performed at the CDP, and the compensation of elemental images for this system is proposed. Some experiments and their results are presented to prove the feasibility of the proposed method.
II. PROJECTIONTYPE INTEGRAL IMAGING SYSTEM USING A PICOPROJECTOR
The picoprojector is a compact and mobile projector that has an infinite focus. As this technology is expected to be embedded in other devices such as smart phones and digital cameras, picoprojectors are drawing much attention. According to its operating method, a picoprojector is categorized according to three types: digital light processing (DLP), liquid crystal on silicon (LCoS), or laser beam steering (LBS).
Figure 1 (a)
shows the basic layout of a LBStype picoprojector
[15]
. As shown, there is no projection lens set. The projected beam with low numerical aperture (NA) leaves the picoprojector directly and creates an image that is always in focus. The projected image grows as its distance from the picoprojector increases, as shown in
Fig. 1(b)
.
(a) Basic layout of an LBStype picoprojector. (b) Field of view of an LBStype picoprojector.
In an integral imaging system, the reconstructed image can be formed near the CDP. The location of the CDP depends on the gap between the lens array and the elemental image, according to the Gauss lens law. In a projectiontype integral imaging system without a diffuser, multiple elemental images can be projected within the depth of field, which generates multiple central depth planes
[16]
. Because the depth of field of a picoprojector is almost infinite, the region of expressible CDP can be expanded by using multiple elemental image planes. As shown in
Fig. 2
, elemental images with different gaps can be projected and integrated into reconstructed images with different CDPs. This means that the location of the CDP can be controlled solely by image processing, without changing the distance between projector and lens array. In this paper, we divide projectiontype integral imaging systems into two cases, to analyze and propose the processing for the compensated elemental images.
Expansion of CDP in an integral imaging system using a picoprojector.
First, as in a conventional integral imaging system, the elemental image is aligned with the gap
g
between the lens array and the displayed image. In this case, when the rate of image expansion is considered, the projected elemental image with gap is equivalent to the expanded image at the lens array. As shown in
Fig. 3
, the size
X
of the elemental image at the lens array is given by
Image flipping problem in a projectiontype integral imaging system.
where
z
is the distance between the picoprojector and the lens array,
g
is the gap between the elemental image and the lens array, and
L
is the total size of the lens array.
Because the size of the elemental image at the lens array is bigger than the lens array, some rays passing through neighboring lenses lead to image flipping. As the gap becomes bigger and the distance
z
becomes smaller, the part of the image passing through neighboring lenses grows.
To solve the image flipping problem in a projectiontype integral imaging system, the size of the elemental image can be matched to that of the elemental lens at the location of the lens array, where the gap is zero in Eq. (1). In this case image flipping does not occur, but the size and location of the reconstructed image are deformed from their intended values, due to gap variation. Therefore, elemental image compensation seems necessary to render the integrated image without distortion or image flipping.
In this paper, we analyze the image transfer matrix for a projectiontype integral imaging system, and propose an elemental image compensation method to resolve the deformation of the reconstructed image, to present the intended size and depth of the integrated image without the image flipping problem.
III. MATRIX ANALYSIS AND PROPOSED SYSTEM
The compensation of the elemental image for the picoprojector is implemented using pixel mapping. Because each pixel has an angle correspondeding to the pixel emitted from the picoprojector, ray tracing at the CDP becomes effective and simple with the matrix method. The elemental image projected at the lens array passes the lens array and travels to the CDP, which is expressed by
where
d_{CDP}
is the distance between the lens array and the CDP,
y_{picoprojector}
is the distance between a ray from the picoprojector and the center of the corresponding lens at the lens array,
θ_{picoprojector}
is the angle of the ray at the lens array,
y_{real}
is the distance between a ray from the picoprojector and the center of the corresponding lens at the CDP,
θ_{real}
is the angle of the ray at the CDP, and
f
is the focal length of the lens array.
The angle
θ_{picoprojector}
depends on the position
y_{picoprojector}
and the distance
z
between the lens array and the picoprojector. Substituting Eqs. (3), Eqs. (2) may be approximated.
where
n
is the number of the corresponding elemental lens. In Eq. (4),
y_{real}
is the linear expression of
y_{picoprojector}
. The magnification is related to the distance
z
, and the shifting distance takes on different values according to the number of the elemental lens in the lens array. In a projectiontype integral imaging system using a picoprojector, the reconstructed image at the CDP is a function of the distance
z
between the lens array and the picoprojector.
A conventional integral imaging system is described by
where
y_{imaging}
is the distance between a ray from the integral imaging system and the center of the corresponding lens at gap
g
, and
θ_{imaging}
is the angle of the ray at the gap
g
.
y_{real}
in Eq. (5) represents the pinholearray approximation; this expression is the same with inverse reaction of the pickup process, but the
y_{real}
in Eq. (4) is different than the inverse reaction of the pickup process.
The elemental image of a projectiontype integral imaging system must be compensated to represent the integrated image at the intended location with the exact intended size. This compensation can be performed to match the corresponding terms of Eq. (4) with Eq. (5), which means
y_{picoprojector}
of Eq. (4) should be calculated using
y_{imaging}
of Eq. (5).
Figure 4
is a schematic diagram of the pixel mapping process for the proposed compensation that adjusts each elemental image according to the change of the subimage region. We implement the proposed algorithm using matrix analysis, and reproduce the elemental image with different
z
and
g
.
Compensation of an elemental image using a pixelmapping process.
IV. EXPERIMENTS AND RESULTS
Figure 5
shows a schematic diagram of the experimental setup. We use the SHOWX+TM laser picoprojector with WVGA resolution (848 × 480). In the experiment, we set the distance
z
between picoprojector and lens array so that the size of the elemental image is aligned at the lens array. The pixel resolution of the elemental image is dictated by the total pitch of the lens array and the distance
z
between picoprojector and lens array. The lens array has 13 × 13 square lenses, the size of each elemental lens being 10 mm × 10 mm. The focal length of the lens array is 15 mm.
Experimental setup.
Like the system shown in
Fig. 2
, two elemental images with different gaps are used to confirm the expansion of the expressible depth. Also, the sizes and locations of the reconstructed images must be adjusted to be like the designed reconstructed images. As shown in
Fig. 6
, images 1 and 2 are intended to represent the reconstructed image. The size of each square in images 1 and 2 is the same. Two elemental images are generated from image 1 with a gap of 2.14 cm, and from image 2 with a gap of 1.8 cm. Comparing the experimental results for three cases, the reconstructed images are observed using a diffuser and measured at the distance
d_{observed}
between lens array and diffuser. In the first case, as shown in
Fig. 7(a)
, the elemental images are not compensated but simply magnified according to the gap g, the expanded elemental images being projected to the lens array. In the second case, as shown in
Fig. 7(b)
, the elemental images are resized and matched to the size of the elemental lens at the position of the lens array. In the last case, as shown in
Fig. 7(c)
, the elemental images are compensated using the proposed method.
Intended images 1 and 2 for computergenerated elemental images.
Projectiontype integral imaging systems and projected elemental images for (a) case 1 (no compensation), (b) case 2 (size adjustment), and (c) case 3 (elemental image compensation).
Figure 8
shows the elemental images on the diffuser located at the different gaps, 2.14 cm and 1.8 cm, generated for case 1 of
Fig. 7
. The elemental image without compensation is used to check the changes of the elemental image planes. In this experiment, two elemental image planes are implemented.
Figure 8(a)
shows the elemental image when the diffuser is set 2.14 cm from the lens array, while in
Figure 8(b)
it is located at 1.8 cm. Comparison of the period of the elemental image and the size of elemental lens show that the elemental image plane for the red image is located at gap 1, while the elemental image plane for the blue image is set at gap 2. The integrated images obtained from these elemental images prove this assumption, as shown in
Fig. 9
, as they are located at the intended position calculated using the lens equation.
Elemental image planes on a diffuser located at (a) g_{2} = 1.8 cm and (b) g_{1} = 2.14 cm.
Experimental results for reconstructed images for the three cases in Fig. 7.
Figure 9
shows the experimental results. The first column in
Fig. 9
shows the results for case 1, while the second and third columns show the results for cases 2 and 3. The reconstructed images for images 1 and 2 are located at each
d_{observed}
, as shown in the first and second rows of
Fig. 9
respectively.
As expected, case 1 presents the image flipping problem, and in case 2 the reconstructed images of elemental images 1 and 2 are located at 5.5 cm and 11 cm respectively, which are not the same as the designed depth. Especially, the reconstructed image of elemental image 2 in the second row shows a problem in terms of its size. Meanwhile, case 3 shows the intended size and depth, without image flipping.
V. CONCLUSION
In this paper, we apply a picoprojector to a projectiontype integral imaging system to expand the CDP. We propose an elemental image compensation method using matrix analysis and an image mapping algorithm. The results of basic experiments show that the proposed system has multiple CDPs without the image flipping problem, unlike a conventional system.
The image flipping problem occurs in a projectiontype integral imaging system due to the divergence property of the projector. In this paper, the compensation is performed assuming low numerical aperture light from the picoprojector. However, the proposed method could also be applied directly to a system adopting a conventional projector, considering the working distance of the chosen projector. So to speak, the proposed compensation method can be used to represent the intended integrated image without a flipped image for a projectiontype integral imaging system in general. Experiments are performed to verify the reconstructed image without the image flipping problem. In this paper, reconstructed images with two different depths are investigated using a diffuser. Plane images are used for convenience in the experiments, yet not only separated plane images but also a continuousvolume image can be represented using the proposed system. However, the visibility of the integrated image might be degraded, so as to be unobservable by the naked eye, because of the small aperture of the projection system. This low visibility is caused by the aperture limitation of the projection system and generally occurs in a projectiontype integral imaging system without a diffuser, as reported in previous studies
[12]
. The main proposal and contribution of this paper concern elemental image compensation to resolve image flipping and deformation, not visibility improvement. The reconstructed image on a diffuser located at the CDP can verify the proposed method to modify the size and location of the integrated images, without image flipping. Visibility enhancement could be achieved by adopting special viewing aids like a 3D screen, which is an optical device to resolve the aperture limitation due to the projection lens set by keeping the parallaxes of a 3D image, and can be a proper solution to improve the visibility of projectiontype integral imaging
[17

19]
. Further studies are needed to apply a 3D screen to the proposed system effectively, which could increase the practical utility of the projectiontype integral imaging system.
Acknowledgements
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2013067321).
Lippmann G.
1908
“La photographie integrale,”
Comptes Rendus Acad. Sci.
146
446 
451
Min S.W.
,
Kim J.
,
Lee B.
2005
“New characteristic equation of threedimensional integral imaging system and its applications,”
Jpn. J. Appl. Phys.
44
L71 
L74
DOI : 10.1143/JJAP.44.L71
Javidi B.
,
Okano F.
2002
Three Dimensional Television, Video, and Display Technologies
Springer
Lee B.
,
Park J.H.
,
Min S.W.
,
Poon T.C.
2006
Digital Holography and ThreeDimensional Display
Springer
“Threedimensional display and information processing based on integral imaging,”
333 
378
Park J.H.
,
Hong K.
,
Lee B.
2009
“Recent progress in threedimensional information processing based on integral imaging,”
Appl. Opt.
48
H77 
H94
DOI : 10.1364/AO.48.000H77
Kim Y.
,
Choi H.
,
Kim J.
,
Cho S.W.
,
Kim Y.
,
Park G.
,
Lee B.
2007
“Depthenhanced integral imaging display system with electrically variable image planes using polymerdispersed liquidcrystal layers,”
Appl. Opt.
46
3766 
3773
DOI : 10.1364/AO.46.003766
Liao H.
,
Iwahara M.
,
Hata N.
,
Dohi T.
2004
“Highquality integral videography using a multiprojector,”
Opt. Express
12
1067 
1076
DOI : 10.1364/OPEX.12.001067
Jang J. S.
,
Oh Y. S.
,
Javidi B.
2004
“Spatiotemporally multiplexed integral imaging projector for largescale highresolution threedimensional display,”
Opt. Express
12
557 
563
DOI : 10.1364/OPEX.12.000557
Jang J.S.
,
Javidi B.
2004
“Threedimensional projection integral imaging using microconvexmirror arrays,”
Opt. Express
12
1077 
1083
DOI : 10.1364/OPEX.12.001077
Okui M.
,
Arai J.
,
Nojiri Y.
,
Okano F.
2006
“Optical screen for direct projection of integral imaging,”
Appl. Opt.
45
9132 
9139
DOI : 10.1364/AO.45.009132
MartínezCuenca R.
,
Navarro H.
,
Saavedra G.
,
Javidi B.
,
MartínezCorral M.
2007
“Enhanced viewingangle integral imaging by multipleaxis telecentric relay system,”
Opt. Express
15
16255 
16260
DOI : 10.1364/OE.15.016255
Park S.g.
,
Song B.S.
,
Min S.W.
2010
“Analysis of image visibility in projectiontype integral imaging system without diffuser,”
J. Opt. Soc. Korea
14
121 
126
DOI : 10.3807/JOSK.2010.14.2.121
Kim Y.
,
Park S.g.
,
Min S.W.
,
Lee B.
2009
“Integral imaging system using a dualmode technique,”
Appl. Opt.
48
H71 
H76
DOI : 10.1364/AO.48.000H71
Jang J.S.
,
Jin F.
,
Javidi B.
2003
“Threedimensional integral imaging with large depth of focus by use of real and virtual image fields,”
Opt. Lett.
28
1421 
1423
DOI : 10.1364/OL.28.001421
Davis W. O.
,
Sprague R.
,
Miller J.
2008
“MEMSbased pico projector display,”
Proc. IEEE/LEOS Int. Conf. Opt. MEMs Nanophoton.
31 
32
Kim Y.
,
Park S.G.
,
Min S.W.
,
Lee B.
2011
“Projectiontype integral imaging system using multiple elemental image layers,”
Appl. Opt.
50
B18 
B24
DOI : 10.1364/AO.50.000B18
Min S.W.
,
Hong J.
,
Lee B.
2004
“Analysis of an optical depth converter used in a threedimensional integral imaging system,”
Appl. Opt.
43
4539 
4549
DOI : 10.1364/AO.43.004539
Song B.S.
,
Choi S.
,
Sung H.
,
Min S.W.
2014
“Reflectiontype threedimensional screen using retroreflector,”
J. Opt. Soc. Korea
18
225 
229
DOI : 10.3807/JOSK.2014.18.3.225
Min S.W.
2014
“A novel 3D screen,”
SPIE Newsroom
http://spie.org/x108294.xml