Advanced
Three-Dimensional Face Point Cloud Smoothing Based on Modified Anisotropic Diffusion Method
Three-Dimensional Face Point Cloud Smoothing Based on Modified Anisotropic Diffusion Method
International Journal of Fuzzy Logic and Intelligent Systems. 2014. Jun, 14(2): 84-90
Copyright © 2014, Korean Institute of Intelligent Systems
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • Received : June 02, 2014
  • Accepted : June 24, 2014
  • Published : June 25, 2014
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Suryo Adhi Wibowo
Sungshin Kim

Abstract
This paper presents the results of three-dimensional face point cloud smoothing based on a modified anisotropic diffusion method. The focus of this research was to obtain a 3D face point cloud with a smooth texture and number of vertices equal to the number of vertices input during the smoothing process. Different from other methods, such as using a template D face model, modified anisotropic diffusion only uses basic concepts of convolution and filtering which do not require a complex process. In this research, we used 6D point cloud face data where the first 3D point cloud contained data pertaining to noisy x -, y -, and z -coordinate information, and the other 3D point cloud contained data regarding the red, green, and blue pixel layers as an input system. We used vertex selection to modify the original anisotropic diffusion. The results show that our method has improved performance relative to the original anisotropic diffusion method.
Keywords
1. Introduction
The current trend of face recognition research is toward greater use of three-dimensionl (3D). This trend is due to the rapid evolution of technologies that have enabled the development of image acquisition tools such as 3D scanners and the Microsoft Kinect sensor. In addition, the emphasis of research has moved towards 3D in order to improve performance accuracy of 2D face recognition under difficult conditions. To obtain highly accurate results, appropriate preprocessing steps are needed because their result will affect the results and processes of feature extraction and classification.
Mian et al. [1] have researched automatic 3D face detection, normalization, and recognition. They used data from the face recognition grand challenge (FRGC), which was taken from the Minolta Vivid 900/910 series sensor. The CyberwareTM 3030PS laser scanner was used by Blanz and Vetter [2] to record 3D face data. It was also used as an interactive tool for filling holes and removing spikes. Unfortunately, the price of these data acquisition devices is prohibitive. Although the 3D face data has high resolution [2] ; it requires a smoothing process in preprocessing. They used a median filter for the smoothing process. Li et al. [3 , 4] used the Microsoft Kinect sensor, which is the cheapest hardware available for 3D face recognition. Resampling and symmetric filling was used for smoothing. Chen et al. [5] used geometric hashing and a sample training set of labeled images as their denoising method. Mean and median filtering are also used for mesh smoothing which has been applied to face data [6] ; to get a smooth result, these methods require many iterations. However, to improve the results of the 3D point cloud texture in the preprocessing step, especially using RGBD images, a researcher usually needs the assistance of a 3D face template model [2 - 4] . When a 3D face template model is used, a fitting method is absolutely necessary. Iterative closest point (ICP) is used to fit a 3D face point cloud to a 3D face template model [3] , but this method requires a long computation time. In 3D face recognition [7 , 8] , an affine transform or homogeneous transform is used to replace the ICP method for fitting [4] . We emphasize that these methods are not very efficient.
Anisotropic diffusion is the method introduced by Perona and Malik [9] . This method is very useful because it is simpler than the others. Anisotropic diffusion can be implemented on 1D, 2D, and 3D data. Gerig et al. [10] used anisotropic diffusion for filtering MRI data where it has been applied to 2D and 3D image data with one or two channels [11 , 12] . Linear anisotropic diffusion mesh filtering has also been investigated [13] .
Anisotropic diffusion has also been used for geometric surface smoothing [14] and has applications in image processing [15] .
In this paper, we propose a smoothing 3D face point cloud method based on modified anisotropic diffusion. We use input data from the Curtin Faces Database [4] . The data are derived from the Microsoft Kinect, which produced red, green, blue, and depth (RGB-D) data. These data have been processed into 6D data. Because the 3D point cloud data is noisy we used the relation between 2D data and 3D point cloud data to process it more easily. To denoise and smooth the 3D point cloud, we used the basic anisotropic diffusion method. However, we were required to modify the original anisotropic diffusion to increase performance because the original anisotropic diffusion could not compute perfectly in our case. Selected vertices have been used to detect the value from vertices.
The contributions of this research are as follows.
  • 1)We have developed an easier way to process 3D face point cloud data that uses the relation between 2D images and 3D object point clouds.
  • 2)We have developed a robust, modified smoothing method, where the output has a smooth texture and the same number of vertices as the input in the smoothing process. Our proposed method could be implemented with the Microsoft Kinect, which is a cheaply priced device for 3D object processing.
The paper is structured as follows: In Section 2, we present the method and approach. In Section 3, performance evaluation and results are described. In Section 4, we present our conclusions.
2. Method and Approach
Triangle mesh is one method that can represent a 3D object based upon vertices and face data. This method is simpler than the other; however, we must know about the relationship between vertices and faces. It also requires complex computation to process, if we only use vertices and faces as the information. We propose that the easiest way to process a 3D object point cloud is by using the relation between a 2D image and a 3D object point cloud. Figure 1 shows the result of each step of the cropping process for a 3D face point cloud.
PPT Slide
Lager Image
(a) RGB image, (b) chrominance red image, (c) cropped image, (d) 3D point cloud, (e) cropped 3D face point cloud before multiply with NaN and (f) cropped 3D face point cloud after multiply with NaN.
- 2.1 Point Cloud Reconstruction
We used a data sample from the Curtin Faces Database [4] , which has 307,200 rows and six columns of point cloud matrix data for each subject. For easy calculation, we should rearrange the point cloud matrix to 460 rows and 640 columns, where the first, second, and third columns contain x -, y -, and z -coordinate data, respectively. The fourth, fifth, and sixth columns contain data regarding the red, green, and blue layers, respectively. Eq. (1) illustrates the data in a 6D point cloud.
PPT Slide
Lager Image
where PC 1.1 ··· PC 1.307,200 , PC 6.1 ··· PC 6.307,200 are the first and last columns. It has length 307,200. A 1 ··· A 6 pertains to the rearranged result of the point cloud to the matrix i, j where i = 480 and j = 640.
- 2.2 Skin Detection
For skin or face detection, we can use existing algorithms such as Lucas-Kanade, Viola-Jones, etc. However, in this experiment, we used a simple method to detect the face in 2D images. We used the YCbCr color format to separate the face from the background. We know that Ak is point cloud data, which has x -( k = 1), y -( k = 2), and z -coordinates( k = 3), as well as red( k = 4), green( k = 5), and blue layer pixel information( k = 6), where the sizes are 480 rows and 640 columns, respectively. Because it is equivalent to the red, green, and blue layer color format, we converted it to the YCbCr color format for skin detection. The chrominance red layer from the result of converting the color format has much more information than the other because skin colors are strongly influenced by red pixels.
- 2.3 3D Face Point Cloud Cropping
The chrominance red layer was converted to the monochrome format for easy cropping of 2D faces. Because the object and environmental conditions are always changing, an adaptive threshold should be used to obtain an optimal result for the black and white color format. To solve this problem, we used the Otsu method to adaptively calculate the threshold. Although the adaptive threshold was already used in this system, the result of this system has some noise. We used a median filter and selected an area to reduce its noise. We used a kernel size of 7 × 7 for the median filter. Although using the median filter caused closing and opening of pixels in the image Imf , the result still provides much better noise reduction. To remove undesired objects, we selected by area for each object which had the label lb before
PPT Slide
Lager Image
where Irn and Imf are the image result from noise removal and the image result from the median filter, respectively. We used a threshold th of 5,000 to select the area. The Face part could be determined if it met the requirements that have been defined before. The histogram was computed from Irn and then cropping face part depended on information from the histogram. Eqs. (2), (3), and (4) describe cropping by using histogram information
PPT Slide
Lager Image
PPT Slide
Lager Image
PPT Slide
Lager Image
where Irns1 , Irns2 are the sum of pixels by rows and columns in Irn , respectively. After computation of the histogram, we must find the histogram’s result that is not equal to zero. Maximum and minimum boundaries for cropping coordinates can be determined from these steps. Pmin1 , Pmin2 is the minimum boundary, Pmax1 , Pmax3 is the maximum boundary, and Δ 1 2 , Δ 3 are the thresholds for cropping. Δ 1 , Δ 2 , Δ 3 have the values are 125, -23, and 130, respectively. The variable Ifc represent the cropping face from 2D image that can be done after we solved Eq. (5).
Because A 4···6 is related to A 1···3 , a 3D face point cloud could be extracted using Eq. (5) and we can call it Dck . An additional steps are required to capture the texture of a 3D face. The first step is multiplying it by the 2D image, which is the result from face cropping Ifc ,
PPT Slide
Lager Image
Unfortunately, the result of the extraction process still has some noise. This happens because point cloud x -, y -, and z - coordinates are always sensitive to deformation. To solve this problem, we take a second step to calculate the absolute value that represents absolute face texture
PPT Slide
Lager Image
,
PPT Slide
Lager Image
The result usually has range 0 ≤
PPT Slide
Lager Image
≤ 3, 000, depending on the distance when we acquire the data.
PPT Slide
Lager Image
Eq. (8) describes idxface , the index location of 3D face point cloud. Now we have the rough 3D face point cloud Droughface , because we have the index face information.
In Droughface , there are some minor noises that can cause strange shapes on the 3D face point cloud. To remove this noise, we suggest following Eqs. (9) and (10).
PPT Slide
Lager Image
PPT Slide
Lager Image
where O is the matrix that has same size as the input system. α is the constant that should be added to remove noise. We used α equal to NaN . This is the key to our research. Because is NaN implemented in our system, it can eliminate the noise.
- 2.4 3D Face Point Cloud Smoothing
Because a 3D face point cloud has nonlinear characteristics, the anisotropic diffusion method has been used to smooth it. Anisotropic diffusion extends isotropic diffusion with a nonlinear term limiting diffusion across boundaries defined by a large gradient in image intensity. In order to enable the isotropic diffusion to preserve features, Perona and Malik [7] modified it.
PPT Slide
Lager Image
Eq. (11) describe anisotropic diffusion, where div is the divergence operator with ∇,Δ is the gradient and Laplacian operator with respect to the space variables [7] . Eq. (11) also can be defined as
PPT Slide
Lager Image
(12) where I is the image, s and p are pixel positions, η 4 (s) is the 4-neighborhood of the pixel at s , λ is a scalar controlling the rate of diffusion, t is a discrete time step, and g(x) is an edge stopping function. Perona and Malik [7] propose g(x) that has two functions.
PPT Slide
Lager Image
PPT Slide
Lager Image
where σ is a parameter controlling how large a change in pixel intensity is considered an edge [16] .
In our system, we computed 3D face data like 2D images, so I in Eq. (12), can be replaced by
PPT Slide
Lager Image
. Because the z -coordinate has a strong influence on the texture of a 3D face point cloud, we only processed it using anisotropic diffusion. Unfortunately, because of the cropping process, there are some point cloud values that could not be computed perfectly using anisotropic diffusion. To solve this challenge, we modified the anisotropic diffusion by
PPT Slide
Lager Image
PPT Slide
Lager Image
where V S is selected vertices and V S (0) is the V S that has NaN value, and convert it equal to zero. For original anisotropic diffusion, we used public code on mathwork [17] .
3. Performance Evaluation and Result
We tested the proposed method using data from the Curtin Faces database. In this system, we propose three scenarios to investigate the method. The first scenario is a performance comparison of smoothing using five different angles as an input. The second scenario is a performance comparison of smoothing using five different faces, but the same angle as an input. The last scenario is to investigate the performance of smoothing by iteration. The processes were carried out on a 2.4 GHz Intel core i3 processor with 2 GB memory.
- 3.1 Performance Comparison
In this experiment, we compared our proposed method and the original anisotropic diffusion method. In the first scenario, the subject was one female face posed at various angles. The angles that we used in this experiment were 0°, 60°, ―60°, 90°, and ―90°. Table 1 shows the simulation result for 0°, 60°, ―60°, 90°, and ―90°. From this table, modified anisotropic diffusion substantially improves the original anisotropic diffusion performance at all angles. Our proposed method can maintain the same number of vertices input in the smoothing process. The computation time of our modified anisotropic diffusion and the original anisotropic diffusion is almost the same; there is no significant difference between the methods.
Results of comparison methods with different angles
PPT Slide
Lager Image
MA, modified anisotropic diffusion; A, anisotropic diffusion.
In the second scenario, we used two females and one male face, all posed at the same angle. We used a 0° angle in this experiment. Table 2 shows that modified anisotropic diffusion also can maintain the same number of vertices input in the smoothing process. Both methods have almost the same computation time; there is no significant difference between our modified anisotropic diffusion and the original anisotropic diffusion.
Results of comparison methods with different faces
PPT Slide
Lager Image
MA, modified anisotropic diffusion; A, anisotropic diffusion.
- 3.2 Iteration Effect
We also investigate the influence of iteration on smoothing process. We used the follow number of iterations at each stage of our experiment: 2, 4, 6, 8, 10, 20, 30, and 40. Figure 2 shows the results from our experiment. From these results, we can see that when the number of iterations was less than 10, the result still left some noise in the texture of 3D face point cloud. A smooth result was only reached when the number of iterations exceeded 10. Unfortunately, these smooth results could cause the disappearance of some features in the 3D face point cloud.
PPT Slide
Lager Image
The 3D face point cloud smoothing result arranged by the number of iterations. (a) using 2 iterations, (b) using 4 iterations, (c) using 6 iterations, (d) using 8 iterations, (e) using 10 iterations, (f) using 20 iterations, (g) using 30 iterations, (h) using 40 iterations.
The number of iterations that produced smooth texture in the 3D face point cloud while preserving its features is 10. In 10 iterations, we can still can see features from the 3D face point cloud such as the eye, nose, and mouth regions.
4. Conclusion
We have presented a modified anisotropic diffusion for smoothing a 3D face point cloud. The key idea of our research is how to smooth 3D face point clouds, the results of which must have the same shape as the original input smoothing and also have the same number of vertices. We have demonstrated that our modification method is more robust than the original anisotropic diffusion. Our method maintains the same shape and number of vertices as the original input in the smoothing process.
Our method is more robust than the original anisotropic diffusion because we modified it by selecting vertices which have values that could not be computed perfectly in anisotropic diffusion. Different faces and different angles in the 3D face point cloud did not influence our system. Our result still has the same shape and number of vertices. There are also no significant differences between the computation times of our modified anisotropic diffusion and the original anisotropic diffusion. The optimal number of iteration is 10 because it will produce a smooth result and preserve the appearance of 3D face point cloud features.
No potential conflict of interest relevant to this article was reported.
Acknowledgements
This work was supported by BK21PLUS, Creative Human Resource Development Program for IT Convergence and supported by a 2-Year Research Grant of Pusan National University.
BIO
Suryo Adhi Wibowo received his B. Eng. Degree and M. Eng. degree in Electrical Engineering Majoring Telecommunication Engineering from Telkom Institute of Technology, Indonesia, in 2009 and 2012, respectively. He is currently a Ph.D. candidate at Department of Electrical and Computer Engineering, Pusan National University, Korea. His research interests include computer vision, intelligent system, pattern recognition and signal analysis.
E-mail: suryo@pusan.ac.kr.
Sungshin Kim received his B.S. and M.S. degrees in Electrical Engineering from Yonsei University, Korea, in 1984 and 1986, respectively, and his Ph.D. degree in Electrical Engineering from the Georgia Institute of Technology, USA, in 1996. He is currently a professor at the Electrical Engineering Department, Pusan National University. His research interests include fuzzy logic controls, neuro fuzzy systems, neural networks, robotics, signal analysis, and intelligent systems.
E-mail: sskim@pusan.ac.kr
References
Mian A , Bennamoun M , Owens R 2006 “Automatic 3D face detection, normalization and recognition,” Proceedings of the 3rd International Symposium on 3D Data Processing, Visualization, and Transmission Chapel Hill, NC June 14-16, 2006 http://dx.doi.org/10.1109/3DPVT.2006.32 735 - 742    DOI : 10.1109/3DPVT.2006.32
Blanz V. , Vetter T. 2003 “Face recognition based on fitting a 3D morphable model,” IEEE Transactions on Pattern Analysis and Machine Intelligence http://dx.doi.org/10.1109/TPAMI.2003.1227983 25 (9) 1063 - 1074    DOI : 10.1109/TPAMI.2003.1227983
Li B. Y. L. , Mian A. S. , Liu W. , Krishna A. “Using Kinect for face recognition under varying poses, expressions, illumination and disguise,” IEEE Workshop on Applications of Computer Vision Tampa, FL January 15-17, 2013 http://dx.doi.org/10.1109/WACV.2013.6475017 186 - 192    DOI : 10.1109/WACV.2013.6475017
Li B. Y. L. , Liu W. , An S. , Krishna A. 2012 “Tensor based robust color face recognition,” Proceedings of the 21st International Conference on Pattern Recognition Tsukuba, Japan November 11-15, 2012 1719 - 1722
Chen Y. L. , Wu H. T. , Tong F., Shi, X. , Chai J. “Accurate and robust 3D facial capture using a single RGBD camera,” Proceedings of the IEEE International Conference on Computer Vision Sydney, Australia December 1-8, 2013 http://dx.doi.org/10.1109/ICCV.2013.449 3615 - 3622    DOI : 10.1109/ICCV.2013.449
Yagou H. , Ohtake Y. , Belyaev A. “Mesh smoothing via mean and median filtering applied to face normals,” Proceedings of the Geometric Modeling and Processing Wako, Japan July 10-12, 2002 http://dx.doi.org/10.1109/GMAP.2002.1027503 124 - 131    DOI : 10.1109/GMAP.2002.1027503
Ma C. M. , Yoo S. H. , Oh S. K. 2012 “Design of face recognition algorithm based optimized pRBFNNs using three-dimensional scanner,” Journal of Korean Institute of Intelligent Systems http://dx.doi.org/10.5391/JKIIS.2012.22.6.748 22 (6) 748 - 753    DOI : 10.5391/JKIIS.2012.22.6.748
Choi S. H. , Cho S. , Chung S. T. 2010 “Improvement of face recognition speed using pose estimation,” Journal of Korean Institute of Intelligent Systems http://dx.doi.org/10.5391/JKIIS.2010.20.5.677 20 (5) 677 - 682    DOI : 10.5391/JKIIS.2010.20.5.677
Perona P. , Malik J. 1990 “Scale-space and edge detection using anisotropic diffusion,” IEEE Transactions on Pattern Analysis and Machine Intelligence http://dx.doi.org/10.1109/34.56205 12 (7) 629 - 639    DOI : 10.1109/34.56205
Gerig G. , Kubler O. , Kikinis R. , Jolesz F. A. 1992 “Nonlinear anisotropic filtering of MRI data,” IEEE Transactions on Medical Imaging http://dx.doi.org/10.1109/42.141646 11 (2) 221 - 232    DOI : 10.1109/42.141646
Jang H. , Ko H. , Choi Y. , Han Y. , Hahn H. 2005 “A new face tracking method using block difference image and Kalman filter in moving picture,” Journal of Korean Institute of Intelligent Systems http://dx.doi.org/10.5391/JKIIS.2005.15.2.163 15 (2) 163 - 172    DOI : 10.5391/JKIIS.2005.15.2.163
Oh S. K. , Oh S. H. , Kim H. K. 2013 “Design of threedimensional face recognition system using optimized PRBFNNs and PCA: comparative analysis of evolutionary algorithms,” Journal of Korean Institute of Intelligent Systems http://dx.doi.org/10.5391/JKIIS.2013.23.6.539 23 (6) 539 - 544    DOI : 10.5391/JKIIS.2013.23.6.539
Taubin G. “Linear anisotropic mesh filtering,”
Tasdizen T. , Whitaker R. , Burchard P. , Osher S. “Geometric surface smoothing via anisotropic diffusion of normals,” Proceedings of the IEEE Visualization Boston, MA November 1, 2002 http://dx.doi.org/10.1109/VISUAL.2002.1183766 125 - 132    DOI : 10.1109/VISUAL.2002.1183766
Weickert J. 1998 Anisotropic Diffusion in Image Processing B.G. Teubner Stuttgart, Germany http://www.lpi.tel.uva.es/muitic/pim/docus/anisotropic diffusion.pdf
Jones T. R. 2003 “Feature preserving smoothing of 3D surface scans,” M.S. thesis Massachusetts Institute of Technology Cambridge, MA
Lopez D. “Anisotropic diffusion (Perona & Nalik),” http://www. mathworks.com/matlabcentral/fileexchange/14995-anisotropic-diffusion-perona-malik