Advanced
Lane Detection and Tracking Using Classification in Image Sequences
Lane Detection and Tracking Using Classification in Image Sequences
KSII Transactions on Internet and Information Systems (TIIS). 2014. Dec, 8(12): 4489-4501
Copyright © 2014, Korean Society For Internet Information
  • Received : July 22, 2014
  • Accepted : November 05, 2014
  • Published : December 31, 2014
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Sungsoo Lim
Department of Electronics and Radio Engineering, Kyung Hee University Youngin, Korea
Daeho Lee
Humanitas College, Kyung Hee University Yongin, Korea
Youngtae Park
Department of Electronics and Radio Engineering, Kyung Hee University Youngin, Korea

Abstract
We propose a novel lane detection method based on classification in image sequences. Both structural and statistical features of the extracted bright shape are applied to the neural network for finding correct lane marks. The features used in this paper are shown to have strong discriminating power to locate correct traffic lanes. The traffic lanes detected in the current frame is also used to estimate the traffic lane if the lane detection fails in the next frame. The proposed method is fast enough to apply for real-time systems; the average processing time is less than 2msec. Also the scheme of the local illumination compensation allows robust lane detection at nighttime. Therefore, this method can be widely used in intelligence transportation systems such as driver assistance, lane change assistance, lane departure warning and autonomous vehicles.
Keywords
1. Introduction
I n many advanced driver assistance systems (ADASs), land detection and tracking are practical utilized to detect dangerous situations and take an active part in the driving [1] . The main processes of these systems usually consist of road detection, lane detection and obstacle detection. A robust lane detection method enables the systems to easily distinguish obstacles and traffic structures. Therefore, the lane detection can be applied as the core process for ADASs.
For lane detection, various methods based on Hough transform (HT) are widely used [2 - 5] . HT-based methods have several bottlenecks in the development of commercializable systems as follows [2] : 1) Standard HT has complex computation for every mapped parametric point, 2) Standard and modified HT are often difficult to determine artifacts and lane boundaries, 3) They must be applied after preprocessing such as edge detection, and 4) They are sensitive to shadows, road marks and so on.
Methods using tangent vectors were shown to be quite robust on well-marked lanes, but they may fail to detect poor-marked lanes [6 , 7] . And Lane-finding in ANother domAin (LANA) algorithm [8] using frequency domain features was proposed, but it also used another representation of edge.
Several methods using pixel clustering are proposed; however, they need excessive computation time. Mathematical model-based methods can be used to fit the lane boundaries; however, simple models cannot correctly estimate lane information and complicated ones are vulnerable to disturbance [9] .
Classification methods based on training for lane or road detection were proposed; however, their performance may not be guaranteed when road and illumination conditions are not encountered in training [2 , 10 , 11] . The performances of some state-of-the-arts methods using multiple kernel sparse representation based on classifiers [12] or adaptive hypergraph learning [13] seem to have high performances; however, they need complex features and classifiers.
In this paper, we propose a robust lane detection scheme based on the use of both structural and statistical features of candidate lane marks. Lane marks are usually characterized by bright line features. To discriminate the lane marks from various types of road marks such as arrows, rectangles or diamond-shaped road marks, the horizontal projection function of the candidate shape and its derivation are fed to the neural network. The performance of the lane detection is shown to be robust to the variation of illumination conditions.
In addition, the internal angle of the vanishing point to track lanes in the current frame is used to locate the lane marks if track lanes are not found in the next frame. Most of all other methods are designed for daytime environments; however, the proposed method can be used at nighttime. The average processing time of this algorithm is less than 2msec so it is fast enough for real-time operation. It is overwhelmingly faster than other methods using HT and pixel clustering.
2. Classification-Based Lane Mark Detection
Because the terms related to roadways are confusedly described in many places, we first define some terms as shown in Fig. 1 . A lane is a guidance region for use by a single line of vehicle, and lane marks are the solid or dash painted lines to divide lanes, traffic lines are both side lines to separate lanes, and a vanishing point is the cross point of both side traffic lines.
PPT Slide
Lager Image
Definition of terms related to roadways.
To detect and track traffic lanes, the overall scheme of the proposed method is shown in Fig. 2 . Traffic marks belonging to lane marks are brighter than road color, so we detect bright evidences in traffic images. Local illumination conditions are, however, various because of camera angles and lighting conditions, so we first compensate the local illumination variables by a DC-notch filter [14] . The input luminance image f ( x , y ) is used for lane detection, so the filtered image g ( x , y ) is calculated by
PPT Slide
Lager Image
where
PPT Slide
Lager Image
is the local average calculated by a N × N local convolution mask.
PPT Slide
Lager Image
Overall scheme of the proposed method.
Bright evidences are segmented by binarization using the threshold method. The threshold is simply calculated by the Otsu’s method. However, the traffic marks are not obviously observed in g ( x , y ) in nighttime images as shown in Fig. 3 . So we perform the contrast stretching as follow:
PPT Slide
Lager Image
where Cg and
PPT Slide
Lager Image
denote the contrast ranges of g ( x , y ) and
PPT Slide
Lager Image
, respectively,
PPT Slide
Lager Image
is the stretched image, and i 5 is the intensity of the upper 5 percent for the histogram of g ( x , y ). It means
PPT Slide
Lager Image
is represented as g ( x , y )Ⅹ255/ i 5 for g ( x , y ) below i 5 and 255 for values upper i 5 to linearly enhance the contrast of the most popular intensity level (lower 95 percent).
PPT Slide
Lager Image
f(x,y), g(x,y) and .
Followed by binarization, bright evidence regions are labeled into Bi (0< i < n , where n is the number of the regions) by connect component analysis, and their bounding boxes are built. As shown in Figs. 4 and 5 , the shape of Bi belonging to lane marks is similar with line segments, but the thickness is gradually grown in the down direction because of perspective. So, we use the horizontal projection function p ( y ) defined by
PPT Slide
Lager Image
where w is the width of B ,and the differential of the projection function p '( y ) defined by
PPT Slide
Lager Image
to calculate features. By the above clues, we use standard deviations of p ( y ) and p '( y ), σ ( p ( y )) and σ ( p '( y )), for features of classification.
PPT Slide
Lager Image
p(y) and p'(y) of Bi belonging to lane marks.
PPT Slide
Lager Image
p(y) and p'(y) of Bi not belonging to lane marks.
In addition, the size, position and thickness are useful information for classifying lane marks, so we use the width and height of bounding boxes ( w and h ), the x - y coordinate of center of bounding boxes ( cx and cy ), and the average thickness, that is the mean of p ( y ), μ ( p ( y )), as features. Because these features ( w , h , cx , cy , μ ( p ( y )), σ ( p ( y )) and σ ( p '( y )) are obvious for classifying traffic lines and others, simple or typical classification methods, e.g. heuristic rules, k-NN (nearest neighbor), neural networks and so on, can be applied. It is hard to find out the appropriate heuristic rules that can classify traffic lanes and other road marks accurately. Moreover, k-NN method needs a large number of training data set. Neural network method, however, can classify with a few training of simple features. In this paper, we used a very simple classification by heuristic rules and the neural network as shown in Fig. 6 , where the variables described τ are the thresholds for classification. In the neural network, we adopted an MLP (multiple layer perceptron). The network consists of an input layer, an output layer and one hidden layer. As shown in Fig. 7 , the number of nodes for input, hidden and hidden layers are 4, 8 and 2, respectively. The last input node is used for training the bias, and the number of hidden nodes is determined to be twice as large as the size of the input layer. For training we use the error back propagation algorithm.
PPT Slide
Lager Image
Classification rule for traffic line detection.
PPT Slide
Lager Image
The structure of the neural network.
After classification, we take candidate traffic lines Ci (0< i < m , where m is the number of the candidates) from detected lane marks, and then we calculate some parameters are defined in Table 1 for Ci .
Parameters for traffic lines.
PPT Slide
Lager Image
Parameters for traffic lines.
The direction of traffic lines ( d ) are classified into left, right and alternative ones by subregion intensity as shown in Fig. 8 (a), and the average intensity of subregions ( μ ( Ri )) are used as follows:
PPT Slide
Lager Image
where λ is the positive margin for alternative marks because the marks may be tilted in the opposite direction when the vehicle changes the lane or is driven in a curved lane. The other parameters are calculated using Bi by detecting two points ( p 1 and p 2 ) on upper and lower positions as shown in Fig. 8 (b). The slope of traffic lines ( m ) is the slope of p 1 and p 2 . The intersection points between traffic line and top and bottom horizontal lines ( pt and pb ) can be calculated using the equation of straight line set up by p 1 and p 2 .
PPT Slide
Lager Image
Subregion division (a) and calculation of line information (b).
3. Lane Detection and Tracking
Because a lane is marked with two traffic lines, we detect left and right traffic lines for lane detection. After detecting both side traffic lines, the left and right traffic lines of the tracked lane at the current frame are Lt and Rt , respectively.
If many left lane marks are detected, we reject traffic lines having low values of two similarities for lane width and slope from Ci such that d = l or d = a .
PPT Slide
Lager Image
PPT Slide
Lager Image
After this rejection, we track Ci having the minimum distance between
PPT Slide
Lager Image
and
PPT Slide
Lager Image
. We use the same algorithm to track right lane marks.
When only one side traffic line is tracked, we estimate the other side traffic line as shown in Fig. 9 . In our estimation method, we assume two constraints, 1) the vertical position of the vanishing point of the tracked lane is not changed, and 2) the internal angle of the vanishing point is not changed. When the only left side lane mark is tracked, the rotated angle of the left traffic line is calculated by
PPT Slide
Lager Image
So, the slope of the estimated right traffic line is
PPT Slide
Lager Image
and the horizontal position of the vanishing point of the tracked lane is calculated by
PPT Slide
Lager Image
The line information of
PPT Slide
Lager Image
and
PPT Slide
Lager Image
is can be updated by (10). The same method can be utilized to estimate untracked left traffic lines.
PPT Slide
Lager Image
Estimation of untracked traffic line.
When a vehicle is changing the lane with the left (or right) direction, a left (or right) traffic line is changed to a right (or left) line. At this time, the lane should not be tracked, so we determine the state of changing lane by two OR conditions as follows:
PPT Slide
Lager Image
PPT Slide
Lager Image
where c and s denote the center and side horizontal ranges, respectively. When the state of changing lane is detected, a traffic lane can be detected without tracking as the same detection with the initial state.
4. Experimental Results
The proposed method was implemented in Visual C++, and tested on a PC (Intel® Core™ i7-3770 CPU @ 3.40GHz 3.40GHz). The test image sequences were grabbed from one smart phone and four car black boxes with 320×240 resolution.
Detection results of lane marks are shown in Fig. 10 , where ROI (region of interest) is manually set by the camera angle and it is represented in magenta. Detected left marks are represented in red and the right marks are represented in blue. As shown in Fig. 10 , most of all lane marks are successfully detected, and other traffic marks are not detected.
PPT Slide
Lager Image
Experimental results of lane mark detection.
Lane detection results of some tested images are shown in Fig. 11 , where the high performance of the proposed method is revealed in various illumination and road conditions. In addition, curved lanes are verisimilarly detected because the lanes can be approximated by line segments in ROI areas.
PPT Slide
Lager Image
Experimental results of lane detection and tracking.
When the driving car is changing the lane, the tracked lane is successfully changed as shown in Fig. 12 . The following graph of Fig. 12 shows
PPT Slide
Lager Image
at consecutive frames, where the red lines indicate the state of changing lane.
PPT Slide
Lager Image
An example of changing lane.
The average processing time is 1.930 msec. It is fast enough for real-time operation and much faster than other methods. The processing time is inconceivable to HT-based methods. Quantitative comparison with other methods is not feasible since standard database images are not provided. The proposed method has, however, a remarkable simple computation in lane detection researches, and the method is capable of correctly detecting and tracking lanes in various operating conditions. In addition, HT-based methods may not detect short traffic lines as shown in Fig. 13 , and may detect non traffic lines such as crosswalk, road crack and shadow as shown in Figs. 14 and 15 . However, our method can classify crosswalk and do not detect road crack and shadow as traffic lines because we do not use edge information.
PPT Slide
Lager Image
A scene in which short traffic lines are observed.
PPT Slide
Lager Image
A scene in which many lines are observed.
PPT Slide
Lager Image
Scenes in which edges which are similar to traffic lines are observed.
5. Conclusion
In this paper, we proposed a novel lane detection method based on classification. Because the preprocessing and extracted features are very robust for classification, this method can be said to accurately detect traffic lanes. To estimate untracked traffic lines, the inner angle of two side traffic lines are used. In addition, the state of changing lane is determined in order to decide the driving lane. The processing is very simple and fast enough to be determined, the average time of processing is less than 2msec, so it is very suitable for real-time systems. Furthermore, most of currently used methods are concerned only daytime but our method can robustly detect traffic lanes even at nighttime by using appropriate preprocessing. Therefore, the proposed method can greatly contribute to intelligence transportation systems (ITSs) such as driver assistance, lane change assistance, lane departure warning and so forth. The proposed method can detect short dashed traffic lanes much accurately than HT-based methods but there are still some cases that cannot be detected short dashed traffic lines when the camera is viewing down excessively. For future works, we are going to develop an algorithm that can detect vehicles ahead in order to propose a smart car black-box system using the proposed methods.
BIO
Sungsoo Lim received the B.S. degree in Electronic Engineering and Biomedical Engineering from Kyung Hee University, in 2013. He is currently pursuing for his M.S. degree in Electronic Engineering at Kyung Hee University. His research interests include computer vision, image processing, ITS (intelligent transportation system), HCI (human computer interface), medical image processing, EIT (electrical impedance tomography) analysis and digital signal processing.
Daeho Lee received M.S. and Ph.D. degrees in Electronic Engineering from Kyung Hee University, Seoul, Korea, in 2001 and 2005, respectively. He has been an Associate Professor in the Humanitas College at Kyung Hee University, Korea, since 2005. His research interests include computer vision, pattern recognition, image processing, computer games, ITS (intelligent transportation system), HCI (human computer interaction), EIT (electrical impedance tomography) analysis and digital signal processing.
Youngtae Park received the B.S. degree in Electronic Engineering from Seoul National University, Seoul, Korea in 1979, the M.S. degree in Electrical Science from Korea Advanced Institute of Science and Technology, Seoul, Korea in 1981, and the Ph.D. degree in Electrical and Computer Engineering from the University of California, Irvine in 1990. He has been a Professor of Electronics Engineering at Kyung Hee University, Korea, since 1992. His research subjects include computer vision, pattern classification, image processing, intelligent transportation systems, and vision-based human identification.
References
Hillel A.B. , Lerner R. , Levi D. , Raz G. 2012 “Recent progress in road and lane detection: a survey,” Mach. Vis. Appl.    DOI : 10.1007/s00138-011-0404-2
Borkar A. , Hayer M.H. , Smith M.T. 2012 “A novel lane detection system with efficient ground truth generation,” IEEE Trans. Intell. Transp. Syst. 13 (1) 365 - 374    DOI : 10.1109/TITS.2011.2173196
Ghazali K. , Xial R. , Ma J. “Road lane detection using H-maxima and improved Hough transform,” in Proc. of Int. Conf. Computational Intelligence, Modeling and Simulation Kuantan, Malaysia Sep. 2012 204 - 208
Lee D. , Park Y. 2011 “Discrete Hough transform using line segment representation for line detection,” Opt. Eng. 087004 50 (8)
You F. , Zhang R. , Zhong L. , Wang H. , Xu J. 2013 “Lane detection algorithm for night-time digital image based on distribution feature of boundary pixels,” J. Opt. Soc. Korea 17 (2) 188 - 199    DOI : 10.3807/JOSK.2013.17.2.188
Furusho H. , Shirato R , Shimakage M. “A lane recognition method using the tangent vectors of white lane markers,” in Proc. of JSAE Annual Congress Yokohama, Japan 2003 9 - 13
McCall J.C. , Trivedi M.M. “An integrated, robust approach to lane marking detection and lane tracking,” Int. Intell. Vehicles Symp. Parma, Italy June 2004 533 - 537
Kreucher C. , Lakshmanan S. 1999 “LANA: a lane Extraction algorithm that uses frequency domain features,” IEEE Trans. Rob. Autom. 15 (2) 343 - 350    DOI : 10.1109/70.760356
Chen Q. , Wang H. “A real-time lane detection algorithm based on a hyperbola-pair model,” in Proc. of Intell. Vehicles Symp. Tokyo, Japan June 2006 510 - 515
Kim Z. 2008 “Robust land detection and tracking in challenging scenarios,” IEEE Trans. Intell. Transp. Syst. 9 (1) 16 - 26    DOI : 10.1109/TITS.2007.908582
Jeong P. , Nedevschi S. 2005 “Efficient and robust classification method using combined feature vector for lane detection,” IEEE Trans. Circ. Syst. Video Techn. 15 (4) 528 - 537    DOI : 10.1109/TCSVT.2005.844453
Zheng H. , Ye Q. , Jin Z. 2014 “A novel multiple kernel sparse representation based classification for face recognition,” KSII Trans. Internet & Inf. Syst. 8 (4) 1463 - 1480    DOI : 10.3837/tiis.2014.04.017
Yu J. , Tao D. , Wang M. 2012 “Adaptive hypergraph learning and its application image classification,” IEEE Trans. Imag. Proc. 21 (7) 3262 - 3272    DOI : 10.1109/TIP.2012.2190083
Park Y. , Lee D. 2010 “3D vision-based security monitoring for railroad stations,” J. Opt. Soc. Korea 14 (4) 451 - 457    DOI : 10.3807/JOSK.2010.14.4.451