Advanced
Automatic Extraction of Route Information from Road Sign Imagery
Automatic Extraction of Route Information from Road Sign Imagery
Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography. 2015. Dec, 33(6): 595-604
Copyright © 2015, Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • Received : December 12, 2015
  • Accepted : December 12, 2015
  • Published : December 31, 2015
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Junhee Youn
Member, Korea Institute of Civil Engineering and Building Technology, ICT Convergence and Integration Research Institute (E-mail:younj@kict.re.kr)
Kyusoo Chong
Corresponding Author, Member, Korea Institute of Civil Engineering and Building Technology, ICT Convergence and Integration Research Institute (E-mail:ksc@kict.re.kr)
Abstract
With the advances of the big-data process technology, acquiring the real-time information from the massive image data taken by a mobile device inside a vehicle will be possible in the near future. Among the information that can be found around the vehicle, the route information is needed for safe driving. In this study, the automatic extraction of route information from the road sign imagery was dealt with. The scope of the route information in this study included the route number, route type, and their relationship with the driving direction. For the recognition of the route number, the modified Tesseract OCR (Optical Character Recognition) engine was used after extracting the rectangular-road-sign area with the Freeman chain code tracing algorithm. The route types (expressway, highway, rural highway, and municipal road) are recognized using the proposed algorithms, which are acquired from colour space analysis. Those road signs provide information about the route number as well as the roads that may be encountered along the way. In this study, such information was called “OTW (on the way)” or “TTW (to the way)” which between the two should be indicated is determined using direction information. Finally, the route number is matched with the direction information. Experiments are carried out with the road sign imagery taken inside a car. As a result, route numbers, route number type, OTW or TTW are successfully recognized, however some errors occurred in the process of matching TTW number with the direction.
Keywords
1. Introduction
Those devices to gather the massive imagery from the roads have emerged lately; such as cameras designed to be installed in cars and mobile devices designed for drivers. With the rapid advances in the big-data technology, the real-time analysis of such massive imagery will be possible in the near future. This will be enable drivers or autonomous vehicles to immediately obtain information that are important and meaningful for driving. For that reason, the automatic extraction of transportation information from the massive imagery obtained from the roads has become a significant issue of late.
The Most of previous researches on transportation information extraction from massive road imagery focused on traffic signs and used colour space analysis, histogram analysis, the support vector machine, and other computer vision techniques. Benallal and Meunier (2003) developed a computer vision system for embedding in a car to enable traffic signs to be identified and located. For their study, they adopted the colour segmentation strategy based on the behaviour of the RGB components of several traffic signs from sunrise to sunset. The extraction of information from traffic signs through pattern matching and edge detection is proposed by Broggi (2007) . A support vector machine approach for classifying the pixels of traffic signs is presented by Bascon . (2008) . In the latter, normalized fast Fourier transform is used for determining the shape of the traffic sign. Lee and Yun (2013) developed a traffic sign recognition van in which a GPS, an IMU, a DMI, a camera, and a laser sensor are installed. Using sun van, they determined the positions of the traffic signs with the laser sensor and other positioning sensor, and recognized the attribute data of traffic signs with the support vector machine. A natural scene is difficult to analyse due to many unpredicted weather and illumination conditions. This principle is also applied to traffic signs. Zakir (2011) proposed a method of resolving such problems (e.g., colour illumination and occlusion, etc.). Lee et al. (2015) also tried to solve the illumination problem by proposing an illumination invariant traffic sign recognition method using the modified census transform and Fisher vector coding with the Gaussian mixture model. Chong (2014) proposed text area detection from road sign imagery, suggesting an IRBP (incremental right-to-left blob projection) method to overcome the limitations of template-matching-based text area detection.
The above approaches concentrate on the extraction of information from traffic sign imagery, and there have been relatively few researches on the automatic extraction of information from road sign imagery. The automatic extraction of road information from road sign imagery can be categorized into three types: text information extraction, direction information extraction, and route information extraction. For the researches related with text recognition in road signs, various computer vision techniques (e.g., image segmentation, edge detection, colour space analysis, pattern analysis) are sequentially applied ( Kim et al., 2013 ; Gonzalez , 2012 ; Huang ., 2012 ). With regard to direction information extraction, composing a skeleton model with the Hausdroff distance calculation approach ( Sastre ., 2005 ) and the arrow shape template matching approach ( Vavilin and Jo, 2009 ; Kim ., 2013 ) has been tried. Kim . (2013) extracted the arrow regions with their proposed line scan algorithm, and extracted the directions of the arrowheads through template image matching. Although the route information is one of the most important information around the vehicle for manual safety and autonomous driving, there have been a few studies related with the extraction of route number information. In manually driving, the driver will be embarrassed with wrong route number information, which could induce the accident. Also, the car checks the prescheduled routes with the automatic recognition of route information in autonomous driving. Soetedjo . (2010) developed an algorithm for extracting the symbols and text regions from road signs with a normalized RGB chromaticity diagram by detecting only the route number area, without recognizing the number itself. Vavilin and Jo (2009) extracted some route numbers, arrow regions, and texts with colour segmentation and geometry. Those route number pixels are extracted in this approach, but the pixel group could not be recognized as a number, and the approach can extract a route number that had been laid on the arrow region. From studying the previous researches, we could know that the researcher was able to develop a route number region extraction method to differentiate the route number from other previous developed techniques. There are challenges to be overcome; however, in recognizing the route number type and matching it with the direction information.
In the paper, an approach for extracting route number information from road sign imagery is introduced. This paper is organized as follows; the following section presents the proposed route number recognition algorithm. The closed-curve areas are extracted using the chain code tracing algorithm, and non-route number areas are excluded by using the regularized size of the route number in the road sign. The route type recognition algorithm is proposed after performing colour space analysis. The route number recognition with Tesseract is also introduced in this section. Secondly, an algorithm matching the extracted route number with the direction information will be discussed with defining terminologies of the “OTW (on the way)” and “TTW (to the way).” A recognition algorithm to determine, which of OTW and TTW, is proposed based on the arrowhead’s direction and position in the road sign. The matching of route numbers and directions will be dealt with in the section. Lastly, the experiment results, achievements, and limitations of the proposed algorithm will be discussed, followed by the study’s conclusions.
2. Recognition of Route Number
In South Korea, the size (height and width) and shape of the route number in the road signs should follow the specifications in Korea Road Act. In the act, there are four types of routes: expressway, highway, rural highway, and municipal road. The Fig. 1 shows examples of the routes described in Road Sign Regulation published by the Ministry of Land, Infrastructure, and Transport of the Republic of Korea. In the Fig. 1 , “H” represents the height of the Korean word in the road sign. As can be seen in the Fig. 1 , all the routes numbers are in a closed-curve area. Therefore, the closed-curve areas are first extracted for the recognition of the route number.
PPT Slide
Lager Image
(a) Expressway; (b) highway; (c) rural highway; and (d) municipal road in the Road Sign Regulation (source: MOLIT (2014))
All closed-curves are extracted using the Freeman chain code tracing algorithm. The Chain codes are used to present a boundary consisting of connected straight-line segments with a specified length and direction ( Gonzalez and Wood, 1992 ). The original colour image is converted into a binary image (black and white). From searching the whole image, a point on the boundary is selected, and its coordinates are transmitted. The encoder then moves along the boundary of the region, so as transmits a symbol representing the direction of this movement at each step. If the encoder returns to the starting position, then the boundary is defined as the closed-curve. After extracting the closed-curve areas, the rectangular closed-curve areas (or the so-called “smallest enclosing boxes”) are determined. The Fig. 2 shows the procedures; firstly, the longest pair of points is determined for each point. Each pair composed of one line and the lines are projected into a line and a sample axis. The longest projected lines in the line and sample axes are the height and width of rectangular closed-curve area. The Fig. 3 shows the results of the rectangular closed-curve area extraction overlaid on the original image. The red rectangles in the Fig. 3 denote the extracted closed-curve areas. As seen in the Fig. 3 , not only the route number areas can be extracted but also boundaries of the image and letters. The extracted letters are the Korean letter “ㅇ” and the English letters “o,” “g,” “a,” “e,” and “d.”
PPT Slide
Lager Image
Process of determining the rectangular closed-curve area
PPT Slide
Lager Image
Extracted all rectangular closed-curve area
Among the extracted closed-curve areas, the non-route number areas are excluded by using the size regulation defined in the “Road Sign Regulation.” In this regulation, the sizes of the road signs and the height of the Korean word in the road signs are regulated by road sign type (e.g., three-direction notice sign in a two-lane road, three-direction sign in a two-lane road, three-direction notice sign in a more-than-two-lane road, etc.) For an instance, the width and height of a three-direction sign in a two-lane road should be 445 and 220 cm. In the same two-lane road, the height of the Korean word should be 30 cm. Based on the ratio of the width and height of the road sign image, and the Korean word’s “height” (“H”), road signs can be classified. Then the Korean word’s height in the image coordinate ( Himg ) can be determined in proportion to the image size of road signs. As mentioned earlier, the size of the route number area is regulated with “H,” as shown in the Fig. 1 . Therefore, among all the rectangular closed-curve areas ( RAclose ), the route number area RArout is calculated as follows:
PPT Slide
Lager Image
where i = 1, 2, ⋯⋯N (N is the maximum number of extracted rectangular closed-curve areas) and, RAhi is the height of RAi , and RAwi is the width of RAi ). The Fig. 4 presents the determined rectangular closed-curve areas for the route number. As can be seen in the Fig. 4 , the route number areas containing 30, 32, and 616 were extracted.
PPT Slide
Lager Image
Extracted rectangular closed-curve areas for the route number
Based on the colour differences, the route numbers can be classified. As discussed earlier, there are four types of routes: expressway, highway, rural highway, and municipal road. The route number is enclosed in the symbol, which denotes the type of route. The salient difference between the different symbols is the local colour distribution. In the case of the expressway, the upper part of the symbol coloured the red. The most of highway and rural highway symbols are filled with the blue and orange colours, respectively. Also, the saturation value for municipal road symbols is higher than others because a big portion of the symbols is coloured white. With the previously proposed algorithm, the route number area for an expressway contains only a number, as can be seen in the Fig. 4 . In case of the expressway’s red colour characteristic, if there is a red pixel at the upper part of rectangle, the rectangular closed-curve area should be expanded for it. To differentiate the red (expressway), blue (highway), and orange (municipal road) symbols, the hue image was analysed. To extract the white colour distribution ratio, the saturation image was analysed. With these heuristic analyses, the route-number-type classification process presented in the Fig. 5 is proposed. In the Fig 5 , R , O , W , and B are defined as follows;
PPT Slide
Lager Image
PPT Slide
Lager Image
Flowchart for the route number type classification method
in which NR hs is a cardinality of pixel set R hs , which is
PPT Slide
Lager Image
where j = 1, 2, ⋯⋯M (M is the total number of pixels in the rectangular route number area), and s(R hsj ) is the saturation value for the pixel, and H ( Rhsj ) is the hue value for the pixel, while N sm20 is a cardinality of the pixel set whose saturation value is more than 20 in the rectangular route number area.
PPT Slide
Lager Image
in which NO hs is a cardinality of pixel set O hs , which is
PPT Slide
Lager Image
PPT Slide
Lager Image
PPT Slide
Lager Image
in which NB hs is a cardinality of pixel set B hs , which is
PPT Slide
Lager Image
Applying equations (2)-(8) and the process presented in the Fig. 5 , the rectangular route number areas containing the numbers 30, 32, and 616 are classified as an expressway, a highway, and a rural highway, respectively.
After extracting the rectangular route number areas, the numbers are recognized with Tesseract, which is an open-source OCR (optical character recognition) engine. A Tesseract is a free software program released under Apache License ver. 2.0, and its development has been sponsored by Google since 2006 ( Wikipedia, 2015 ). The Tesseract engine was originally developed as a proprietary software at Hewlett Packard Labs, and was released as an open-source engine in 2005. Although Tesseract is considered one of the most accurate open-source OCR engines at present ( Willis, 2006 ), many misrecognized numbers could be found when if its provided dictionary is directly applied. For instances, the number “1” could be recognized as the English small letter “l,” and the number “101” could be recognized as the number “1” or the Korean word “이.” To avoid such misrecognitions, the route number’s font in a road sign was constructed as a learning data, which Tesseract engine could modify signs based on. After the application of the modified Tesseract engine, the route numbers 30, 32, and 616 are already correctly recognized, as shown in the Fig. 4 .
3. Matching Extracted Route Number with Direction Information
In Chapter 2, a method of recognizing route numbers and their type (i.e., expressway, highway, rural highway, or municipal road) was proposed. The present chapter deals with matching of the route number and type with direction information. Before the discussion of method, there are terminologies to be defined; OTW (On the Way) and TTW (To the way). The OTW means “the driver is on the way,” as the TTW means “drivers will encounter the way.” If Expressway 7 is located towards the east (as indicated by the east direction arrow), it means “If you go to the eastern direction, you will find yourself on Expressway 7”; with the terminologies, this line can be simply expressed as “North OTW Expressway 7.” If the Highway 4 is located near the northern direction arrowhead and is not located on the arrow itself, it means “If you go to the northern direction, you will encounter the Highway 4”; which could be simply expressed as “Straight TTW Highway 4.” In the Fig. 4 , three kinds of information can be extracted: “Straight TTW Expressway 30,” “Straight OTW Highway 32,” and “Right TTW Rural Highway.” These terminologies will be used in this paper for matching the route number and the direction information.
The direction information is recognized with the algorithm proposed by Kim . (2013) . The seed area for the arrow is firstly extracted using the line scan method in Kim . (2013) , prior to extract the arrow area with the region growing method. After detecting the feature points using the good-features tracking algorithm, the template-matching algorithm is applied to extract the arrowhead. Around the feature points, five kinds of arrowhead template expressing the east, west, north, northeast, and northwest directions are matched in the order. Finally, the direction information can be extracted by selecting the template that produces a high correlation coefficient value.
OTW or TTW route numbers are differently located depending on the type of road sign. In MOLIT (2014) , there are three types of road signs containing route numbers. The Fig. 6 presents these; the first one is DNS (direction notice sign) in the Fig. 6(a) that contains 2/3 DNSs and 2/3 direction in MOLIT (2014) ; the second one is DS (direction sign), which contains 2/3 DSs, as shown in Fig. 6(b) , while the third one refers SDNS (simple direction notice sign), which contains 2/3 DNSs. These three road sign types can be differentiated from one another based on the ratio of the width and height of the road sign imagery following to the sizes of road signs in Road Sign Regulation. As the route numbers are differently located depending on the type of road sign, a different recognition method needs to be applied for each type.
PPT Slide
Lager Image
Three types of road signs containing the route number: (a) direction notice sign (DNS); (b) direction sign (DS); and (c) simple direction notice sign (SDNS)
OTW or TTW for route numbers are recognized after composing closed polygon for the case of DNS. The Fig. 7 shows the concept of composing closed polygon for DNS. The closed polygon is comprised of the end of extracted arrowheads and the base point, which is generated by perpendicular projecting the end of highest arrowhead onto lower end of the image. A, B, and C denotes the end of extracted arrowheads, and D is a base point in the Fig. 7 . After composing the closed polygon, the position of extracted route numbers is examined. If the extracted route number is inside the closed polygon, then it is classified as OTW. The other route number, located outside of the closed polygon, is classified as TTW, while the direction of each route numbers is acquired from its nearest arrowhead. With these method, following information is acquired in the Fig. 7 : “Left OTW highway 37 and 47”, “Right OTW highway 47”, “Straight OTW highway 37”, “Left TTW highway 25”, “Right TTW highway 27”, and “Straight TTW highway 26”.
PPT Slide
Lager Image
Concept of composing a close polygon in the case of DNS
The recognition of OTW or TTW for route numbers starts from the image segmentation in the case of DS and SDNS. As stated earlier, DS and SDNS have two or three direction signs. The image is equally divided perpendicularly into two parts (two-direction road sign) or three parts (three-direction road sign) in the case of DS. Likewise, the image is equally divided horizontally into two or three parts in the case of SDNS. Then, the existence of the route number from the end of arrowhead to the opposite direction of arrowhead is searched. If the route numbers are detected during searching, the route numbers are classified as OTW. Otherwise, the route numbers are classified as TTW. The searching is processed within each segmented image. The Fig. 8 presents the concept of image segmentation and determination in either OTW, or TTW. In the Fig. 8 , the image is divided into three equal parts since it is a three-direction sign. In left part of Fig. 8, route number 513 and 21 are detected during the search. Therefore, the following information is acquired with the suggested method: “Left TTW Highway 21”, “Left OTW Rural Highway 513”, “Straight OTW Highway 21”, “Straight TTW Expressway 35”, and “Right TTW Highway 17” in the case of Fig. 8(a) . “Straight OTW Highway 32” in the case of Fig. 8(b) .
PPT Slide
Lager Image
Concept of image segmentation and OTW or TTW determination in the case of DS (a) and SDNS (b)
4. Experiment and Results
The proposed algorithm is applied to road sign imagery. The used imagery is acquired by MMS (Mobile Mapping System) equipped with Ladybug5 sensor, which is generated by PointGrey corporation, and laser sensor LMS 151, which is generated by SICK corporation. During the acquisition of image, the velocity of MMS is about 60 km/h. The imagery is orthorectified by using point cloud from laser sensor to manually crop the road sign section. The processing time for one image is less than one second in the Intel(R) Core™ i5-2400 CPU @3.10 GHz computer. Among the acquired imagery, the partially occluded imagery by branches or wire is excluded for the experiment.
The Fig. 9 presents imageries that were used in the experiment, and the results lists in the Table 1 . In the Fig. 9 , each image presents each case; (a)-(d) of DNS cases, (e)-(f) of DS cases, and (g)-(h) of SDNS cases. As seen in Table 1 and Fig. 9 , all the route numbers and their types are correctly recognized. Also, The OTW or TTW recognitions results successfully. Some errors occurred, however, during the matching of the route number with the directions. By comparing with the extracted results ( Table 1 ) and the experiment imageries ( Fig. 9 ), those of TTW Highway 34, TTW Highway 1, and TTW Expressway 30 have supposed to be matched with the ‘Straight’, but they turned out to be the ‘Left’ in the extracted results. The reason for error occurrence is that the route number’s direction is determined based on the distance between numbers and arrowhead. For an instance, the distance between the route number 34 and the the same route number and the northern direction arrowhead. Compared with other cases, it can be concluded that such problems occur when the place’s name is relatively long. Therefore, if the route number is matched with the place’s name after matching the place’s name with the arrowhead, such errors will not occur.
PPT Slide
Lager Image
Imagery used for the experiment in this study: (a)-(d) are DNS cases; (e)-(f) were DS cases; and (g)-(h) are SDNS cases
Experiment results for the items listed inFig. 9
PPT Slide
Lager Image
Experiment results for the items listed in Fig. 9
5. Conclusions
This study dealt with the extraction of route information (number and type, OTW or TTW, and direction) from road sign imageries. For the recognition of route number, the rectangular closed-curve areas for route numbers were extracted using the Freeman chain code tracing algorithm, while the size of the route number was regulated by Road Sign Regulation. The route number was extracted using the modified Tesseract software with a cropped route number area image. The route number was classified using the proposed process and equations. The equations were generated through colour space analysis with the hue and saturation value percentage for the route number area. The arrowheads were used for determining that the route number was either OTW (On the Way) or TTW (To the way). A different determination method was proposed for each route type. The extracted route numbers were matched with the direction information, based on the arrowhead direction and the distance between the route number and the arrowhead. With the experiment results, we could conclude in that the route number, route number type, OTW, or TTW were successfully recognized, but some errors occurred in matching the TTW route number with the direction information. The future research will be aimed at improving the algorithm for matching the route number with the direction information by using the sequential matching method, which matches place’s name and route number after matching the arrowhead and place’s name.
Acknowledgements
This research was supported by a grant from a Strategic Research Project (Development of driving environment prediction platform based on big data) funded by the Korea Institute of Civil Engineering and Building Technology.
References
Bascon S.M. , Arroyo S.L. , Siegmann P.H. , Moreno G. , Rodriguez F.J.A. (2008) Traffic sign recognition system for inventory purposes IEEE IEEE Intelligent Vehicles Symposium Eindhoven University of Technology Eindhoven Eindhoven, Netherlands 4-6 June 590 - 595
Benallal M. , Meunier J. (2003) Real-time color segmentation of road signs IEEE IEEE CCECE 2003 Canadian Conference on Electrical and Computer Engineering 4-7 May 3 1823 - 1826
Broggi A. , Cerri P. , Medici P. , Porta P. , Ghisio G. (2007) Real time road signs recognition IEEE Proceedings of the 2007 IEEE Intelligent Vehicles Symposium Istanbul, Turkey 13- 15 June 981 - 986
Chong K. (2014) Text area detection of road sign images based on IRBP method The Journal of the Korea Institute of Intelligent Transport Systems (in Korean with English abstract) 13 (6) 1 - 9
Gonzalez A. , Bergasa L.M. , Yebes J. , Almazan J. (2012) Text recognition on traffic panels from street-level imagery 2012 Intelligent Vehicles Symposium Alcala de Henares, Spain 3-7 June 340 - 345
Gonzalez R.C. , Wood R.E. (1992) Digital Image Processing Addison-Wesley Publishing Company 484 - 485
Huang X. , Liu K. , Zhu L. (2012) Auto scene text detection based on edge and color features 2012 International Conference on Systems and Informatics Yantai, China 19-20 May 1882 - 1886
Kim G. , Chong K. , Youn J. (2013) Automatic recognition of direction information in road sign image using OpenCV Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography (in Korean with English abstract) 31 (4) 293 - 300
Lee T. , Lim K. , Bae G. , Byun H. , Choi Y. (2015) An illumination invariant traffic sign recognition in the driving environment for intelligent vehicles Journal of KIISE (in Korean with English abstract) 42 (2) 203 - 212
Lee J.S. , Yun D.G. (2013) The road traffic sign recognition and automatic positioning for road facility management International Journal Highway Engineering 15 (1) 155 - 161
MOLIT (2014) Road sign regulation Ministry of Land, Infrastructure and Transport
Sastre R.J.L. , Arroyo S.L. , Siegmann P. , Jimenez P.G. , Reina A.V. (2005) Recognition of mandatory traffic signs using the hausdorff distance Proceedings of the 5th WSEAS International Conference on Signal Processing, Computational Geometry & Artifical Vision Malta 15-17 September 216 - 221
Soetedjo A. , Yamada K. , Limpraptono F.Y. (2010) Segmentation of road guidance sign symbols and characters based on normalized RGB chromaticity diagram International Journal of Computer Applications 3 (3) 10 - 15
Vavilin A. , Jo K.H. (2009) Graph-based approach for robust road guidance sign recognition from differently exposed images Journal of Universal Computer Science 15 (4) 786 - 804
Wikipedia (2015) Tesseract https://en.wikipedia.org/wiki/Tesseract_(software)
Willis N. (2006) Google’s Tesseract OCR engine is a quantum leap forward http://archive09.linux.com/articles/57222
Zakir U. (2011) Automatic Road Sign Detection and Recognition, Ph.D. dissertation Loughborough University United Kingdom