Automatic Traffic Sign Detection and Recognition Using Colour Segmentation and Shape Identification

. The paper describes a colour-based segmentation method of European traffic signs for detection in an image and a feature-based recognition method for categorizing them into given classes. At first, we have performed analysis of several well-known colour spaces as the RGB, HSV and YCbCr often used for segmentation purposes. The HSV colour space has been chosen as the most convenient for segmentation step and colour-based models of traffic signs representatives were created. Next, the fast radial symmetry (FRS) detection method and the Harris corner detector were used to recognize circles, triangles and squares as main geometrical shapes of the traffic signs. For these purposes a new gallery of real-life images containing traffic signs has been created and analysed. Overall efficiency of our recognition method is approx. 93 % on our gallery and is usable for real-time implementations.


Introduction
Machine vision in a traffic is common technology at these days.A plenty of various system for traffic signs recognition were developed and some of them has ambition to be a part of superior car system of online navigation and for autonomous driving [1,2].Such systems are generally referenced as the driving assistance system.Some automotive manufacturers like Daimler-Benz, Opel or BMW already uses their own technologies for assisting driver or preceding accidents [3].Despite a lots of designed and implemented systems there is no one generally efficient all over the world.Relatively simple system introduced in this paper takes into account a region of Europe and its scheme of traffic signs.

Traffic signs differences around the world
In different countries are very different opinions on how to give instructions or provide information about traffic conditions to drivers and other road users.The basic concept of the traffic signs used in the European Union is based on a usage of simple graphical pictogram inside a plain geometrical shape of one principal colour.Similar idea determines traffic signs in the United States, nevertheless more additional text is used on a plenty of signs and thus subconscious perception of the traffic signs is slightly more complicated and is not as straightforward as in the EU.On the contrary, almost arbitrary traffic meaning (intent) can be visualised by this more flexible way established in the US.As an example of such typical difference the informative traffic sign "One-way street" in the last row of the Table 1 can be pointed out.
For quick and clear overview of differences in the EU and US region, the Table 1 shows two example signs in each row related to one of the five basic traffic sign classes: warning, regulatory, obligatory, priority and informative.One-way street Dead end (no through road) The overall difference of traffic signs within the EU are conceptually reduced by a multilateral treaty "Vienna Convention on Road Signs and Signals" from 1968.Such treaties effectively limit differences in traffic signs and thus facilitate deploying of autonomous recognition systems despite of high number of all traffic signs defined.For example more than 200 traffic signs, each assigned to one of the five main classes, are defined in traffic regulations of the Czech Republic in the EU.Algorithms and methods designed and described in this paper take into account the only EU traffic signs concept.

Traffic sign recognition approaches
There are two principally different approaches for traffic sing detection and recognition task in a theory of the computer vision.First one, the template-based approach computes similarity between unknown signal (part of an input image with some object) and a set of given templates of traffic signs.A template of traffic sign of the highest similarity with queried sub-image is then chosen as the correct one.
The other approach, a so-called feature-based, computes a set of features on the input image with unclassified objects and classifying mechanism searches for similar structure and values of these features in an internal database.Corners, radial symmetry, SIFT, SURF, BRISK, MSER and plenty of other descriptors can be used for detection and description of the so-called points of interest.

Our design
Briefly, we have selected the feature-based approach in this paper.At first, the input image is transformed from a native camera picture format to the HSV 8-bit colour coding.Further, a colour-based segmentation is used for a detection of red, blue and yellow objects in the input image.These objects are then filtered by means of given geometrical and colour conditions valid for traffic signs e.g.only circles and rectangles are interesting in blue colour etc.A shape identification and recognition step is carried out as last step before final sign classification.Well-known features as Harris corners and Fast Radial Symmetry (FRS) are used at this stage.Next chapters introduce mentioned approach in more details.

Colour-based segmentation
A colour information in an image is one of several fundamental features of each object together with e.g.edges, corners, brightness level etc.Because of exact definition of appearance of traffic signs in various national regulations of each well urbanized state, it is convenient to use entire or only part of these definitions.Traffic signs generally have well defined geometrical shape and very limited number of permissible shapes (often only circle, triangle, rectangle and octagon).The other well defined feature of traffic sign is its colour.Only the one specific red, blue and yellow colours are used for definition of the most crucial signs.
Plenty of various colour spaces and coding schemes are introduced in a theory and only some of them are used in practical implementations [4].We have analysed almost 350 own images of traffic scene with at least one traffic sign present on each image.The analysis has been performed in the RGB, HSV and YCbCr colour schemes.Results of the RGB and HSV colour-based segmentation are brought out in the following subsections.

Segmentation in RGB colour space
The RGB (Red-Green-Blue) colour space is the most frequent output format of almost all image acquisition devices as webcams, industrial cameras, handy-cams, etc.This general popularity of the RGB is given by a construction of CCD and CMOS sensors equipped with the well-known Bayer colour mask R-G-G-B.Nevertheless, this format is not usually convenient for image processing and more generally for computer vision tasks because of lack of an explicit brightness channel and thus its implicit presence directly in all the R, G and B channels values.
As can be seen in the Fig. 1 on the left, the RGB colour space is in form of cube.Colour-based segmentation in such space is simply given by limits of min and max values along all three axis related to R, G and B channel respectively.These limits are given by a model of an object looked for.The limits can form either a simple orthogonal block in the RGB colour space or some more sophisticated shape as a sphere, ellipsoid, octahedron etc. 5Each pixel in the input image of colour given by a (R, G, B) triple lying inside the mentioned shape is classified as an element of an objects and vice versa.A binary map of pixels memberships' to particular colour is a result of such colour-based segmentation.
We have used a so-called comparative method for our experiments described later.This method applies limits mentioned above to a ratio of two particular colours R, G and B as it is introduced by the equation (1) for the red component.Binary maps for the other two colours G and B are given by analogical equations.All three maps can be finally visualized in only one combined RGB image.

Segmentation in HSV colour space
The HSV space (Hue-Saturation-Value) is closer to intuitive perception of colours by human eye and is cone shaped.As can be seen in the previous Fig. 1 on the right, the Hue component is angle from 0° to 360° and represent a basic shade of colour.The other two components are numbers from 0 to 1 and represent saturation (purity) of colour and value (brightness) respectively.
Segmentation in the HSV colour space is very similar to the previous one with only one difference that shape determining an area of object colours is distorted in accordance with a shape of the HSV space.In the Fig. 2, the three clusters of pixels related to red, blue and yellow parts of the traffic signs can be seen.Geometrical representatives (hulls of clusters) serve as segmentation rule at the same way as in the case of the RGB colour space segmentation.

Shape recognition
One or more regions of interest (ROIs) are an output of the previous colour-based segmentation step.Now classification of its shape has to be carried out.The task of traffic sign recognition is very limited by itself: only circles, triangles and rectangles are considered as permissible classes of shapes.
A so-called fast radial symmetry detection method (FRS) introduced in [5] has been employed for circles recognition in [6] and was employed also in our experiments.This algorithm outputs a grayscale map with a peak at that position, where any circle is present in the input image.The input ROI and a response of the FRS algorithm are shown in the Fig. 3.More circle-shaped the input object is, higher response the FRS method gives at its output.For our case of traffic signs, two examples of responses of the FRS detection method can be seen in the Fig. 4. Note that almost ideal circle-shaped sign No entry (on the left) results in a clear and high peak at output image.On the contrary, the triangle-shaped sign Warning does not create any significant peak and the response is blurred.Recognition of valid circle-shape object in the input image is then done based on the peak level and its compactness.
The other two basic shapes (triangle and rectangle) are distinguished by means of the Harris corner detector [7].We use the knowledge from the previous steps that object of traffic sign (or other similar object) is normalized in the ROI and is straight in it.Under these conditions, the Harris corner detector gives a strong response at the four principal corners of the ROI in case of rectangular object in the input image and the three corners in case of triangular object.Based on mutual position of these corners the proper class of input object can be specified [6].

Experimental results
We have carried out a series of experiments in both laboratory and real life.Whole procedure is described in following chapters in detail.

Image gallery
First of all an image gallery was created on our own.We have acquired many hundred images and almost 350 of them were selected as suitable for next processing during our experiments.An example of two gallery images is depicted in the following Fig. 5.The gallery has been employed for creating of model representatives of the red, blue and yellow signs classes at first stage.Hundreds of pixels were semi-manually picked out to create a robust and reliable colour models (red, blue and yellow).See the Table 2 containing numbers of images and related numbers of pixels selected for individual colour classes of traffic signs.

Colour segmentation
In accordance with the theory described in the previous chapters, a colour-based segmentation in the RGB, HSV and additionally for comparison in the YCbCr colour spaces have been performed on the whole gallery during the tests.Even though the HSV segmentation needs a transformation step RGBoHSV, its results are more suitable for traffic sign's shape recognition.The most important is a lower over-segmentation factor and a higher compactness of segmented objects in comparison with the RGB model.It is caused by pure colour specification in the HSV colour model with strong ability to handle over-exposed and under-exposed images.In the previous two figures you can see results of segmentation step in the RGB and HSV colour space applied on the two images from the Fig. 5.All the three binary maps of the red, blue and yellow colours are showed in one image together by the red, blue and green colour respectively.Overall efficiency of segmentation algorithm is recorded in the following Table 3 in association with an average time required by the algorithm for processing of each one image from the gallery.A long duration of segmentation in the HSV colour-space has been anticipated due to RGBoHSV.Images of reduced dimensions 640 by 480 pixels have been taken into account during this measuring.Efficiency was defined as successful/unsuccessful segmentation of complete area of traffic sign without any connection with its neighbourhood.The shape recognition algorithm can be efficiently applied on such distinctive regions in the next step of image processing.

Region normalization
Traffic signs in the both input images in the Fig. 5 are almost ideally perpendicular to an optical axis of a camera used to capture images.However, absolute majority of traffic signs captured do not fulfil this special case and it is the reason why a normalization of object of presumable traffic sign is performed.
An example of the normalization procedure is shown in the Fig. 8.An original image of the traffic sign No left turn is depicted on the left in grayscale.Its binary map containing corresponding ellipse with highlighted major and minor axis is shown on the right.Each ellipse-shaped object always corresponds to some circular traffic signs (or less probably some other circle object).The ellipse is then transformed to a circleshaped object with minor and major axis of the same length and with the major axis to be exactly vertical.Result of this transformation is depicted in the Fig. 9 on the left.In the middle and on the right pictures are illustrated rotational correction of the transformation and a trimmed image as final output of segmentation and normalization steps.Reliable region normalization is crucial stage for later recognition of type of a traffic sign.An example set of normalized traffic sign objects resampled to dimensions of 100 by 100 pixels is introduced in the Fig. 10.

Real-scene recognition experiments
By implementation the HSV colour-based segmentation algorithm, the shape recognition algorithm and a pictogram inside a traffic sign detector, we obtain interesting results.At first, responses of the FSR method to the two input images introduced in the Fig. 5 can be seen in the following Fig. 11.It is quite clear that a recognition of circular traffic signs is performed by evaluation of height of peaks in these maps.The presence of triangular or rectangular traffic sign was determined from a distribution of the strongest corners within a region of an object.
In the next Fig. 12 two semi-synthetic testing images are depicted in association with imprinted position and types of detected traffic signs.Finally, we have applied our design directly in urban traffic as the driver assistance system aforementioned in the introductory chapter.Several selected pictures of this system are introduced in the previous Fig. 13.Such images serve as system's output for a driver on the LED display inside a car.

Conclusions and future work
During our experiments we have implemented a combination of the colour-based and shape-based methods to achieve better results in recognition of traffic signs.Results obtained were really highly promising, nevertheless a deep verification both on a broad gallery and in the traffic should be carried out.
Future work lies in an improvement of basic colour segmentation in cases, when a traffic sign is connected with some other near object due to its similar colour.Next, a better algorithm for pictogram recognition has to be developed instead of correlation technique used here.

Figure 1 .
Figure 1.The RGB and the HSV colour spaces.

Figure 2 .
Figure 2. Sets of red, blue and yellow pixel representatives in images with traffic signs.

Figure 4 .
Figure 4. Difference of results of the fast radial symmetry algorithm on circular and triangular traffic sign.

Figure 5 .
Figure 5. Weak and good contrast input images with signs.

Figure 6 .
Figure 6.Results of segmentation in the RGB colour space.

Figure 7 .
Figure 7. Results of segmentation in the HSV colour space.

Figure 8 .
Figure 8. Traffic sign on the projective transformation in the input image and its binary map.

Figure 9 .
Figure 9.Fixed projection image, corrected rotation and trimmed image of the traffic sign.

Figure 10 .
Figure 10.Normalized traffic sign objects as input for recognition algorithm.

Figure 11 .
Figure 11.FRS responses to the input images from Fig. 5.

Figure 12 .
Figure 12.Verification of traffic sign recognition on semisynthetic images.

Figure 13 .
Figure 13.Traffic sign recognition system results.

Table 1 .
Different signs in EU and US region.
InformativeAn exact meaning of the traffic signs in the Table1is following:

Table 2 .
Image gallery for basic classes of traffic signs.

Table 3 .
Comparison of colour spaces for segmentation.