Employing economical methods for pavement defects estimation

It is a common practise that measurements of road surface conditions are made using professional and expensive apparatus. Typically a van or a truck equipped with a set of professional sensors i.e. laser scanners of surface is used, therefore the measurement update period is often quite long. Two alternative low-cost methods for estimating road pavement defects and failures were proposed and investigated by the authors. The first one is based on accelerometers application and the other one employs image analysis acquired by cameras installed on a vehicle.


Introduction
Every year, governments of Polish cities spend between 4 and 8% of their budget on maintaining public roads, which turns into millions of PLN [1]. The main reason for this is the relatively rapid degradation of roadways. Important factors affecting damage are atmospheric conditions, ever-increasing traffic, erosion and others. Also the constant deficiency of funds causes this situation. Poor road quality causes damage to vehicles, increased fuel consumption and collisions.
In addition to the necessity to invest funds, it is also an important problem to choose those roads that should be renovated first and to assess the current status of individual sections. The General Directorate of National Roads and Motorways (GDDKiA) makes measurements of the roads it manages every year, using a professional, expensive equipment for that purpose. High costs of carrying out road condition tests reduce their frequency, thus extending the time needed to locate damages requiring repair. Meanwhile, the expansion and miniaturization of mobile electronic devices allows for a development of cost-effective solutions that can be applied for this purpose. Although the precision of lowcost surface scanners may be lower than of an expensive professional equipment, they can be more widely used by the general public and thus they may potentially increase the effectiveness of road damage detection.
In the Ordinance of the Polish Minister of Transport and Maritime Economy of 2 March 1999 on technical conditions to be met by new surfaces of public roads (Journal of Laws, No. 43, item 430), the requirements for the smoothness parameters that should characterise the new pavement of a public road were defined and the requirements for measuring equipment were also included. Profile measurements can be made using laser profiling or other devices that meet the requirements of the above-mentioned regulation.
In the Polish standard, there are four states of technical conditions of the pavement discerned. Class A -very good condition or class B -satisfactory condition are being the desired levels. Class C is a warning level applied to pavements with a significant damage requiring a repair. The critical level or class D requires immediate repairs. The pavements of class D roads usually mean numerous and extensive damages.
Professional measuring instruments usually employ laser distance sensors. These sensors can be used for the detection of asphalt discontinuities and for the testing the depth of individual defects. Using many of these sensors in a single line allows for representing the condition of the surface, effectively.
In the next sections, two proposed methods for estimating the surface condition and the results of their performance are presented, followed by a review of previously known solutions in this field. The first proposed method is based on the acceleration sensors application and on the analysis of signals obtained from inside the vehicle. The second method is based on the analysis of the image obtained from external cameras attached to the vehicle, and it assumes an application of neural networks to detect unevenness in the road image representation.

Solutions based on the vibrations analysis
The basic assumption for revealing unevenness with the accelerometer sensor is that the sensor detects a significant shock due to spotting irregular surfaces such as potholes, manholes, sewers and the similar ones. Accelerometer readings are compared to reference shocks when driving on a level surface. Based on the difference in sensor signals, the level of unevenness on the road is determined. The data from the GPS receiver allows for placing the estimated data in the geographical space. This solution was used, among others, in the Pothole Patrol system [2]. The taxi network cars were equipped with a set of terminals including sensors and Wi-Fi modules. Unevenness detection was performed locally, while results and positions were sent to the central server. In this way, data from a specific item could be verified (or rejected) by other vehicles within the system. A similar solution was used in CRSM [3]. In this case, 100 taxis with dedicated sensors were used. Vehicles collected information on the surface in Shenzhen (China) and by adjusting the appropriate algorithm they sent information about irregularities to the central server. As a result, 90% accuracy was achieved in the detection of continuity interruptions in roads with negligible level of false declarations. Among the available solutions, BusNet should also be mentioned [4]. It was decided to use public transport to monitor the surface condition by equipping buses with acceleration sensors. The system consisted of a sensory network based on the MICA platform. The collected data was eventually transmitted for the analysis to the central server. A different solution was proposed in 2011 in Latvia [5]. A session of 10 car rides on 4.4 km segment was carried out, during which the condition of the surface was analysed using 4 Android phones. Each of them, apart from dedicated software, contained an integrated accelerometer and a GPS receiver. Sensor auto-calibration was an important part of this solution, allowing the device to work in different vehicles. In the work of Strazdins et al. [6] they compared the results of measurements using GPS sensors and accelerometers from three different cell phones. Similar measurements were presented by Tai et al. [7], although the vehicle was a motorbike in that case.
The work by Tomiyam et al. [8] presents results from two accelerometers mounted on the outside of the vehicle. Although the results are unaffected by suspension damping (sensor in the unsprung mass area), experiments have been carried out with special sensors, which, in turn, have required additional professional and reliable installation in the vehicle suspension components.

Video analytics methods
One of the applied solutions is the detection of inequalities based on static images. In the study Koch and Brilakis proposed to divide the image into areas containing defects and those that do not contain them [9]. The potential segment with the cavity is subjected to approximation of the geometric shape. For the designated area, the texture is taken from the original image and then compared to the textures of the nearest areas without defects. If the comparison shows a significant difference between the images, the area is marked as a road loss. The algorithm implemented in the MATLAB environment with the support of the Image Processing Toolbox was tested using 120 photos of the road with damage and without. Some of them have been downloaded from the Google graphics search engine. The remaining ones were obtained from a remote controlled robot equipped with a webcam with a resolution of 640×480 pixels. 50 photos were used to train the algorithm, the remaining 70 to test it. The solution tests showed 82% precision result.
Another system for the detection of unevenness in the road surface is based on video data [10]. The solution utilizes the recording of video sequences through a dedicated electronic device installed in the car, and then the analysis of the generated frames to detect the road surface irregularities. The device was designed with a 1280×720, 60 fps CMOS camera, GPS receiver and signal processor. Image processing was realized through removing noise, increasing the contrast, binarization, and finally extracting the area representing the damage.
Another solution based on the video analysis was proposed by Jo and Ryu [11]. A system has been created which allows for detecting breakdowns of asphalt continuity with the use of an in-car camera recorder. The analysis process uses a lane detection algorithm. The method consisted of digitizing the image to detect lines defining the limits of the lane/roadway. Then, after the image denoising, a fragment of the image was determined in which unevenness was to be detected. The testing platform used was a 1920×1080 pixel camera and Cortex-A8 processor for the recording and processing of data. The tests were conducted on the basis of 20 videos recorded in sunny weather on a motorway. The sensitivity achieved was 71%, while the precision reached 88%. The drawback of that solution is the false detection with changing light intensity or lack of detection in case of flat unevenness presence.

Video subsystem
The system based on visual classification (RGB image and depth sensor image) with the use of the neural network was developed. The processing is made in two phases. In the first stage, the pre-processing of depth and RGB image is made in order to discard images that do not contain a distortion element of the road surface. A preliminary analysis of depth image includes: applying a box-type linear low pass filter with a window size of 19×19 pixels, using two Sobel operators with a west-eastern mask and an eastern-western mask both of size 5, image binarization using constant threshold of 30, applying the dilation with a 3x3 mask and double erosion with the same mask. A preliminary analysis of RGB images is then performed with the following steps: desaturate of the original image, the use of a box-type linear low pass filter with a window size of 11×11, the image binarization with Adaptive Gaussian method, the size of the block b = 25, applying the dilation with a 3x3 mask and double erosion with the same mask.
The result binary image was evaluated for the probability with which the damage information on it is likely to be present. This was done by counting the white pixels and comparing them with the experimentally obtained value.
For the classification of road surface damages represented on the images, the neural network was used. Based on a large collection of categorized data input, the process of network training was carried out. As a result, the network identifies the image characteristics into each of the categories. The network trained this way can be used to classify images from the outside of the input set.
In image recognition applications, the CNN (Convolutional Neural Network) networks are leading the forefront [12]. The system described in this section uses 18-layer CNN network called Resnet-18, developed by Microsoft [13]. The basic parameter for choosing this solution was the ratio of training time and prediction time to the average error rate.
The collection of images divided into categories was prepared in order to train the network.
In the process of developing the material, 10.000 photos of the roadway were collected, selected, and then based on the guidelines [14], divided manually into 5 categories: potholes -localised losses in bituminous layers, sewer manholes, mesh cracks and single cracks, filled-in and laid-over patches, road without any damage.
The data set was divided is such way, that first part contained a set of 90% of all photos (training data) and the remaining 10% of the set served as verification data.

Accelerometer subsystem
The effects of irregularities in the road surface are observed in the form of vibrations of the vehicle elements. The car suspension design aim is to minimize the transfer function of the path transmitting vibrations from the rough road to the vehicle's body, over the range of frequencies between 0.1 and 100 Hz [15]. Therefore, individual calibration of the sensor and vehicle is required, as the suspension systems may have different characteristics among car types.
The authors' approach is to use Gabor transform for the detection of the road surface anomalies. The Gabor transform is widely applied for time series analysis, particularly originating from vibrations [16,17]. The experiments focus on the comparison of results of the applied method among various devices, vehicle speed and type. The accelerometer data was acquired using smartphones, as they are commonly available, rich sensor-equipped devices available to everyone. Such mobile data gathering units are easy to use, thus interested car users could participate in the measurements. The first stage of data processing is sensor alignment correction described in authors' previous paper [18]. Then, the Gabor transform is applied on the accelerometer data z-axis representation, thus coefficients are obtained on the basis of the following formula:

Video material
The recording were made in the city streets. The frame rate of the Microsoft Kinect sensor allowed for driving at a speed of about 30 km/h. The total number of 10,000 images of road surface were recorded, and then annotated. Images were provided to previously trained neural network. Fig. 1 presents results of the image classification. A relatively low precision score were achieved for the "Potholes" category. Due to small set of gathered learning data for this category, the set was supplemented with photos from the internet sources. However, the average F1 score for all categories was 72.2%. Moreover, total average method specificity was 88.44% and accuracy as high as 89.19%.

Accelerometer signals
The experimental data presented in this paper were collected within multiple passes of 2 types of cars The data were collected on the roads which were classified into categories A and D. Roads R1, R2, R3 were of a poor quality (category D). Roads R5 and R6 were of a good quality, flat and straight (category A). Short characteristics of roads and length of investigated sections are as follows: R1 -city street, 1.1 km, R2 -city street, 0.7 km, R3suburban road, 1.2 km, R5 -expressway, 3.5 km, R6 -city street, 1 km. Fig. 2 presents the distribution of maximum of magnitude of Gabor coefficients calculated for 100-meters sections of good quality roads R5 and R6 for vehicle 1 and devices D1, D2, D3. Denoted variations of maximum |Gn,k| values depend on the device type while measurements were carried out on the same road employing the same vehicle.   3 presents the distribution of maximum magnitude of Gabor coefficients calculated for 100-meters sections of bad quality roads R1, R2, R3. The median of max |Gn,k| is significantly higher than in case of good surface quality roads. Moreover, for road R3, various controlled speeds (30, 40, 50 and 60 km/h) were compared, and no significant influence of the speed on the achieved results has been noticed.

Conclusions
The experiments were made using accelerometer signal and RGB plus depth camera image analysis in order to assess the road surface quality. In the video image analysis methods, a set of depth and RGB image processing algorithms were created. The collected input data was used for the neural network algorithm training and finally for the classification of defects. Research results have shown above 89% accuracy for the application of digital image processing to the classifying of road damages. In the further development stages, more cameras or sensors should be installed, which would improve the quality of the results. Moreover, the neural networks may be trained using a larger collection of learning sets.
The results of vibrations analysis obtained by means of the Gabor transform-based method show that the magnitude of coefficients is highly dependent on the road surface irregularities, returning larger values at any roughness and considerably lower ones at smooth road surfaces. However, in the non-controlled conditions, an additional factor is driver's reaction to the spotted cavity in the road surface. Nevertheless, the proposed method seems applicable to the monitoring of roads surface condition employing numerous, typical cars.
It is expected that an integration of the both presented methods may result in a more exact road surface damage evaluation, however in the real traffic conditions higher car speeds would require an application of high frame rate cameras.