Automated Road Damage Recognition based on the Sparse Coding Analysis of Vehicle Vibrations

Road pavement damage inspection is a critical yet challenging task. At present, road pavement damage inspection is usually done by DOTs using a manual process. Another emerging method of inspection is via the use of sensors, such as the use of LiDAR. This study proposes an automated road damage recognition method via the Sparse Coding analysis of vehicle vibrations. Sparse Coding is a class of unsupervised methods that learn data patterns based on extracted overcomplete bases. Unlike frequency domain-based analysis, e.g. Spectral Analysis, Sparse Coding analysis preserves the temporal information of the vehicle vibration that contains important patterns related to road pavement damage. A preliminary study was performed with vehicle vibration data collected in College Station, Texas. Results confirm the feasibility of the proposed method in automated road pavement damage recognition. More data points should be collected in the future to further benchmark the effectiveness of the proposed method.


Introduction
The most cost-effective strategy to improve the overall condition of America's road infrastructure is through the Preventive Maintenance, i.e., a planned treatment to an existing roadway system and its appurtenances before deficiencies develop [1]. Empirical and laboratory data indicates that the development of pavement distress follows a predictable course [2]. Based on the established knowledge of pavement distress development, preventive maintenance aims to preserve the system, retard deterioration, and maintain the functional condition of the system without significantly increasing the structural capacity ( Figure 1) [3]. However, making robust preventive maintenance decisions on a relatively large section of road can be a nontrivial task, due to various factors that need to be considered in the calculation. Even in a small area, different sections of a road can be in varying and changing conditions. Decision makers must be able to identify the critical sections, predict the temporal deterioration of each and every section of the roads, and ultimately distribute limited resources in a holistic way to optimize the long-term performance of the entire system, instead of local sections. It requires large amounts of high-quality live data about the pavement surface conditions. Specifically, this study aims to address the technical difficulty of collecting high fidelity data of pavement surface condition by leveraging existing technologies and hardware. To overcome this challenge, this paper proposes an automated road damage recognition method based on the Sparse Coding of moving vehicles' vibration data. The reminder of this paper will introduce the technical background and a preliminary study.

Literature review
At present, road surface inspection in region 6 is usually done with a manual process [4]. DOTs have published standard procedures for manual inspection [5], such as Texas Pavement Management Information System Rater's Manual [6]. The manual approach brings two potential problems: variation in inspection results due to inspectors' personal bias, and the difficulty of high, frequent inspection and coverage area. To overcome the limitations of manual inspection, a variety of automated road surface inspection methods have been proposed [7,8]. Representative technologies include vision-based method [4,9,10,11,12], laser scanning [13], ground penetration radar (GPR) [4,14], natural lighting method [15] and a combination of multiple sensors [16]. For example, Huang and Xu [10] developed a road surface inspection method based on the grid cell analysis of the collected grey images of road surface. Zhang et al. [17] tested the use of deep Convolutional Neural Network (CNN) in categorizing pavement damages based on imagery data. Zhang et al. [17], Ouyang and Xu [18], and Laurent et al. [19] tested the use of car-mounted 3D laser scanners in automated road surface inspection.
However, these sensor-based technologies also have their own limitations. Despite the accuracy and effectiveness of these monitoring methods, they are usually very expensive and, thus, the coverage and collection frequency can remain insufficient for detecting the dynamically changing road conditions. This project proposes to adopt a machine learning approach to predict road conditions based on running vehicles' vibration data, via sensors built in most smartphones. First, regular vehicles would be leveraged to provide a big amount of raw data that supplies rough information about road condition. Vehicle vibration is closely related to the road condition and thus is a valuable data source for road condition prediction. The challenge pertains to the obvious noise contained in data-for example, different brands and conditions of vehicles, different weather and traffic conditions, and varying driving behavior can all affect the quality and consistency of the collected data. As a result, the second step is to analyze the data using deep learning technologies, such as CNN. Evidence has shown that deep learning is very good at processing low quality data if the data size is sufficient [20]. The quality of vehicle vibration data is low in terms of telling the precise road conditions, but the amount of data can be considerable attributed to the big number of participants. As a result, the noise contained in the raw data can be filtered out by leveraging the power of deep learning technologies.

Methodology
We propose the use of vehicle vibration data to enable the automated recognition of road pavement damage. First, the moving vehicles' vibration data is collected using the built-in accelerometer of iPhone with a selfdesigned iOS app. In order to label the raw data appropriately, a DJI OSMO camera is set up to capture the imagery data of pavement conditions. The imagery data is then tagged by domain experts to establish the ground truth in the model training. After the training data is preprocessed and normalized, Sparse Coding will be applied to establish a dictionary of overcomplete bases, or atoms. In Sparse Coding, the atoms are the fundamental elements that can reconstruct the raw data with the minimum coefficient. They allow multiple representations of the raw signal but also provide an improvement in sparsity and flexibility of the representation. As a result, the dictionary of atoms can help extract the features of the original vibration data. Finally, the Support Vector Machines (SVMs) method will be used to classify the damage types. Figure 1 illustrates the method used in this study. Fig. 1. The proposed automated road damage recognition method based on vehicle vibrations; Sparse coding analysis decomposes raw vibration data into fundamental elements called "atoms". Atoms can be used to reproduce raw data and thus they are good features of the raw vibration data.

Data collection
We performed a preliminary study in College Station, Texas. An iPhone SE was selected as our main data collector. The iPhone build-in accelerometer has a specified output frequency of 4,000 samples per second. But the iOS restricts the maximum frequency to 100 Hz in order to save power. An iOS was developed to record the output data of accelerometer and UNIX timestamp at 100 Hz. For the purpose of labelling signals, a DJI OSMO camera was mounted on the front hood of the car to capture surface images of pavement. Furthermore, a phone, displaying UNIX time, was also mounted under the view of the camera to accurately synchronize the time between the camera and the inside iPhone SE (Figure 2b). The pavement damages have a main impact on the Y-axis (Figure 2a) signals. Therefore, we will use Y-axis data as learning signals in order to reduce the noise from other two axes. A total of 23 miles of roads in College Station, TX were used in the data collection ( Figure 2c). We labelled three types of pavement damage: pothole, ravelling and cracking. A pothole is a bowlshaped hole through one or more layers of the asphalt pavement structure, between about 150 mm (6 inches) and 900 mm (3 feet) in diameter. Ravelling is progressive deterioration of pavement surface as a result of loss of aggregate particles from the surface downward. The start time of the damage was determined and we segmented 100-points (1 second) after start time as a sample. Cracking is a common type of distresses caused by repeated heavy wheels load. There are 105 potholes, 38 ravelling and 24 cracking samples in our current dataset.

Sparse coding analysis
In this study we used Sparse Coding to extract the vibration features of moving vehicles. Sparse coding is a class of unsupervised methods that learn data patterns based on extracted overcomplete bases [21]. Although originally designed for image recognition [22], sparse coding has recently been used in classifying vibration data [23,24]. Vibration of the moving vehicles is the combination of engine vibration and response of pavement. The pavement damages could cause acute abnormal impulses in the vibration signals. It is assumed that the vibration patterns of the auto engine are consistent during driving. The acute impulses are regarded as the most characteristic patterns of the pavement distresses, and inspecting the abnormal impulses is equivalent to inspecting the vibration patterns of pavement damage. Therefore, the atoms of vibration signals can be learned by capturing the abnormal impulses and the pavement conditions videos, used to manually label the class of damage. In our experiments, 30 atoms, each with a length of 100-point (1 second of data) were learned from the Y-axis vibration signals labelled as pavement distresses ( Figure  3). The reason for choosing the length of 100-point is that pavement damage would result in an impulse lasting less than 100-point. The dictionary learning process can adapt the most characteristic patterns, i.e., the abnormal impulse, in the vibration signals with the 100-point length. Without loss of generality, three pavement damage conditions were tested respectively, including smooth pavement, big cracks and potholes. We reconstructed the vibrations using the sparse representation, which is a linear combination of learned atoms, of the raw signals. The mean absolute reconstruction error of pothole, pothole part and cracking are 0.11, 0.12, 0.11 amplitude, respectively. Figure 4 illustrates how Sparse Coding was used to reconstruct the temporal vibration data of a crack. Fig. 4. Comparison between a pothole Y-axis signal and its reconstruction based on atoms.

Vehicle vibration feature extraction
Then we performed the Sparse Dictionary Learning to find a dictionary that would adapt well to the raw data. The dictionary was applied with vibration data to solve the sparse representation of input signals, which would be utilized as sparse features for the further classification. Figure 5 illustrates five sample signals and their corresponding sparse representations. It indicates that sparsity was evident for each signal's features, and vibrations signals could be represented as a sparse linear combination of atoms.

Classification
Finally, Support Vector Machines (SVMs) method was used to classify the road pavement damages. SVM is a supervised learning method for classification, regression and outliners detection, which is defined by a separating hyperplane. Given labelled training data, the classifier would output a hyperplane which separate out classes. SVM is effective in high dimensional spaces and versatile with different kernel function.

Preliminary results
In comparison with sparse features, Y-axis amplitude time sequences were used as time domain features, that is, 100 amplitude features were selected for each signal with a length of 100-point. The dictionary with a set of atoms was learned using the minibatch dictionary learning in scikit-learn toolbox. Then apply the dictionary to the Y-axis vibration signals to extract the sparse features for each sample. Pavement damage recognition on these features were performed using SVM classifier found in the scikit-learn toolbox. Table 1 shows the recognition results for the SVM classifier. Overall, recognition is highest for pothole, which were 62% and 69% based on time sequence features and sparse features, respectively. The sparse features approach has little improvement compared to time domain method. The recognition accuracy is low for ravelling and cracking because of lack of data. The other reason why ravelling and cracking have low recognition rates and misclassified as pothole is that ravelling and cracking both involve the similar vibration pattern of pothole.

Discussion and conclusions
In this study we proposed an automated road pavement damage recognition method based on the vibration patterns of moving vehicles. The vibration features of regular moving vehicles are extracted to classify specific road pavement damages such as cracks and potholes. To reduce the deployment cost and difficulties, regular phones equipped with accelerometers are used in vibration data collection. Compared to the special equipment used in similar vibration studies, using regular phones makes data collection much easier and broader, but it could lead to low quality issues of data. To improve the effectiveness of using low quality data, we proposed the use of Sparse Coding, a deep learning method that finds a sparse representation of the input raw data in the form of a linear combination of basic elements called bases or atoms. The dimensionality of the raw temporal vibration data is therefore significantly reduced. A preliminary study was performed with vehicle vibration data collected in College Station, Texas. Results confirm the feasibility of the proposed method in automated road pavement damage recognition. More data points should be collected in the future to further benchmark the effectiveness of the proposed method. The methods under test, including the use of noisy vibration data and Sparse Coding as the analysis tool, involve many uncertainties. This research overall could represent a high-risk high impact topic. We admit the risk of this research topic but also recognize its significance if the hypothesis can be tested. If so, regular vehicles, along with the new deep learning algorithms, can be used to yield high fidelity data of road conditions. It can help significantly reduce the road inspection cost while improve relevant decisions. With more samples we will further test the effectiveness and scalability of the proposed method.