Defect Detection in Pipes using a Mobile Laser-Optics Technology and Digital Geometry

This paper presents a novel method for defect detection in pipes using a mobile laser-optics technology and conventional digital-geometry-based image processing techniques. The laser-optics consists of a laser that projects a line onto the pipe’s surface, and an omnidirectional camera. It can be mounted on a pipe crawling robot for conducting continuous inspection. The projected laser line will be seen as a half-oval in the image. When the laser line passes over defected points, the image moments on the pixel information would change. We propose a B-spline curve fitting on the digitally-convoluted image and a curvature estimation algorithm to detect the defects from the image. Defect sizes of 2 mm or larger can be detected using this method in pipes of up to 24 inch in diameter. The proposed sensor can detect 180-degree (i.e., upper half surface of the pipe). By turning the sensor 180 degrees, one will be able to detect the other half (i.e., lower half of the pipe’s surface). While, 360-degree laser rings are available commercially, but they did not provide the intensity needed for our experimentation. We also propose a fast boundary extraction algorithm for real time detection of defects, where a trace of consecutive images are used to track the image features. Tests were carried out on PVC and steel pipes.


Introduction
Automated surface inspection of pipes refers to a class of methods and algorithms which detect, classify, localize and measure surface defects on the interior surface of pipes. Different sensors like vision, sonar and thermal are used for pipe inspection. In this paper we propose a laseroptics sensor for detecting external and visible defects.
The literature on image-based inspection of pipes is vast. Duran et, al used a perspective camera and circleprojecting LED ring for visual inspection of small sewer pipes, [1][2]. They used a pinhole camera to image the LED ring, and then used artificial Neural Networks for analyzing the brightness of the LED ring to detect the defects. This method was used to detect large defects only. Given the limited Field of View (FOV) of the pinhole cameras, the LED ring must be projected at far to be visible. This trades off the resolution.
Alternative imaging systems, beyond a pinhole projection, with a wide FOV would be required to make the high-resolution imaging of the entire interior surface of the pipes possible. There are three main approaches for panoramic imaging of the interior surface of a pipe: multiple camera systems, rotary imaging platforms, and omnidirectional imaging systems. Multiple-camera and rotary-imaging systems can capture the pipe surface with higher resolution than that in omnidirectional imaging systems. However, because of the difficulties in image feature bundle adjustment, image stitching and mosaicing, multiple-view calibration and also synchronization of the imaging systems, they have been rarely utilized in industry. Furthermore, the overall cost of these systems is prohibitive. In this paper we focus on visual inspection of the interior surface of pipes using a single-view omnidirectional imaging sensor and a laser line used as the structured light image fringe.
Single-view Omnidirectional imaging can be categorized as: Catadioptric and Dioptric. Both can be used for capturing a panoramic view of the scene in one single image. A Catadioptric sensor consists of a perspective camera and a hyperbolic or parabolic mirror. Dioptric imaging sensors, on the other hand, consist of a camera with a compound lens (e.g., a fish-eye lens) with about 180-degree horizontal and vertical field of views. This would enable the camera to see a hemisphere lobe in front of it.
In this paper, we investigate defect detection on the interior surface of pipes using a laser ring, projected onto the pipe's wall, and an omnidirectional imaging system. The proposed methodology can be used with either of the Catadioptric or Dioptric imaging sensors. High distortion in images leads to nonlinear resolution. Further filtering would be required to account for this. On the positive side, however, both offer benefits such as; compact size, wide FOV, and easy calibration.
There are some works on visual inspection of pipes using omnidirectional sensors cited in the literature, [3][4] light for the digital reconstruction of the interior surface of pipes, [5]. Their main focus was on the design of their optical system, thus, they do not address the defect detection in depth.
Two main light sources widely used for illuminating the pipe wall in pipe inspection applications are: diffused and structured. The former uses a high-intensity yet diffused light source, where the latter adopts a collimated structured light using lasers. There are different mechanisms for generating structured light patterns in machine vision, namely, laser pattern projectors and image fringes generated by the common data projectors. The laser projectors create a single-wavelength high contrast pattern of the laser light, whereas the data projectors project a multi-wavelength light (aka, image fringe) at a lower intensity and contrast. Although data projectors have the advantage of creating a dynamic range of patterns on the fly, but because of the lower light intensity and contrast they offer, and also their bigger size factor, they have been rarely used in industry. Recent developments on imaging sensors and actuators under the MEMS/NEMS, though, can be a game changer. Inari, [6], used the laser ring projector to create a narrow ring on the interior surface of a pipe.
In this research we used a high-intensity laser line projector as the lighting subsystem to produce a highly focused illumination inside the pipe. Rather than using laser ring projectors with low fan angle which makes the inspection of large diameter pipes a serious challenge, we use the laser line projector laterally and perpendicular to the pipe surface to produce a half-ring inside the pipe right next to the camera, (see Figure 1a). This half-ring will be seen as a half-oval in the image. The proposed sensor can detect 180-degree (i.e., upper half) of the pipe's surface. By turning the sensor 180 degrees, one will be able to detect the other half (i.e., lower half of the pipe's surface). While, 360-degree laser rings are available commercially, but they did not provide the intensity needed for our experimentation. In this laser-optics imaging system, an elliptical curve is formed within the image, whose size and form would depend on the relative position/orientation (aka, pose) of the laser optics imaging system inside the pipe.  Every surface defect would locally change the curvature of the laser line. The main idea is to extract and analyze these local changes in curvature in the elliptical curve seen in the image to detect, classify, localize and measure surface defects. However, the main focus of this work is on defect detection only.
In this work, we use a dynamic reference approach, rather than using a fixed silhouette as the signature image, for extracting surface defects from the projected laser ring, which further paves the way towards the development of the defect detection algorithms robust to the unwanted sway in motion in the locomotion unit carrying the sensor inside the pipe. In our proposed method, we formulate a novel algorithm to track the local changes in the curvature of the almost-elliptical shape of the laser line in the image to find defects. It is noteworthy, however, that a-priori knowledge of the curve's geometry can drastically help in tracking discontinuity in the local curvature, which will be described in further details.
We conducted several experiments on the steel and PVC pipes. Results were promising. The proposed algorithm could detect small defects of 2-mm in size in all directions (i.e., depth, width, and length) or larger. We present the system setup for our pipe inspection method in the next section. Then we describe the defect detection method in details. Finally, we present a brief discussion about the method and provide some conclusions.

System Setup and Problem Definition
Figures 1a and 1b show a schematic of the experimental setup and the real system, respectively. The system consists of a Catadioptric omnidirectional imaging unit with a hyperbolic mirror, a laser line projecting device, and a linear slide. The sensor is mounted on a motorized linear slide whose motion can be controlled precisely using optical encoders. Also, the sensor is attached to a lab jack. It was used to position the sensor on the pipe's centreline.
A green laser line projector was used as the light source. It provides a high-intensity/contrast image of the scene. The laser produces a thin line of green light, which projects a half-ring on the interior surface of the pipe. This half-ring will be seen as a half-ellipse in the image. One should note that the image of the laser ring with a finite thickness would appear as two converging elliptical curves, one representing the upper boundary and the other the lower one. They intersect, or converge, at the lower part of the image (i.e., vanishing points due to the perspective imaging). Also, the light intensity in the laser ring will change radially due to the texture of the pipe's surface, lens distortion, and reflectivity characteristic of the pipe's surface. A pre-filtering process was implemented to correct for the gradual change in the light intensity.
There are some design factors to be considered in an omnidirectional laser optics system. The lighting system, in particular, must provide the highest visibility and resolution. For instance, the laser line should be seen close to the boundary of the image for the highest resolution, [7]. It has been theoretically and experimentally verified that for achieving the highest longitudinal and depth resolutions, the laser ring must be projected onto the pipe surface right above the reflective mirror in Catadioptric imaging systems, [7]. This means that the projected laser line will be reflected through the mirror's periphery in the Catadioptric sensors and fish-eye lens's periphery in the Dioptric sensors. As one can see in figure 1a, the laser line projector is projecting a beam perpendicular to the optical axis of the camera and close to the periphery of the mirror.
The desired characteristics of the proposed design, to achieve the highest resolution, are as follows: x The optical axis of the imaging sensor and the projection center of the laser must be aligned. x The relative position between the imaging sensor and the laser must be adjusted based on the pipe's size.
x The camera and laser must be centered in the pipe for achieving a uniformly-distributed resolution.
x The relative position between the imaging sensor and the mirror, in case of using a Catadioptric imaging system, must be adjusted for achieving the highest resolution.

Methodology
In this section, we present a set of algorithms used for defect detection in pipes using the proposed omnidirectional laser-optics platform consisting of an omnidirectional imaging sensor and a laser line projector. We used B-splines to fit two curves to the upper and lower boundaries of the laser ring's image. For better accuracy, the defected points, as outliers, were first detected and extracted from the image using a fast elliptical curve fitting and local curvature estimation algorithm at a very course level. The B-spline was then used on the remaining inliers only. At the next stage, the B-spline curves, fit to the upper and lower boundaries, were compared to the measured boundary points along two perpendicular axes in the pixel frame using conventional digital geometry techniques. Figure 2 shows the flowchart of the proposed algorithm. All the major steps are explained in the following sub sections.

dynamic-intensity thresholding of the laser ring's image
The very first task after capturing an image is to extract the upper/lower boundaries of the laser fringe. For this, the RGB image was first converted to a gray-level and then to a B/W after conducting the dynamic-intensity thresholding.
Factors such as the ambient light, the texture of the pipe's surface, the reflectivity of the pipe's surface, and the optical characteristics of the imaging system would lead to a non-uniform distribution of the laser light intensity in the image. In general, the intensity of the light decreases as the polar angle to the central axis of the laser projector and/or the distance to the laser's projection center increase. However, assuming that the laser projector is perfectly aligned with the pipe's central axis, one can conclude that the intensity would change only radially. This piece of evidence was used to design a dynamic-intensity thresholding algorithm.
We modeled the radial intensity variation using a cubic polynomial as an image convolution mask to digitally adjust the average intensity on each pixel in the gray-level image. Figure 3 shows a representative example on fitting a cubical polynomial to the variation of the light intensity in the image. A cubic polynomial will provide a smooth change in the intensity that can be calculated in a very short time. A comparison between our proposed dynamicintensity and the conventional fixed-intensity thresholding methods was done through experiments. Figure 4 shows a representative comparison between them. As can be seen, low values of the threshold used in the fixed-intensity causes some light diffusions to be counted as part of the curve, whereas high values chosen for the threshold would entirely disregard low-illumination part of the curve. On the contrary, the dynamic-intensity thresholding approach not only keeps low illumination parts intact, but also it removes spattered light diffusion found at the center of the curve. The light intensity at any point p on the laser image located at a polar angle, θp is updated as follows: where Iref denotes the reference intensity profile calculated through Algorithm 1.

Fast laser ring boundary extraction through tracking
After digitally masking the light intensity in the image based on dynamic-intensity threshodloing, a simple boundary extraction algorithm can be used to locate the upper/lower boundaries of the laser ring in the image. Through experimentations we have realized that the boundary extraction would be the most time consuming part. Therefore, to improve the computation time, we devised a novel boundary extraction algorithm through which the boundary information obtained in the previous image frames were used for tracking purposes.
Algorithm 2 explains the process. In this algorithm we assume that the position/orientation of the laser ring's image would not change considerably between consecutive frames. Therefore, one can restrict the search domain to a small region than the entire image.
In this algorithm, the local search for the boundaries is done radially with respect to the image centre. Figure 5 shows a schematic of the search domain used in Algorithm 2. A coarse elliptical curve fitting, to increase the speed of computation, was employed to represent boundary curves. Search radially at the vicinity of the point, for an "edge" point. 9: end for 10: Return the list of the edge points found as the "boundary" points.
We tested our algorithm on a 2.00 GHz PC with a duo processor and on a 1280×958 pixel image. The boundary extraction algorithm was done in 20 msec, which is fast enough for real time applications.

Boundary segmentation
The image of the laser ring, when the laser passing over defected areas, will induce sharp and large dispersion in the light intensity along the boundaries. Figure 6 is a representative example of the laser ring projected over a region with dents, cracks, protrusion, and slivers. Detection of those defects and also sizing them, if needed, would be possible by comparing that with the smooth image of the laser ring (i.e., an image taken under ideal conditions). Therefore, one needs to calculate the signature silhouette of the upper/lower boundary curves, as a reference first. This would require a boundary segmentation to exclude the outliers in the image due to the presence of defects. As can be seen in Figure 6, abnormalities in the image such as: isolated islands of high intensity, and local discontinuity are possible. We developed an ANN-based filter, based on some statistics such as; (1) the size of the high-intensity island, (2) the roundness of the islands, (3) the distance between the islands and the closest connected component, and (4) orientation of the discontinuity, to filter out the defected areas and to do the segmentation on the smooth part of the upper/lower boundary curves. Further details on the statistic measures used and the ANN architecture can be found in [8]. Figure 7 shows one representative example of a boundary segmentation results.

B-spline fitting to the upper/lower boundary curves
We used B-splines curve fitting, [9], to approximate all the inliers on the segmented upper/lower boundary curves. B-splines are used when there is no theoretical basis for choosing a fitting function. We started by elliptical curve fitting for tracking purposes. However, we had strong grounds to believe that the image of the laser ring would deviate from an ellipse given the nonlinear lens distortion in the imaging system and also the misalignment between the laser-optics sensor and the pipe's centreline. While an elliptical curve fitting suits to the fast tracking of the boundary curves, a higher-degree precise curve fitting would be required for representing the boundary curves at a higher resolution for defect detection purposes.
We used the method of least squares to minimize the following error function through our B-spline curve fitting method: Where r(x) denotes the datum value of the boundary curves, α denotes the knot values of the B-spline, and W(x) denotes the weight associated with each datum. A higher weight was adopted for points on the boundary curves with larger curvature (i.e., a denser local spline was used around those regions). Figure 8 shows the B-spline fitted curve to the upper boundary of the laser ring's image. As can be seen, there is a good fit between the segmented part of the image and the splined curve (in red colour) despite the presence of the discontinuity in the image due to defected points.

Defect extraction and sizing
As discussed earlier, we eliminate the defected points from the boundary curves to find a precise B-spline fit to the non-defected points. Now these fitted curves to the upper/lower boundaries can be used to extract defected points at a much higher resolution. We refer to this as a cascaded filtering strategy.
To detect and extract the defects on a boundary curve, we calculate the distance from the original boundary points to their corresponding points on the Bsplined curve based on: Vertical and Radial displacements.
Let us take ‫‬ = < ‫ݔ‬ , ‫ݕ‬ > as a point on the boundary curve. We define the vertical displacement of this point as: In fact, the vertical displacement of point p is a Euclidean distance from the point p to the corresponding point on the B-splined fitted curve having the same x coordinates. We also define radial displacement of the point p as (see Figure 9): A weighted sum of the dv(p) and dr(p) was used as a metric to detect and size the defects. We define the defected region as a set of consecutive defected points on the boundaries which have a weighted displacement error larger than a specific user-defined threshold. We set this threshold value as a low-pass filter. It would reject the image noise (whose displacement value would be very small) and return the high values of the displacement metric associated with the defected points. After a number of experimental testing and tuning, we were able to detect defects of 2-mm or larger in all directions with 0% false return. Figure 10 shows the weighted sum of the vertical and radial displacement profile of one single boundary curve. As shown in this figure, the proposed displacement metric can find the defected points effectively (i.e., defected region causes a big spike in the value of the displacement metric) .  Figure 11 shows the variation of the displacement value between the points on the upper and lower boundary curves and that with their corresponding B-splined curves for the entire visible part of the image. A 2mm defect in form of a cut in the pipe was correctly detected. The big spike in the image, seen at the bottom of Figure 11, indicates the presence of a defected area. The snap shot shown in Figure 11 shows a frozen image of the laser ring at the exact time it was passing over the cut out region on the pipe as a man-made defect. It is noteworthy that the defected area was detected by analysing the upper boundary of the laser image. The same pattern was observed on the lower boundary curve after a number of elapsed images when passing over the defect (not shown in the figure). This means that every defect will be picked by both of the boundary curves providing the user with a sanity check on the validity of the computations. One can identify a false positive if the defected area was picked in only one of the upper or lower boundary curves.

Omnidirectional versus pinhole imaging systems
In this work, we used the omnidirectional imaging sensor instead of perspective cameras for a panoramic view of the pipe's surface. The main drawback of the omnidirectional sensors is the high image distortion and lower resolution. However, for larger pipes, the resolution in omnidirectional imaging systems can be comparable to that in pinhole. We tested our proposed algorithm on a 24inch PVC pipe. Our experiments showed that the resolution of the omnidirectional sensor used was good enough to detect defects of 2-mm in size or larger.

Real-time inspection.
In this work we applied five main steps to extract the defects by looking at the upper/lower boundaries in the laser ring's image. Except the dynamic-intensity thresholding algorithm, that needs to scan all the image pixels with a time complexity of O(n 2 ), the remaining steps in the proposed algorithm can be executed in a linear time, simply because they process the pixels on the boundary points only and not on the entire image.

Conclusions and Future Work
We presented a novel approach for surface inspection of pipes using a laser-optics technology that consists of an omnidirectional imaging sensor and a collimated laser line projected onto the pipe's wall. Two B-spline-based curves were fit to the upper and lower boundaries of the laser ring's image as a signature or reference silhouette. These signature curves were used then for detecting defects. We concluded: 1. Despite the fact that the overall spatial resolution provided by the Catadioptric and Dioptric imaging sensors is lower than that in the perspective cameras, [10], they provide a compact, and cost-effective solution for panoramic imaging of pipes of any size. 2. Structured light can highlight defects better than the diffused light. However, the light intensity of the laser, projected onto the pipe's wall, can change radially. We proposed a dynamic-intensity thresholding algorithm to compensate for the light intensity variations in the projected laser ring. 3. Calculating the variation of the local curvature along the upper/lower boundaries of the structured light can identify defects. After some nominal tuning of the parameters, we were able to detect defects with a high confidence. A field trial of the prototype is scheduled for summer 2015.