Automated detection-classification of defects on photo-voltaic modules assisted by thermal drone inspection

A new computational procedure is proposed for the automated detection-classification of defects on photovoltaic (PV) modules-panels. Thermal imaging or IR thermography is an important and powerful nondestructive technique for the investigation of structural or operational defects on PV modules and when it is combined with drones can provide a fully automated inspection, detection and defect classification procedure. The aforementioned image processing approach adopts preand postprocessing tools and methodologies assisting the infrared (IR) thermography for the evaluation of a photovoltaic (PV) module performance. In particular, the passive approach of IR thermography was adopted, a portable thermal imager was used for the in-situ acquisition of images that show the distribution of infrared luminance of the PV panel surface. The acquired images are processed and analyzed for the detection and classification of defects and hot spots on the module’s surface that are potential candidates for faulty operation. The proposed computational methodology adopts gaussian filters for the IR images, thresholding operations, morphological transformations and Artificial Neural Networks. The use of IR thermography assisted by Unmanned Aerial Vehicles (UAVs) for the inspection of PV modules-panels proved to be a very reliable and efficient tool towards the automated detection-classification of


Introduction
Over the passing decades, the international photovoltaic (PV) market has experienced an undeniably impressive and rising growth in terms of installation and investments in the field, thus presenting an exponential rate in sales and in general use for various applications. Because of the exponential growth, PV solar energy has evolved from a small market into a mainstream and competitive field in the global energy market. Moreover, according to long-term scenarios studies, it is anticipated to keep having a robust growth over the following years with a progressively increasing role towards the switch of many countries to low carbon emitting technologies [1].
The power generation of PV plants and the consequent payback time of investments on PV installations, are significantly dependent on the operational lifetime of PV modules and their electrical performance. The experience accumulated, through the recent decades, upon extensive quality control and qualification testing of PV modules at the manufacturing/assembling stages, had a key contribution towards minimizing the infant mortality ratio of such components.
The objective of this study is to detect and classify defects found in thermal images of PV modules in a fully automated manner. In order to achieve this an IR thermal camera has been mounted to an unmanned aerial vehicle (UAV) and accompanies its flight inspections to gather IR thermal images and creating so image data sets. The images are transferred to the image processor module for further analysis and investigation. After the processing of the gathered images, a detailed defect/fault report is sent to the main system controller for the necessary actions towards the optimized operation of the PV energy park. The main computational tool adopted in the present research study was the open computer vision (OpenCV) library [2] that finds applications in areas such as facial recognition, motion tracking and image segmentation. The programming language Python [3] was chosen for the in-house software development towards the detection-classification of defects in PV modules using IR thermography due to the simplicity, versatility and compatibility with other computational tools and libraries that provides. While there are several works [4,5] based on automated PV plant inspection systems, in their majority depend on classic image processing techniques, such as intensity thresholding. These techniques although effective, they rely heavily on the manual tuning by the user and need calibration in every iteration for each individual image. In the present resent effort a combination of UAV flights accompanied by an automatic methodology to identify and classify defects is provided. With the use of both UAV flight scheduling and image processing computational procedure the aforementioned approach can be applied to large scale PV plants for fast and low cost inspections without sacrificing the accuracy of the results.

Digital imaging Algorithm
For the purposes of this study a digital imaging methodology has been developed to process thermal images utilizing various techniques used for thermal imaging [6]. The goal of this computational procedure is the development of state-of-the-art software for the detection of defective PV modules from infrared images. The images are collected via a UAV that performs flights from various altitudes. The computational procedure that has been developed consists of various pre-processing and post-processing steps. The pre-processing phase incorporates the following steps: 1. collection of infrared thermography images and the initialization of the image processor, 2. loading of images and their additional preparation and processing using Artificial Neural Networks (ANN), 3. normalization of infrared thermography images, 4. application of Gaussian filter (Gaussian blurring) and image thresholding, 6. morphological transformations of thermal image (erosion, dilation), 7. Connected Component Analysis, 8. Clustering algorithms. The post-processing phase consists of the following steps: 9. determination of the area of interest (ROI) for the detected fault, 10. characterization of the fault based on its temperature, pixel intensity and shape of the fault, 11. detailed report is prepared and further provided to the system controller (including type and exact coordinates of the fault) for the PV health monitoring status that can be used for additional flight scheduling, if necessary. The aforementioned numerical procedure is presented in Fig. 1. In particular, the integrated image processor module incorporates the following: collection, storage and proper labeling (date, PV park, flight altitude and relative global positioning system characteristics) of the thermal images in the created database, use of Gaussian blurring sequence, use of binary thresholding operation in order to filter out unwanted noise from the thermal images, various types of morphological operations such as erosion and dilation to remove small blobs of noise from the thresholding images, the application of a connected component analysis in order to store the areas that are of interest and lastly a Kmeans clustering algorithm.
Τhe initialization of the computational procedure takes place with the collection of thermal images. The process initiates via the system controller which sends a starting signal to UAV along with a detailed flight plan. During the UAV flight the images are collected and transferred to the image processing module for analysis and processing. During the image data set creation and the development of the image processor module a problem that inevitably encountered was the gathering-accumulation of thermal images not containing PV modules. Instead, these images depicted trees, grass among other characteristics of the surrounding PV module area bringing no valuable information to the image analysis procedure while significantly increasing the computational process time. The solution was the adoption of artificial neural networks, in particular a convolutional neural network (CNN). The CNN is trained with thermal images obtained from various flights that contained PV modules and images containing grass and trees. This methodology enables us to train a reliable model which filters out unwanted thermal images not containing PV modules.
The first step of the computational approach is the normalization of the thermal images and the conversion to a more suitable format in order to proceed with the next steps of the computational procedure. This step is of great importance because the accuracy of the calculations performed are affected by the corrected normalization of said images. After the format conversion the images are normalized into a visible range and lastly the pixel to temperature conversion takes place using the cameras' manufacturer conversion ratio. Also, a colormap is applied to the thermal images for improved visualization capabilities. The computational process continues with the application of Gaussian blurring. The purpose of this image blurring filter is to apply a smoothing over the initial image to remove excess noise contained in the image. This methodology is applied in three steps: the first step is the conversion of the original image into grayscale, this enables us to convert the image into a 2-dimensional matrix. The next step is the application of the blurring to the grayscale image  Moreover, the computational methodology proceeds with the application of image thresholding to the output of the Gaussian blurring procedure. Image thresholding is one of the simplest methods used in digital image processing for image segmentation. In this study a simple binary type of segmentation has been used that proved to be sufficient and yield satisfactory results. The binary segmentation is applied by setting a constant T where everything above this value is converted to white and everything below it is considered as black. The results from the binary type of segmentation are shown in Fig. 3, where at the left side the output of the Gaussian blurring is presented and at the right side the results from the binary type of segmentation are also given. With the application of the image thresholding, the part that is of interest from each IR thermal image is isolated and is ready to further processed. A common technique used in digital image processing is the application of morphological transformations to increase or decrease the area of the isolated segmented parts. In this study two types of morphological transformations were applied: the first one is the dilation where the segmented part has its area increased and secondly the erosion where the segmented part has its area decreased. The results for both types of morphological transformations (erosion and dilation) are shown in Fig. 4. For the case of dilation (Fig. 4 a) we have the input image, we apply thresholding and apply the dilation process to the segmented part. For the erosion (Fig. 4 b) we have an input image we apply image segmentation and apply the erosion process to the isolated areas. It should be noted that a combination of both techniques was utilized in this study. Having applied morphological transformations to the images under processing, the computational process continues with the storage of each segmented component individually. In digital image processing these components are usually called contours. In this stage of the computational procedure, a connected component analysis was performed, which stores each segmented part individually and saves them in a matrix form for further processing. In the following step with have the application of the k-means clustering algorithm, which marks the last step of the pre-processing phase. The k-means algorithm is one of the simplest unsupervised learning algorithms that solve the well-known clustering problem. The post processing phase is presented in Fig. 5, and it includes the detection, marking and extraction of the isolated parts from the preprocessing phase. The post processing phase is necessary for the correct categorization and characterization of the defective PV modules. Based on the shape, pixel intensity and temperature the categorization and characterization take place. After the computational procedure has completed its course, it sends a detailed report to the system controller.

Thermal Imaging Characterization
As it is known, IR thermography is an important and very powerful non-destructive, remote sensing methodology which has proved to be an effective technique providing temperature mappings utilizing the IR radiation which is emitted by all objects proportionally to their temperatures. Using IR thermography the in depth analysis of thermal images gives useful information about the surface temperature, while further image processing can reveal possible abnormalities to the thermal signature of the inspected material or equipment. These data, along with the knowledge of the equipment's physical construction and thermodynamic state, are used to evaluate the degree of deterioration. The development of a computational method for the characterization of faults-defects on PV modules by IR thermography necessitates large and rich datasets of thermal images of PV modules from UAV flights. The image resolution combined with the variety of faults in PV modules can play an essential role in the realistic characterization of PV panels in real conditions (on field-testing) which is the final objective of this research study. These prerequisites contribute to the optimal operation of the numerical algorithm for the identification and characterization of defective PVs. In the present work the fault characterization takes place in two phases following the criteria published in the relative literature [7] which are given in Tables 1 and 2. Firstly, we categorize the hotspot by its temperature range and classify it with respect to the severity stage. In a second step that is based on the shape and temperature of the detected hotspot we assess and analyze the detected fault.

Results and Discussion
With the application of the aforementioned numerical methodology to the data set of thermal images we can apply the computational procedure to test its capabilities on UAV flight missions. When the digital image processing algorithm detects a hotspot on a particular thermal image from PV modules categorizes the fault into a severity stage, assess the fault detected and sends a detailed report back to the system controller for further actions such as scheduling of a new UAV flight. In Fig. 6 (a) various thermal images are presented obtained by UAV flight performed at 30m and for PV modules being in their optimal state (verified by supporting on field electric characteristics and power measurements) without defects. In these images the image processor does not detect a PV module fault in conjunction with the electric characteristics monitoring. In Fig. 6 (b) the image processor detects various hotspots in malfunctioning PV modules. In order to assess and characterize the faults detected on the IR images in detail and higher accuracy a new flight is being scheduled for the UAV in lower flight heights and around the defected PV panels. Representative IR thermal image from the rescheduled 5m-height UAV flight is shown in Fig. 7. The resolution of each image is a crucial factor for the success of the computational methodology and the correct characterization of the faults detected. The image processing algorithm assesses that the hotspot detected in Fig. 7 is a burned cell, which found to be a correct assessment ascertained by on-site inspection with a portable thermal camera of high resolution. The size and shape of the of the fault are determined by the regions of interest that were extracted from the post processing phase of the computational procedure and combined with the temperature extraction from the pre-processing phase can give a correct assessment of the hotspot detected.  It should be noted that in order to have a plethora of data sets containing various types of faults artificial defects were created on PV modules and used extensively in the UAV flights. The purpose of these artificial defects was the enrichment of the database containing thermal images and the increase of detection, classification capabilities of the developed IR thermal image processor.

Conclusions
IR thermal images from UAV flights were combined with digital image processing techniques for the automated detection and characterization of defects in PV modules. The proposed computational procedure succeeds in the detection and characterization of defects found in operational PV modules. The automated thermal drone inspection can give interesting prospects regarding nondestructive testing and structural health monitoring for the prevention of failures in PV modules.