Dynamical Pattern Vector in Pattern Recognition with the Use of Thermal Images

The goal of the following paper was to develop the methodology of object tracking in adverse conditions. Suddenly appearing clouds, fog or smoke could be the examples of atmospheric conditions. We used thermal and visible images in each moment during object tracking. We computed the pattern vectors of the tracked object on the basis of the visual and thermal images separately. The pattern vector and current feature vector for an image of a given type are used to compute the distance between the object pattern vector and feature vector calculated for a given location of the aperture. It is calculated for both: the visual and thermal image. The crux of the proposed method was the algorithm of selection which distance (for visual or thermal image) was used for object tracking. It was obtained by multiplying the values of the distances between a pattern vector and current feature vector by some coefficients (different for thermal and visual images). The values of these coefficients depended on the usefulness of a given type of an image for pattern recognition. This usefulness was defined by the variability of the particular pixels in the image which is represented by calculating gradient in the image. On top of that, this study presented the examples of the object recognition by means of the developed method.


Introduction
Thermal imagery, which is obtained from an infrared sensor and can display invisible thermal energy emitted from objects, is a crucial source of information in low visibility conditions.This information can be very useful for image processing system which is used for surveillance, object tracking or security applications.
We can obtain complementary information of the objects which are located in an inspected scene when we use both: thermal and visible images.
In recent years, thermal imaging technology has been widely used in industrial, commercial and military applications.These applications include detection, tracking and identification of people [1,2,3], tracking vehicles on the road [4], or animals which can appear on the road [5].
Thermal imagery is also useful for a diagnosis of equipment [6] or for inspecting the production process [7,8].
In the following paper, we will consider the problem of object tracking and pattern recognition in the conditions of distorted visibility of the object.We assume that both visual and thermal images can be sometimes distorted.The proposed method is based on the separation of two parts of pattern vector in a pattern recognition task.One part depends on a visual image and the other one depends on a thermal image.The process of pattern recognition is based on this part of the pattern vector for which the image is not disturbed.

The Basic Assumptions of Pattern Recognition
The proposed method assumes that we will acquire the visual and thermal images (f V (i,j) and f T (i,j)) of the size of MxN in each moment of object tracking.All images were transformed to grey scale images.The mutual position of the conventional camera and thermal camera was parallel positioned.This way, we obtained visual image which corresponded to the same area in thermal camera.
We will compute the pattern vectors (J WZV and J WZT ) of the tracked object on the basis of the visual and thermal object's images (f WZV (i,j) and f WZT (i,j) of the size of mxn) separately.
The pattern vectors consist of moment invariants J 1 to J 8 defined in Eq.1.
The majority of the abovementioned definitions of the moment invariants were presented in [24].In [25] pattern vectors (J WZV and J WZT ) and current feature vectors (J V and J T ) are defined in Eq.2. (2) where f VAP (i,j) / f TAP (i,j) denotes a part of the image f V (i,j)/ f T (i,j) which is inside the aperture in the position (i,j).The pattern vector and current feature vector for an image of a given type (visual or thermal) are used to compute the distances ρ V (i,j) and ρ T (i,j).These distances are calculated for a given location of the aperture as it is presented in Eq. 3 We assume that a(0,1) and then we obtain ρ V (i,j) and ρ T (i,j) with values close to 0 for the object located beyond (i,j) position and 1 otherwise.We propose such a modification of ρ * (i,j) in order to obtain normalised values of an error function in the range (0, 1).The crux of the proposed method is the algorithm of deciding which ρ(i,j) (for visual or thermal image) is used for object tracking.
We assume that the usefulness of visual or thermal images may be defined by the variability of the particular pixels in the image which is represented by the function of gradient in the image.The low value of gradient indicates that the image or its part is not useful for our goal.Hence we obtain the values of the coefficients D C , D L , E C and E L as it is presented in Eq.4.
g 0 is the threshold value of the gradient.The coefficients calculated in Eq.4 allow us to create the function ρ(i,j) which consists of two parts which depend on visual and thermal pattern vectors separately.The function ρ(i,j), presented in Eq.5, is based on the correlation between the values of coefficients D C , D L , E C , E L and values of ρ V (i,j) and ρ T (i,j).) , ( ) , ( The coefficients close to 1 mean that the quality of the image is correct and at the same time ρ V (i,j) and ρ T (i,j) close to 1 mean that the object is recognized in (i,j) position.In this case, both parts, for visual and thermal pattern vectors, are used for pattern recognition.The coefficients close to 0 mean that the quality of the image is low (fog, smoke) and the corresponding part (visual or thermal) of the pattern vector is not used.

The Algorithm of the Use of the Dynamical Features Vector
The presented method is based on the simultaneous use of a visual and thermal image.
The first stage of the proposed algorithm consists of visual and thermal images acquisition.This process should provide two images in which the same parts of images represent the same parts of the observed terrain.The second stage is conducted with the assumption that we have good quality visual and thermal images which allow to obtain correct pattern vectors of the object for both images.This stage consists of calculating the searched object's pattern vectors for both representations of a searched object, separately for visual and thermal images according to Eq.2.
In the third stage, it is necessary to analyse whole images in order to define the potential parts of the images where the object may be found.This analysis is conducted for whole images (D C , E C denote the usefulness of the whole images for pattern recognition) and for every location of the aperture in the image (D L (i,j), E L (i,j) denote the usefulness of a given part of an image inside the aperture for pattern recognition).This procedure will be helpful for an image processing system to take into account during calculations only these parts of the images in which there are some artefacts which may constitute the elements of the searched object.Therefore, in this case, we will calculate grey scale gradient.
The last stage is to calculate distance between the pattern vector and current feature vector.In this stage, we use the parts of the current feature vectors according to the analysis carried out in the previous stage.It is realised by calculating the abovementioned distance (Eq.5) with the proper coefficients.The values of these coefficients depend on whether the whole image or its part is disturbed by smoke or other disturbances.The object is located in position (i m , j m ) in the image if and only if ρ(i m , j m ) = min ρ(i, j).

Examples
The proposed examples will present the problem of vehicle tracking in various visibility conditions.The figures present the visible and thermal images pairs and the location of the searched object.The following cases correspond to the following situations: 1) An off-road vehicle was the tracked object.According to Fig. 1, both visual and thermal images are of a good quality (D C , = 1; E C =1); the parts of images containing object can be used to obtain pattern vectors for visual and thermal object images.The recognition will be ICCAE 2016 2010 0 (2016 conducted on the basis of both images.Fig. 1c presents correctly calculated location of the object. 2) According to Fig. 2, the visual image is partially distorted by smoke whereas the thermal image is correct (D C , = 1; E C =1).In this case, searching the object will be conducted on the basis of both images; however, D L (i,j) in the smoky regions will assume zero values and in these places only thermal image will be used.Fig. 2 presents object location which is slightly shifted with respect to Fig. 1c; however, we can assume that it is correctly calculated location of the object.
3) According to Fig. 3, the visual image is correct whereas thermal image is distorted because the temperature of the part of the searched object has changed.In this case, object searching will be conducted on the basis of both images; nevertheless, the value of the thermal image is minor.However, the thermal image cannot be eliminated from calculations because without additional information about the changes of the object temperature we are not able to interpret the thermal image correctly.Fig. 3c presents correctly calculated location of the object.

Conclusions
The paper has developed the methodology of object tracking in adverse conditions.In order to conduct the tests, we used the example of the disturbance in the form of smoke and changes of the searched object temperature.The obtained results could be used for other disturbances such as suddenly appearing clouds, fog or adverse atmospheric conditions.We used thermal and visible images, in each moment during object tracking.We described searched object features by computing the pattern vectors containing moment invariants.These vectors were calculated on the basis of the visual and thermal images separately.The distance between the object pattern vector and current feature vector was calculated for a given location of the aperture separately for visual and thermal images.The crux of the proposed method was the algorithm of selection which distance (for visual or thermal image) was used for object tracking.This algorithm was based on multiplying the values of the distances between a pattern vector and current feature vector by some coefficients (different for thermal and visual images).The algorithm of the calculation of these coefficients, based on a gradient function which corresponds to the usefulness of the image for pattern recognition, was presented in the paper.There were proposed coefficients which were calculated for the whole image (global) and for a current aperture location (local).Finally, this study presented the examples of the object recognition by means of the developed method.

Figure 1 .
Figure 1.The visual image a), thermal image b) and calculated location c) of the searched object for the case 1.

Figure 2 .
Figure 2. The visual image a), thermal image b) and calculated location c) of the searched object for the case 2.

Figure 3 .
Figure 3.The visual image a), thermal image b) and calculated location c) of the searched object for the case 3.
. In order to assess this distance between vectors we will use Euclidian metric.