Diagnosis of machine vision of an unmanned vehicle

There is an increase in the number of cars using artificial intelligence. Therefore, it is necessary to provide quality maintenance of artificial intelligence components, such as machine vision (MV). The paper considers a general approach to the diagnosis of the unmanned vehicle. Based on the analysis of the use of existing systems, general requirements for the diagnosis of unmanned vehicle MV were formulated, diagnostic parameters were proposed. To solve the problem of testing, debugging and diagnostics of the MV, it is proposed to use virtual polygons built using the methods of procedural computer graphics. After diagnosing the MV video cameras, if necessary, they are calibrated according to the images of a special test object.


Introduction
The automotive industry is a leader in the application of machine vision technology (MV) and is its largest consumer. According to analysts [1,2], the automotive industry forms 23% of the market of MV products in Germany. And according to VDMA, for Europe this figure is 21%. Therefore, it is not surprising that the algorithms of the Ministry of Health gradually began to be used in the cars themselves -autopilots, and not only at the stages of their production.
Practice shows that the autopilot works correctly in 80% of road situations. Modern unmanned vehicles have a maximum of the third level of autonomy. Autopilot only helps the driver. One of the reasons for the imperfection of the autopilot is the imperfection of the MV. And until an optimal solution to this problem is found, the level of autonomy of unmanned vehicles will not rise above 3+ [3].
Continuity: due to the lack of fatigue, the MV system can be used in continuous use mode without stopping. Repeatability: control is carried out under fixed and uniform conditions, which guarantees the adequacy of the assessment [5].
Consistency: automation of the process avoids human subjectivity, and therefore allows you to maintain a constant level of quality. Also, reduce costs [6].
Among the disadvantages should be noted: the need for constant and clear lighting; the need to calibrate the camera [7].
Difficulties in recognizing objects that overlap or have an adjacent color to the background; higher final cost of works [8].

Basic material
Based on the analysis, the diagnostic system of the Ministry of Health for use in automotive services is proposed.
MV diagnostic system -a set of tools and methods for diagnosing MV, which allows you to detect a faulty element of MV in the most rational way [9]. The troubleshooting procedure is expected to be performed automatically. First, the MV can be in two mutually exclusive and discriminatory states (operational and inoperable). Secondly, it is possible to allocate elements (blocks), each of which is also characterized by the distinguishing states defined as a result of checks. [10,11,12].
The main components of the MV are conditionally presented in Fig. 1. This figure shows all possible combinations of MV elements at once. In practical tasks, these components can be used in various combinations.
The object of interest of the Ministry of Health is an object of the external world, information about which is of interest for use in a practical task. The scale and size of the objects of interest can be very different, as may the information required to obtain through the MV. Lighting and reflective properties of objects are those processes and phenomena that allow contactless information about the illuminated objects. Lighting can be controlled (specially organized) or external to the MV. Optical system -a system of lenses, through which the light flux from the area of interest is projected on the transducer "light-signal". Most often, such a system uses a standard photo, micro and television lenses. A Light-signal converter is a device that converts the energy of incident light into an object into electricity. Traditionally, in such devices, the voltage of the output signal is proportional to the number of incident photons on the corresponding surface of the converter. Computer or special computer -equipment that implements algorithms for collecting and processing visual data. Mathematical software -a set of mathematical models, algorithms and programs. Analog data input device -converter of analog visual data into digital. Its main part is a high-speed analog-to-digital converter and a bus connected to computer memory. There are options for both built-in boards (framegraphers or video capture cards) and external devices built into digital video cameras. When using this device it is necessary to take into account the scale of real-time, "light-signal" set by the converter. External devices -external to the MV and devices that affect the object of interest, the processes that take place in real-time. Actuators -devices that are connected to the MV and with which you can influence the observed scene and, in particular, the object of interest.
Analysis of literature sources and exploratory research revealed the main factors influencing the accuracy of the Ministry of Health: -when forming the image: the type and type of television camera and video card, the settings of the equipment, the distance between the optical center of the television camera and the object; lens aperture, image magnification, MV calibration, position and location of light sources, object illuminance, background-object contrast ratio, shape, size and material of the object of interest, ratio of linear dimensions and object areas and frame, the position of the object on the background plane, the time factor; -when processing the image: the type of the operator of the selection of the boundaries of the object, the allowable amount of error when approximating the contour of the part, the number of additional points when approximating the edges. Fig. 1. The main components of the MV: 1 -object of interest; 2 -lighting; 3 -optical system; 4converter "lighting-signal"; 5 -computer; 6 -mathematical software; 7 -analog data input device; 8external devices; 9 -actuators; 10 -the driver.

Formatting the
Diagnostic parameters (DP) -parameters, the value of which indirectly characterizes the technical condition of the object we choose: pixel; focal length/distance to the lens; field of view (FOV) -the area that can be seen by the MV device; working distance (WD) -the distance between the lens and the object; depth of field (DOF).
The Ministry of Health requires high reliability and quality work in a variety of conditions. Development and further use of MV is impossible without debugging, testing and diagnostics of compliance of parameters and characteristics of MV with operating conditions. At this stage, a serious problem is obtaining test data and organizing the testing and diagnostic process.
To solve the problem of testing, debugging and diagnostics of the MV, it is proposed to use virtual polygons built using the methods of procedural computer graphics [13]. The virtual landfill (VL) for testing consists of three main parts: the VL generator, the interface between the MV and the virtual unmanned vehicle (UC), the UC simulator ( Fig. 2): VL generator is the most important subsystem. Its task is to build and paint a synthetic model of the road. The generation process is controlled by the operator by setting a small number of parameters that determine the appearance of the road. The road is built by combining three levels of detail: a global map of vertical and horizontal low resolution (built procedurally on the basis of parameters), a three-dimensional grid (built on a global map of vertical and horizontal and three-dimensional "smart noise" using the marching cubes) and a high-frequency fractal noise component (generated and added during the output process). VL is painted procedurally, on the basis of characteristics of roughnesses and average slopes of separate sites of a road surface. During generation, there is no repetitive texture patterns, a variety of generated surfaces and their similarity to real surfaces.
The interface between the MV and the virtual UC is provided by transmitting control messages (requests for photos from cameras and UV control signals) from the MV and receiving information messages (photos with debugging information, data on the actual position and orientation of the UC) from the virtual UC. Fig. 2. The structure of the elements of VL: 1 -generator VL; 2 -interface; 3 -UC motion simulator; 4 -device for entering evaluation criteria; 5 -simulator (processor-computer); 6 -device for entering parameters and characteristics; 7 -output device of the obtained results.
The UC simulator provides a simple physical model of UC movement and the ability to visualize the test site from the positions of the cameras located onboard the UC. This takes into account the parameters and characteristics of the elements of the EP, their importance on the basis of selected criteria.
The use of a virtual test site allows you to get the following advantages over real sites: 1. Speed and low cost of obtaining test data.
2. Ability to obtain test data on various types of road surfaces: from asphalt to ground. 3. Ability to determine the accuracy of the MV, because of the exact surface of the road. 4. Ability to interrupt the work of the Ministry of Health at the time of the error and quickly find its cause.
5. Repeatability of test results. Development of VP for testing and diagnostics of MV allows to estimate the size of the influence of casual errors of external parameters of the environment and internal parameters of elements of MV on quality of work of MV, and also to model the form of space of diagnostic parameters, to define its sizes and to analyze characteristics of the distribution of errors [14].
In systems of passive-type, light energy from third-party sources are used for the transmission of measuring information, not related to functionally with the measurement system. In optoelectronic passive systems such as receivers of the measuring information, machine visual cameras (MVC) are used, the accuracy of the perception of measuring information which significantly depends on the perfection of the optical part. Although modern lenses are projected using computers for complex calculations and modeling, even with such technology it is impossible to completely eliminate all distortions. Measuring geometric disfirms image devoted to work [15,16]. Conditional publication can be divided into two groups.
The first group of works is devoted to the diagnosis of optical systems in order to determine distortions in the calculation of optical systems and the assessment of their quality. To compensate for distortions, hardware compensation methods are used, which leads to complications of optical systems and their significant emergency.
The second group of works is devoted to the calibration of MVC to determine the distortions and their further compensation (including the distortion of the image).
The most promising method is the method of calibrating the camera in the pictures of a special test object [17]. The essence of the method is to obtain calibration coefficients that take into account the effects of all systematic distortions existing in real shooting. And further program compensation of distortions based on a mathematical model describing distortion.

This calibration method is the most widespread now because it is easily implemented in practice
When calibrating the camera, test objects are used, both flat and spatial. By distorting images of test objects formed arrays for measuring the chamber distortion [16]. Most often used in calibration test objects are presented in Fig. 3.
The test object A is used to adjust promising distortions and distorts, in the process of calibration of the binding is carried out to the corners of the square. In Figure B, the test object is used to correct the distortion, which the binding to the centers of the points. There are modifications of this test object that are characterized by the form of points. In Figure C, the test object for calibration systems is used to measure angular variables. Binding during calibration is carried out to the corners of triangles. The test object D is an array of squares that can be used to obtain coefficients of several types of distortions (promising distortions, distortion).
On test objects A and B, it is much easier to carry out the binding process, on test object B an attachment to the points of points, the quality of the calibration process will depend on the accuracy of the center of points. a b c d
Despite the variety of polynomials, all of them are based on two basic ideas. The first is the integral systematic error of DX, DY, described by the polynomial, is presented as the sum of members of individual systematic distortions (distortion, deformation, etc.), for example, in works [18,19].
The second idea is that the presentation of a polynomial of an integral error does not bind to individual types of distortions. Integral error is described by static polynomial, for example, in works [20,21].
The most appropriate calibration method for the MV camcorder is a method based on the classical method of Tsaa [22]. This calibration algorithm involves performing multiple operations: -conducting internal calibration to find the matrix АIC -a conversion matrix from the image coordinate system into the camera coordinates; -adjustment of linear deviations and distortion; -external calibration for finding the matrix АWC -matrix of conversion from the camera coordinate system into the system of coordinates UV; -development of a direct task of the kinematics of the UV movement for finding the matrix АBC -matrix of transformation from the UV coordinate system into the baseline coordinate system.
To find the matrix АIC you need to determine the location of the image coordinate system relative to some constructive cell element [23].
The advantage of this method is the possibility of calibration with the camera installed on the Camera body. The use of the UV housing as measurement base increases the accuracy of the camera calibration by eliminating the errors of additional kinematic fastening chains (turns, movements).
In the process of internal calibration as the beginning of the coordinates of the basic system of coordinates XBYBZB is selected a calibration base. The beginning of the coordinates of the XCYCZC camera is collected to it. The image coordinate system binds to the calibration field (Fig. 4).
The image of the image coordinates is the projection of the central calibration field label on the plane А [26]. When calibrating the distance from the center of the lens to the central tag template is selected equal to the distance from the center of the lens to an object of interest (for example, a column of a petroleum curb side by side). The start of the coordinate system is the projection of the central label of the calibration field on the plane А. Next it is necessary to determine the internal cameras parameters: the focal length F and the coordinates of the main point (u0, v0) [27].
At the time of the second calibration phase, a known mutual location of the image coordinate system and a camera coordinate system that allows you to compare the image model and a real image received by the camcorder.
The calibration algorithm for diagnosis is proposed -the sequence of execution of inspections included in the diagnostic test, and the rules for processing the results of inspections in order to obtain a diagnosis -information about the object of diagnostics, which allows localization of the system's malfunction (assess its technical condition), or to identify the reason for its incapacity. Based on the analysis of diagnostic parameters or symptoms (forms of manifestation of the deviation of the diagnostic parameter from its permissible values).
The algorithm of internal calibration of the chamber contains elements (Fig. 5): 1. With the calibration device pins, the camera is installed above the gauge template so that the optical axis of the chamber and the central layer of the template coincided.
2. Multiple template shots are made.
3. There is a recognition of a calibration template label on the resulting image and calculating their pixel coordinates in the sampling coordinate system.  Angular orientation of the coordinate system of the image relative to the object coordinate system is a matrix of guide cosines of elements of external orientation of a picture, which is determined by the formula: (2) Provided the colonality of the axes of the image coordinate system and the coordinate system of the picture of the orienteering matrix B gaining the appearance: Then the Kolinearity equation is written as follows: Calibration is carried out according to the least squares method. At the same time, a minimum is achieved by the union of the Kolinearity Center for the camera design, points of the object and corresponding points (Fig. 6).
From equation (4) it follows that the arguments of functions ex and ey is a value u0e, v0e and fe,and function ex and ey tend to zero with equal projected and actual values u0, v0 and f. Thus, internal calibration algorithm reduces to the problem of finding the minimum functions ex and ey method of least squares. In vector corrected parameters include u0, v0 and f. Estimated location u0, v0 is located in the center of the image and is defined in a coordinate system image formulas: Translate the coordinates of the projected point of gauge fields with coordinates in the image coordinate system images using dependence: where x, y -coordinates of points in the shooting coordinate system obtained from the recognition of points of the calibration field; lp -the length of the pixel, μm.
Further, minimization of ex and ey functions to provide an average deviation of the position of control points from the given and obtaining necessary accuracy.

Conclusions
Considered generalized approach to the diagnosis of MV unmanned vehicle and formulated general requirements for diagnostic MV UV proposed diagnostic parameters to assess the serviceability of the system MV. To solve the problem, testing, debugging and diagnostics MV proposed use virtual polygons constructed using procedural methods of computer graphics. After the diagnosis of cameras is their calibration images special test facility.