Image inpainting using a neural network

The paper describes a new method of two-dimensional signals reconstruction by restoring static images. A new method of spatial reconstruction of static images based on a geometric model using a neural network is proposed, it is based on the search for similar blocks and copying them into the region of distorted or missing pixel values.


Introduction
Estimation of true pixel values of images to some extent is necessary for most digital imaging tasks.This problem is especially important for the automatic processing of images obtained in photosensitive matrices in the digital photo-and video cameras and computer vision systems.Currently, there are various methods of digital processing aimed at solving the problems of recovering partially lost image areas, but for their effective use, considerable amounts of prior information about the useful image are needed.
In connection this, the reconstruction of images is an important area of application of modern digital computer technology to obtain a reliable estimate for visual and especially automatic analysis.

Method of reconstruction
At the first step, for each pixel of the boundary j i S ,  using the inversion method the shape of the domain for similarity search is adaptively determined by combining two adjacent homogeneous sub regions in the direction of the maximum gradient [3,4,9].
The second step calculates the priority value ) ( S P  for each pixel value of the boundary, which consists of two factors (Figure 1): where:  -normalized multiplier, which for eight bit images is equal to 255 [1,4].
First, it is assumed that the value of the confidence coefficient С for pixels from the area S is equal to 1, and for the domain  equal to 0 Calculation of the priority using the expression (1) allows you to give more weight to the pixels that are on the differences in brightness (boundaries), thus restoring them in the first place.Accounting for the confidence factor At the third step, we determine blocks in the area of available pixels S for which the Euclidean metric is minimal (Figure 3): . min The number of similarity blocks R is determined using the confidence interval [6,11]: where:

R h , 1 
;  -significance level.The pixel values in the region  adjacent to the pixel with the highest priority p are reconstructed by averaging the corresponding pixels from chosen areas in the area of available pixels S using a neural network, in particular, a multi-layer perceptron.
The coefficient of trust С for restored pixels is assigned to the current value ) ( p C .After that, the procedure of priority correction and search of similar areas with subsequent replacement is repeated.
At the training stage of the neural network, preprepared data was fed to the input: a block with random coordinates 9x9 in size was allocated to the image, then the central pixel was removed, and five most similar blocks were found on the whole image comparing them by MSE.Then the procedure was repeated on 35 images, and 100,000 blocks were obtained, they were used as a training sequence for this network.
This network contains three layers: the first layer contains 20 neurons, the second layer also contains 20 neurons, the third layer contains ten neurons.The network created ten inputs, 5 of them were fed only the central pixels in the blocks found, and the other five inputs fed the MSE of these blocks.

Study of the effectiveness of the proposed method
To test the effectiveness of the proposed reconstruction method 3 test images are selected.Figures 6-8 show the results of the proposed method.Analysis of the results shows that the proposed method allows you to effectively restore lost blocks, allows you to "correctly" restore the boundaries of the image.It should also be noted that the proposed method does not lead to blurring of texture and structure when restoring large areas with lost pixels

Conclusion
In conclusion, we can draw the following conclusions.The proposed method will provide a smaller recovery error than traditional methods by an average of 15-20%.The effectiveness of the new approach is shown in the examples demonstrating the effectiveness of the algorithm when restoring test images when objects are deleted.

Fig. 2 .
Fig. 2. Adaptive selection of the shape of the region to search for similarity.

Fig. 4 .
Fig. 4. Original 9x9 block with remote pixel and five similar blocks.

Fig. 5 .
Fig. 5. Structure of the neural network training stage.As an output, there were the central pixels of the original blocks.The TRAINLM training function is employed as network training function that modifies the values of weights and biases by the Levenberg-Marquardt optimization method.The activation function is TANSIG, a sigmoid function in the form of a hyperbolic tangent [5].

Fig. 6 .Fig. 7 .Fig. 8 .
Fig. 6.The result of the proposed method:a) the original image, b) the image with the mask of the distorted pixels, c) the image restored by the proposed method.