Gradient-wise search strategy for blind image deblurring

Blind image deblurring is a long-standing challenging problem to improve the sharpness of an image as a prerequisite step. Many iterative methods are widely used for the deblurring image, but care must be taken to ensure that the methods have fast convergence and accuracy solutions. To address these problems, we propose a gradient-wise step size search strategy for iterative methods to achieve robustness and accelerate the deblurring process. We further modify the conjugate gradient method with the proposed strategy to solve the bling image deblurring problem. The gradient-wise step size aims to update gradient for each pixel individually, instead of updating step size by the fixed factor. The modified conjugate gradient method improves the convergence performance computation speed with a gradient-wise step size. Experimental results show that our method effectively estimates the sharp image for both motion blur images and defocused images. The results of synthetic datasets and natural images are better than what is achieved with other state-of-the-art blind image deblurring methods.


Retalted work
In many applications, ranging from computer vision and pattern recognition to machine intelligence, the demand for image deblurring is widely encountered as a prerequisite [1]- [3]. Often a blurred image needs to be restored to improve its sharpness before further steps, such as segmentation, detection, and classification [4]. Typically, the image deblurring problem can be classified into two categories: non-blind, in which the blur kernel is assumed to be known [5], and blind, in which the blur kernel is unknown [8]- [20]. For nonblind image deblurring (NBID), it is to deblur an image with a known blurring kernel. In a real application, it is usually difficult to know the blurring kernel in advance. Therefore, blind image deblurring (BID) is much more required than NBID. The goal of the BID is to recover a blur kernel and a sharp latent from a blurred input, which is a typical ill-posed problem. Solution for the ill-posed problem can be categorized into direct methods via some effective regularization [10], [11], [12], [17], [20], [21] and iterative methods via some updating and stopping rules [15], [22]. Recently, learning-based BID approaches have drawn much attention due to the satisfactorily restored qualities [16]. However, note that the effectiveness of the learning-based algorithms highly depends on the training image database, and this kind of algorithm generally requires a high computational complexity. Most recently, significant progresses have been made in [17], [18]. In order to make the blind deblurring problem well-posed, image prior, and blur kernel model exploitation is the key to most effective methods. Well-developed image priors include salient intensity prior [15], hyper-laplacian prior [7], gradient prior [24], surface-aware prior [18], to name just a few. Meanwhile, the blur kernel model includes the depth variation model [26], Forward motion model [27], non uniform model [28]. Most of these methods exploit the image prior and blur kernel model under the Bayesian framework [29]. Even though incorporating regularization to exploit image prior and kernel model helps to increase the probability for achieving a good local solution, the ideal deblurring result cannot be easily acquired.
In this work, we focus on iterative methods by designing a new strategy to improve the comprehensive effect of the BID process. In [14], the alternating direction method of multipliers (ADMM) is employed to solve the proposed model, which has high computational complexity than the general iterative method. However, its applications are constrained due to poor convergence. In [32], [33], Quasi-Newton (QN) method and conjugate gradient (CG) method were proposed to solve the image reconstruction problem. However, the step size was calculated by an inexact line search method that uses a series of iterations to test a suitable value [34]. It requires a large amount of storage and computation and lacks adaptiveness and robustness, performing less satisfyingly with various regularization terms and images, which is less suitable for further application. Therefore, we need to design a more efficient iterative algorithm for BID with suitable step size, reasonable computation cost, and quick convergence speed.

Contribution
Based on the above analysis, we develop several strategies that are significantly different from previous works in the following aspects. a) A blind image deblurring method is proposed that in troduces a gradient-wise search strategy to calculate a proper step size under an improved BID framework. When dealing with different scenes, the step size adjusts automatically to the change of gradient. It takes less time to generate a more accurate solution with less computation cost. The adaptive iteration method effectively reduces ringing artifacts and increased the fidelity of the deblurred image. b) A more effective BID framework. It effectively solves the image deblurring model we proposed through combining the conjugate method, which embedded adaptive step size with bilateral total variation (BTV) regularization [30]. The solution space of the latent image is reduced via introduces the BTV regularization, which mainly represents the intrinsic properties of the original image. Furthermore, another useful regularization term can replace the BTV regularization to improve the effiffifficiency with the fundamentally prior information.
Extensive experimental results on blind image deblurring have been provided to evaluate the performance of the proposed method. Compared with other state-of-the-art methods, the results show that this method obtains a superior image quality with less computation cost; qualitative and quantitative comparisons also verify the value of our algorithm. . The clear image degrades into a blurred image under the impact of various factors. Therefore, the image degradation process is formulated as follows: G WB I N (1) where G , I , and Q represent the matrices of the blurred image, clear image, and the additional Gaussian noise acquired during image acquisition; represents the convolution operator, and B is the blurring kernel. We introduce the warping matrix W as one of the degrading factor considering the real application. Generally, assuming that all low-quality images are generated under the same condition, we can further simplify the model into G A I N where A is the total decimation matrix, which has taken all the degrading factors above into consideration. The BID aims to recover the original sharp image from the blurred and warped noisy image without information about the blurring kernel and warping process. We introduce the CG-based BID approach and infer our proposed gradient-wise search strategy. Besides, we adopt 2 L norm as the fidelity term and employ BTV as the regularization term, and a cost function is always utilized to measure the distance between the estimation and the observation as follows: In ( Note that the information related to shape and texture is primarily present in the Y component. We only dispose of the Y component of degraded image G , processing b C and r C components with bi-cubic interpolation. Then, we convert the estimated image to the RGB color space. From the discussion above, we rewrite (3) into I , we use the CG-based method to solve the problem iteratively, and the procedure will be described detailedly in the next part.

Proposed algorithm
After the formulation of the BID model in (5), we can see that estimating a sharp image is equivalent to finding the solution to minimize the cost function Y F I . For this minimization problem, we use the CG-based method to obtain a numerical solution, as follows: where j d and j S represent the descent direction and the step size in the jth iteration, respectively. The current descent direction depends on the former descent direction and the current negative gradient direction. Setting a proper step size plays a vital role in the convergence of the CG method. Since an oversized step size will lead the iteration to a divergence, whereas an undersized step size will cause a slow convergence. Therefore, the optimal step size varies in different scenes, which is challenging to preset appropriately. The Line Search [34] is a widely-used strategy that derives j S by minimizing the objective function as ( ) arg min Although an accurate step size can improve the convergence, the line search procedure cannot quickly solve the large-scaled minimization, and computation cost is expensive. Therefore, we design a gradient-wise search strategy that considering accuracy and computation cost. Firstly, we calculate the first-order derivative of Y F I in (5) as: It is clear from the above equation that S still exists on both sides of (13). A general solution is to design an iterative process with S and set a small value for initialization. Different initial values of S make little difference to the calculated step size due to the large fidelity error at the beginning of the iteration. Nevertheless, because of the unideal quadratic property of the BID model, such as the red curves shown in Fig.1, employing the step size in (13) cannot generate a stable convergent solution. In reality, the initial input is generated by bi-cubic interpolation, which is with low sharpness and far from the final optimal solution. From (5), we note that the 2 L norm fidelity error is much larger to the BTV regularization error, and the fidelity term shares more weight in early iterations. Therefore, the total error becomes more quadratic, such as the part of the red curve after about 10 iterations shown in Fig.1. However, with the increasing of iterations, fidelity error has been corrected continuously as well as the restored image becomes sharp. That is, the regularization term shares a heavier weight, whereas the fidelity term has a lighter weight and the objective function increasingly non-quadratic, such as the red curve between 10 and 30 iterations, as shown in Fig.1. Besides, consider that the BTV regularization term is the sum of the 1 L norm errors, whose gradient change quickly at some points. Hence, (5) based general line search strategy cannot converge with an incorrect step size, which means equation (13) has to be modified to lead a more stable solution. Intuitionally, (13) is reformulated as Along with the increasing of ieration times, E can prevent S from becoming oversized. Then, we calculate the step size according to (14) and substitute it into the CGbased BID method. In the experiment, motion blur and defocus blur are designed to simulate the actual situation to illustrate the effectiveness of the method. The image baby from our synthetic dataset, which is shown in Fig.3 is adopted as an example. We process images with ranged values of E and use root-mean-square error (RMSE) to evaluate the quality of the sharp image. The convergence performance estimated is depicted in Fig.1.  From Fig.1 we note that the smaller E is, the faster of algorithm convergence in the earlier stage of iterations, but it cannot stabilize the solution in a later period. On the contrary, a larger E maintains the final solution stable, but convergence speed is slow. The 15th iteration is used as the cut-off point. Before, it is monotonically decreasing for all E , but when the E is smaller, RMSE decreases faster and the algorithm converges. For example, when E = 10 6 , its convergence rate is significantly lower than that when E = 10 2 . After that, the smaller the beta, the more unstable it is and tends to diverge. Our ideal convergence tendency is to converge quickly in the early stage and to stabilize quickly in the later stage. Consequently, to have the fast convergence property and obtain a final stable solution, E needs to change with iterations going on, and it should be proportional to the number of iteration. Conveniently, for illustrating the property of E , we introduce a parameter [ to express divergence degree, which is related to the 2 L norm of descent direction. It depends on the gradient difference between the last iteration and this iteration.  Table I, and the curve of these positive factors is depicted in Fig.2. To facilitate comparison, we normalize all the modified functions and limit the range of all functions to [0, 1].
The whole procedure of the proposed gradient-wise step size search strategy is described in Algorithm 1, and Algorithm 2 is the pseudocode of we proposed algorithm.

Implementation details
In this section, we introduce our implement details method ically including datasets and compared methods, parameters setting, and quantitative evaluation metrics. All experiments are coded in MATLAB R2016a and running on a Core i5-4210U @2.4 GHz CPU with a 4GB RAM. We conduct our experiment on our synthetic dataset, which is shown in Fig.3. The clear images from the dataset of Set5&Set14 dataset, Pan's dataset [23] dataset and our real captured images.  The proposed scheme is compared visually and quantitatively with state-of-the-artschemes, including Wen et al. [19], Pan et al. [23], Pan et al. [24], Krishnan et al. [25] using different images. For synthetic scenes, the blurred image is generated and further adopted to estimated the sharp image. In the degrading process, warping the clear images by no more than 5 pixels, then the displacement deviation with L = 10, the angle of deviation with T = 45, to simulate the motion effect. Making a defocused blur image from the warped image that blurred by 5×5 Gaussian kernel with g = 1, which represents the defocus blurring.
Finally, Gaussian noise with n = 1.5 is added. We employ BTV regularization terms to our proposed strategy that the parameter of BTV is the same as the value in [49].
Meanwhile, the BTV regularization weight with J = 0.1. To evaluate and compare the performance of estimated image quantitatively, root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), and mean structure similarity (SSIM) are employed.

Results of synthetic images
In our experiment, the number of iterations is fixed to 15 and compares the estimated results both visually and quantitatively. We first show the results of defocused image and motion blurred images processed by our method and other methods in Fig.3 and Fig.4. It is seen that the red box region is the zoomed-in portion of the test image about three times larger, which can be observed that the proposed approach generates more sharp images than other algorithms while suppressing the ringing artifacts and eliminating additional noise. Among all of the competitive methods, the results show that the fidelity of our deblurred image is better than others. To make quantitative comparisons, we calculated the same quality index of different images and can see from the results that our method outperforms competing methods for quality metrics. Table.II shows the results of defocused image comparisons, and Table.III shows quantitative metrics of the motion blurred image.      Table.II and Table.III only list the specific data of the two images and the average value of the data set. We select the 10 images from the data set of SSIM and plot into a histogram for intuitive comparison, as shown in Fig.6. It is mentionable that our method reaches a higher precision among the synthetic dataset , obtains over 0.1 increase in SSIM compared with most other methods.

Convengence performance
We vary the number of iterations to study how the proposed method converges to the solution. RMSE and SSIM are regarded as the convergence indexes. What can be seen from Fig.7 is that our proposed method has a fast convergence performance with a remarkable decrease in RMSE and SSIM at 15 iterations. From Fig.8, for comparing the running time of each method, six images are selected from the synthetic dataset and processed in five methods. It is worth noting that the time used to calculate the step size is very little compared with the overall time of the algorithm, mainly because the step size does not design complex operations, but it can effectively accelerate the convergence of the algorithm. We notice that Pan et al. [23] needs the most time to process these images. On the contrary, our method has the least execution time.

Results of natural images
The results of the proposed algorithm on two real natural images are shown in Fig.9. seen that the proposed gradient-wise search strategy helps to make accurate estimation, such as the Chinese characters on the cover of the red book, and the leaf. There are slight ringing artifacts around corners and edges. A main reason is that the real natural images often have saturated pixels which interfere with the iteration process.

Conclusion
This work have proposed a gradient-wise step size search strategy for adaptive iteration. It is simple yet effective in selecting suitable step size from blurred image to clear image. Further, modified CG method is used to obtain the global op timal solution of objective function with BTV regularization. Finally, extensive experiments on both specific and natural images demonstrated that it not only can achieve state-of-the art deblurring quality, but also can improve the computational efficiency under the CG-based framework. In brief, in terms of both the restoration quality and computational efficiency, the proposed algorithm is superior to all the other algorithms compared in this work.