Medical image fusion based on variational and nonlinear structure tensor

Medical image fusion plays an important role in detection and treatment of disease. Although numerous medical image fusion methods have been proposed, most of them decrease the contrast and lose the image information. In this paper, a novel MRI and CT image fusion method is proposed combining rolling guidance filter, structure tensor, and nonsubsampled shearlet transform (NSST). First, the rolling guidance filter and the sum-modified laplacian (SML) operator are introduced in the algorithm to construct the weight maps in non-linear domain, then the fused gradient is firstly obtained by a new weighted structure tensor fusion method, and the fused image is firstly acquired in NSST domain, finally, a new energy functional is defined to constrain the gradient and pixel information of the final fused image close to the pre-fused gradient and the pre-fused image, experimental results show that the proposed method can retain the edge information of source images effectively and preserve the reduction of contrast.


Introduction
Image fusion is the technique which can combine two or more images into an image, eliminate redundant information and complement unique features between each other.which is widely used in the field of computer vision, the fusion of medical images is an important branch of it.Medical personnel will have more accurate diagnosis after mastering detailed features of human organs.The multi-scales decomposition algorithms are commonly used in medical image fusion, such as discrete wavelet transform (DWT) [1], nonsubsampled contourlet transform (NSCT) [2], and nonsubsampled shearlet transform (NSST) [3].In these methods, the source images are decomposed into different frequency components, and then different fusion strategies are adopted according to the characteristics of each frequency band, the final result is reconstructed by inverse multi-scales transform.Such approaches can effectively extract the features of the source images.However, they may produce artifacts in the process of image fusion, which results in the reduction of contrast.In order to solve the problems mentioned above, the fusion method based on guided filtering (GFF) is proposed [4], which has achieved remarkable results in eliminating artifacts and preserving the spatial consistency.Based on the idea of the guided filter, an efficient edge-preserving filter named rolling guidance filter was put forward by Zhang Qi et al [5], The method possesses nice properties of bilateral filter and gaussian filter, which can remove the small structure of original images and maintain the sharp image boundary.
Image fusion methods based on variational are proposed in recent years.The boundary, corner and other structural information can be regarded as a special spatial arrangement of gradient information [6].By adding fusion algorithm into the gradient domain, the fused image can retain the important features of source images efficiently.Considering the advantages of the rolling guidance filter and the gradient domain fusion method, an adaptive image fusion algorithm based on the rolling guidance filter and the structure tensor is proposed in this paper.The flowchart diagram of the proposed method is shown in figure 1. 2 Proposed method

Weighted structure tensor fusion
The structure tensor is used to represent the gradient information, and the fused structure tensor is firstly obtained in the nonlinear domain.For a set of source images Ik (k=1,2,3…), their structure tensor can be expressed as: Eq1. implies that all the gradients contribute equally to the fused gradient, which results in blurred images and losing of the details of source images.
In order to solve the problems mentioned above, we introduce a new weighted structure tensor fusion, and weight maps are constructed in non-linear domain, which combines the advantages of rolling guidance filter and SML operator, for simplicity, we use RGF to represent the rolling guidance filter, the weight maps can be calculated as: , and P determines the initial value of the weight map, the weighted structure tensor is obtained by the following formula： The remainder of the theoretical derivations are given by literature [6], so the fused gradient can be defined as：

Image fusion based on NSST
In the optimization model, we need to set a criterion for the pixel level fusion.Considering the shift invariance property of the NSST operator [3], we fuse the source images in the NSST domain.First, the source images Ik are decomposed into a low-frequency sub-image { � � } and a set of high-frequency sub-images { � �th }, then the average method is employed to combine low-frequency and maximum energy rule is adopted to fuse high-frequency, which can be expressed as follow: Finally, the fused image CF is constructed by inverse nonsubsampled shearlet transform.

Optimization model construction
In order to retain more gradient information and feature information in the fusion result, the energy functional constraints the gradient and the initial fused result close to pre-fused gradient VF and the fusion result CF, which can be defined as follow: Parameter � is used to control the balance between the gradient and the fused image, for simplicity: we use I1, I2 to denote the source images, and W1, W2 to represent the corresponding weight maps, then the gradient descent method is adopted to compute the optimization model: ( ) ( ) , and . The optimal valve of P is obtained by the iterative method.
3 Experimental results
The iteration number of RGF is set to 4, � � is 1.2 and � h is 0.05.The decomposition level of DWT and LAP is 4. Average coefficient is adopted for low frequency sub-images, and absolute maximum choosing scheme is used to fuse high-frequency coefficients for DWT, LAP, NSCT and NSST.Four decomposition levels with 2, 8, 8, 16 directions from coarser scale to finer scale are adopted for NSCT and NSST methods.The parameter for GF-based fusion method is given by the author [4].

Experimental results and discussion
Figure 2 show the fusion results obtained with different methods for "Med-1", figure 2(d) shows the result of DWT-based method consists of some obvious artifacts, figure 2(c) and figure 2(g) are acquired by LAP-and GF-base method, respectively, which show fewer artifacts, but contrast has been decreased, figure 2(e) shows the result of NSCT-base method, we can see some details are missed in the process of fusion.Figure 2(f) is obtained by NSST-based method, which also has the problem of contrast reduction.The proposed method has a better performance in visual effect, which retains more edge information and details, and enhance the contrast.Another experiment is shown in figure 3, as we can see from the marked region, some details of source images are lost, but our method preserve the details effectively.We don't show the three addition fusion results, they are similar to previous results.Quantitative comparison of different fusion methods' performance for five test images are given in table 1.It is obvious that the proposed method outperforms the other methods in all object criteria.The highest MI illustrates the proposed method obtain more details than other methods, highest Q AB/F show that more edge information is transferred tos the fusion result, and highest Q0 represents our result is more similar to the source images.

Conclusion
In this paper, a novel image fusion algorithm for MRI and CT images is proposed, we take advantage of rolling guidance filter and SML operator to construct weight maps in nonlinear domain, then obtain the fused gradient by weighted structure tensor and fused image in NSST domain, finally, a new optimization model is built to get the final fused result, experimental results show the proposed method has a good performance in both visual effects and object evaluation.