Ghost-removal image warping for optical flow estimation

Traditional image warping methods used in optical flow estimation usually adopt simple interpolation strategies to obtain the warped images. But without considering the characteristic of occluded regions, the traditional methods may result in undesirable ghosting artifacts. To tackle this problem, in this paper we propose a novel image warping method to effectively remove ghosting artifacts. To be Specific, when given a warped image, the ghost regions are firstly discriminated using the optical flow information. Then, we use a new image compensation technique to eliminate the ghosting artifacts. The proposed method can avoid serious distortion in the warped images, therefore can prevent error propagation in the coarse-to-fine optical flow estimation schemes. Meanwhile, our approach can be easily integrated into various optical flow estimation methods. Experimental results on some popular datasets such as Flying Chairs and MPI-Sintel demonstrate that the proposed method can improve the performance of current optical flow estimation methods.


Introduction
The concept of optical flow, which describes the apparent motion between images or video frames, arises from the studies of biological visual systems. Optical flow has been widely used in many computer vision applications, such as video segmentation, three-dimensional reconstruction, and video compression. The first available optical flow estimation method was proposed by Horn and Schunck [1] in 1981. Although there is abundant literature on this topic, accurate estimation of optical flow from real-world still remains a challenging problem.
The methods of optical flow estimation can be roughly divided into two categories. The first one is to estimate optical flow directly from the images in original size, the other category is based on the coarse-to-fine schemes. Both categories are popular and have massive fresh achievements. Full Flow [2] optimizes a classic optical flow objective over the full space of mappings between discrete grids without descriptor matching. Bailer et al. [3] proposed a dense correspondence field approach with much less outlier prone, which achieves the best results on MPI-Sintel dataset. Flownet, the first method ， using convolutional neural networks (CNNs) for optical flow estimation was introduced by Dosovitskiy et al. [4]. The coarse-to-fine scheme is a classic strategy due to its capacity to handle large motions [5], and later it becomes popular again because it can combine naturally with deep learning [6].
Be it conventional methods or latest approaches based on convolutional networks, image warping is the essential ingredient in coarse-to-fine optical flow estimation schemes. The concept of image warping for optical flow estimation was proposed by Lucas and Kanade [7]. Sun et al. [8] verified that the spline-based bi-cubic interpolation is better than the MATLAB built-in function--interp2 which is the widely used by many researchers. However, in the warped images, the existing work neglects the ghost regions, which may result in erroneous information propagated in the coarse-to-fine optical flow estimation scheme. For instance, we can clearly observe the ghosting artifacts in the examples shown in figure 1. Therefore, high-precision warped images without ghosting are necessary in the coarse-to-fine optical flow estimation schemes. In this paper, we propose a novel image warping strategy to handle the ghosting artifacts in the warped images. With the optical flow information, we first locate the ghost regions in the warped images. Then, the corresponding pixels in unwrapped images are determined according to the optical flow. Finally, the ghost effect is eliminated using an image compensation technique. In summary, the key contribution of this work are: 1) a ghost-removal image warping strategy which can be easily integrated into traditional and CNNs-based methods for optical flow estimation. 2) some insights about the importance of image warping for optical flow estimation.

Method
Image warping is an essential part of the coarse-to-fine spatial pyramidal structure for optical flow estimation. What's more, the warped images are directly related to the quality of optical flow estimation. We briefly introduce the coarse-to-fine optical flow estimation scheme in the first subsection. In the second subsection, we review the existing methods and propose our ghost-removal image warping method.

Coarse-to-fine optical flow estimation scheme
Optical flow is the pattern of apparent motion of objects in adjacent images, I1 and I2. In general, the first step is to construct the image spatial pyramids for both images respectively. We first estimated at the coarser resolution levels of the pyramids, is successively refined by taking into account the evidence of the finer resolution levels later. The overview of the coarse-to-fine optical flow estimation algorithm is presented in figure  2.
To be specific, we construct the spatial pyramids with n levels for both images firstly. We denote the k-th levels of pyramid of Ii as Ii k , i ∈ {1,2}, k ∈ {1,2,...,n}. It should be point out that the bottom levels of pyramids, which labeled I1 1 and I2 1 , are the images in original size. Then, we estimate the optical flow Fn at the coarsest resolution level using I1 n and I2 n . Then the warped image ω (I2 n-1 , Fn) is obtained based on the optical flow Fn. Specifically, the result Fn needs to be interpolated before image warping. Next, with I1 n-1 and ω(I2 n-1 , Fn), the optical flow Fn−1 is estimated successively. Finally, the optical flow of every pixel in I1 against I2 can be effectively estimated after finite times of iteration.

Fig. 2.
Illustration of the coarse-to-fine spatial pyramidal optical flow estimation scheme.

Ghost-removal image warping method
Previous image warping method has neglected a disastrous problem-the ghosting artifacts in the warped images. The primary reason for ghosting is the warping image changed from I1 to I2. The occluded regions, which are covered by the occlusion regions, are virtually invisible in I2. Without take into account this problem, the pixels of the occluded regions are replaced by those of the occlusion regions in the warped images wrongly. More specifically, the warped images record the moment before the motions, at which the occlusion regions and the occluded regions are separate. When the two regions coincide in I2, only the occlusion regions are retained. However, the reason of ghosting in the warped images is that the occlusion regions and the occluded regions correspond to the same positions in I2. Based on above descriptions and discussions, we propose our method which can discriminate and eliminate the ghosting artifacts in the warped images effectively.
Based on the statistics [9] of Flying Chairs and MPI-Sintel datasets, it is verified that the displacements of the foreground are larger than the background. Meanwhile, the occluded regions only belong to the background in most cases. As mentioned above, when the occlusion occurs, two or more positions in the warped image will trace back to the same pixel in I2. According to the transformation matrix, it is easily to discriminate whether the positions in the warped image point to the same pixel. And the next step is to distinguish which position belongs to the foreground. In view of above-mentioned reasons, when the occlusion happened, only the position with larger displacement, meaning it belongs to the foreground, retains the corresponding pixel in I2. Under this circumstance, the positions with small displacement are discriminated as the occluded regions, which are the ghost regions in most cases.
After the steps of mentioned above, we obtain the warped images in which the ghost regions are blank. The ways to fill the blank areas are closely related to the final quality of the warped images. Extensive experiments show that filling with 0 is not a wise choice. Since 0, likes the value of the occlusion regions to some extent, is not the correct value of pixels in the blank areas. We propose two image compensation strategies for different situations. The first is utilize the neighbour pixels around the blank areas to filling the ghost regions, while the second one is filling the ghost regions with pixels in the corresponding positions in I1. Details of the comparing experiments will be given in the next section, according to which filling with the pixels in I1 is the optimal strategy for the optical flow estimation. It should be point out that I1 is not the additional information for optical flow estimation since both images, I1 and I2, are required during computation.

Experiments
Image warping methods are the essential ingredient in the coarse-to-fine optical flow estimation schemes. Two optical flow estimation algorithms are used as the basic methods in this section. The first one is pyramidal LK optical flow (PLK) [10], and the second one is LDOF [5]. Replacing the traditional image warping methods with our ghost-removal image warping method are defined as our methods, called GrWarp-PLK and GrWarp-LDOF, respectively.

Image compensation strategies
In this subsection, we experimented with three image compensation strategies to filling the blank areas. One is filling with 0, another is utilizing the neighbour pixels in I2, and the third is using the corresponding pixels in I1. For instance, the differences among three kinds of strategies mentioned above are shown in figure 3. By comparing with the traditional method, the first strategy, which is filling with 0, is not an acceptable choice, since 0 is not the correct value of pixels in the blank areas either. Filling with the neighbor pixels is similar to the image smoothing in some ways, which may reduce the possibility of mismatching in the ghost regions to some extent. The last strategy obtains the most accurate warped images, meanwhile achieving the best results. We can observe the significant differences on the borders of motion, which are the regions of ghosting, in the examples shown in figure 3. It is clearly that the ghosting artifacts directly influence the results. Meanwhile, our method eliminates the ghost regions effectively and achieves the best results.

Flying Chairs dataset
As discussed above, our method improves the results significantly. We have compared performance and quality with the basic methods in this subsection. Figure 4 illustrates the differences between the results produced by our method and those of the baseline method.
For each method, we calculate the average of endpoint error (AEE) for the test data.

MPI-Sintel dataset
Compared to the Flying Chairs dataset, the MPI-Sintel dataset is more difficult because of the complicated motion. Due to space limitations, we use temple 3, which including the self-occlusion and the occlusion, as examples in the following. The differences between the results produced by our method and those of the baseline method are shown in figure 5. As mentioned above this sequence is too difficult for the PLK. However, our method can reduce the endpoint error while using the rough optical flow estimation algorithm. It verifies our method still works at the coarse resolution levels of pyramid with the coarse optical flow results. As shown in figure 5, the GrWarp-LDOF has a great advantage in selfocclusion, such as the wings of bat in the first row and the clenched claws in the thrid row, and occlusion, such as the bat in the second row.

Supplementary details
We also compared performance and quality with the basic methods on the Middlebury and KITTI datasets and obtained the comparative results on both datasets. It is generally known that the Middlebury dataset is a classical optical flow benchmark, and it almost only contains small displacement motions. Without large displacement, our method does not present obvious improvement over the basic methods since the ghost regions are small. Due to space limitations, we will summary the results in table 1 without further details.

Conclusion
This paper presents an improved ghost-removal image warping method for optical flow estimation. We provide some insights about the importance of image warping, meanwhile, discriminating and eliminating the ghosting artifacts in the warped images effectively with utilizing the optical flow fully. Removing of the ambiguity of pixels of the ghost regions, improve the accuracy of optical flow estimation significantly. Furthermore, we hope that our ghost-removal image warping method, which can be easily integrated into the CNNsbased approaches, will support further advances.