Multidirectional motion estimation algorithm for frame rate up-conversion

In this paper, we propose a new motion estimation algorithm for motion-compensated frame rate up-conversion. To reduce computational complexity, it constructs a hierarchical structure using Gaussian image pyramid and unidirectionally estimates motion vectors (MVs) between two original frames at the top level. Since the accuracy of the motion vector may deteriorate at low resolution, each block has five candidates for motion vector so that disadvantages of hierarchical motion estimation are reduced. After motion estimation (ME), MVs of bottom level are corrected based on the sum of absolute difference (SAD) values of the blocks. All unidirectional MVs are assigned to bidirectional MVs, and new frames are interpolated using these MVs. As a result, proposed algorithm shows up to 1.3 dB increase in PSNR compared with conventional algorithm with low computational complexity, and it shows clearer images than conventional algorithms.


Introduction
Frame rate up-conversion (FRUC) is a technique which interpolates frame by inserting new frames between originally existed frames.In the early days of technology, frame interpolation was performed using frame repetition or simple averaging of two frames.However, this causes image quality deterioration.In recent years, motioncompensated FRUC (MC-FRUC) algorithm has been developed [1][2][3].This algorithm generally has three steps: motion estimation (ME), motion vector smoothing (MVS), and motion compensated interpolation (MCI).
ME is a process of finding motion of each object using original frames, and motion vectors (MVs) are obtained in this step.Depending on the direction in which the motion is predicted, ME is divided into two major categories: unidirectional motion estimation (UME), bidirectional motion estimation (BME).Motions are predicted unidirectionally in the viewpoint of the original frame, and overlapped areas and hols occur in the interpolated frame.BME bidirectionally predicts motions in the viewpoint of the interpolated frame, so there are not holes and overlapped areas.However, it can find false MVs since there in no information about the current reference block.
Basically, ME has a high computational complextiy, so for real-time processing of high-resolution video, it is essential to reduce the complexity.Several studies proposed low-complexity FRUC algorithms; [13] used Gaussian pyramid structure and Laplacian pyramid structure in ME step, and [15] used subsampling of the original image for ME.These algorithms, however, have reduced the accuracy of MVs due to the resolution changes of the original images.In this paper, we propose a multidirectional motion estimation (MME) algorithm to reduce complexity and maintain the accuracy of the motion vector.
For the MVS process, [4] used mean and variance of neighboring MVs and weighted sum of these motion vectors replace the current motion vector, and in [6], current motion information is used to correct false motion vectors.However, when the current block is at the boundary of an object or when the current motion vector is not correct, these algorithms can cause an error.In this paper, an adaptive global MVS algorithm is proposed to overcome these limitations.
The remainder of this paper is organized in the following order; secsion 2 reveal the proposed algorithm in detail, section 3 shows experimental results compared with other algorithms, and section 4 concludes this paper.

Proposed algorithm
Fig. 1 shows overall flowchar of the proposed algorithm.Two original frames form a pyramid structure using [7].The unidirectional motion estimation is performed at the top level, in which five candidate positions are designated.At the lower levels, the candidates are modified by small search range motion estimation around each position.Once the MVs of the all blocks are determined, these MVs are modified according to the sum of absolute difference (SAD) value.Final MVs are converted to the bidirectional MVs using [5].As a result, a new frame is interpolated in the MCI step.The details of the proposed algorithm are described in the following subsections.

Motion estimation
Prior to the motion estimation process, each frame forms an image pyramid as shown in Fig. 2. As shown in Fig. 3, After top level processing , each candidates points are updated using ME with small search range.The size of this small search range is determined based on the SAD value of each candidates, and it is given by : where ܵ is a SAD value, and ܴܵ ௦ is the size of the search range.ߚ means maximum size of the search range, and ߙ is weighting factor.We set ߚ = 50 , ߙ = 0.0007689.At the bottom level, current block decides final motion vector which has the smallest SAD among the candidates

Adaptive global motion vector smoothing
When the ME process of one frame is ompleted, the MVS is performed to reduce an erorr generated in the ME.where ‫)ܤ(ܴ‬ indicates reliability of the current block ‫,ܤ‬ and ܶℎ means threshold value.
To determine threshold value, we used recursive algorithm in [8], and it is shown in Fig. 4.

Experimental results
Objective and subjective evaluations are performed in this section.For the objective ecaluation, peark signal to noise ratio (PSNR) is used, and complexity of each algorithms are considered.High efficiency video coding (HEVC) standard test sequences were chosen for the test so that high resolution videos are includes in the experiment.Block size is set to 16×16, and search range of the top level is set to ±12 pixels.
Table 1 shows PSNR results compared with conventional algorithms [15,4,10].In all sequences, the proposed algorithm shows the highest PSNR value; up to 1.17 dB compared to [15], up to 1.34 dB compared to [4], and up to 1.05 dB compared to [10].Average values also shows that proposed algorithm has highest PSNR value.Table 2 shows computational complexity of the algorithms.ܰ means block size (16), ܴ means search range (25), and ܴ ௦ denotes small search range (2).The small search range of the proposed algorithm is adaptively determined, but approximated to an average value of 2. Ratio represents the relative amount of computation when the computation complexity of the proposed algorithm is 1.It can be seen that the proposed algorithm has less computational complexity than the other algorithms.[15] does not make much difference; however, when MV is modified at lower levels, there are many overlapping parts, so the actual computational complexity of the proposed algorithm is smaller than 1.
Fig. 5 shows results images in BasketballDrive sequence.In the square part, the result image of the proposed algorithm is clearer than the other algorithms.

Conclusion
In this paper, ME algorithm and MVS algorithm for the MC-FRUC were proposed.In the ME process, multidirectional motion estimation was proposed to find accurate MVs with low computational complexity, and adaptive global motion vector smoothing algorithm was proposed in the MVS step, so the error of the MVs is corrected by adaptively determining the threshold value in frame unit.As a result, the proposed algorithm shows high PSNR in objective evaluation and clear image in subjective evaluation while keeping the computational complexity low.
This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT and future Planning(NRF-2015R1A2A2A01006004)

Fig. 3 .
Fig. 3. Four regions of the search area.Proposed algorithm utilizes SAD values of final MVs to determine which MV is incorrect one.This process is shown as follows: