A New Method for Automatically Labeling Aircrafts in Airport Video

For the problem that the airport video monitoring could only provide the image information while the label information including the flight number is not provided, a new method for automatically labeling aircrafts in airport video through the fusion of video and ADS-B data has been proposed. First, the image coordinates of aircrafts will be obtained through the image tracking of video. Then, the homography matrix between two projection planes will be calculated with the four and above point and line correspondences selected from the airport map and video image, respectively to the map image coordinates into the map coordinates. Finally, the aircrafts in video can be automatically labeled through the fusion of image tracking data and ADS-B monitoring data. Because an image coordinate measurement error exists at the time of selecting points from the image, the resulting coordinate conversion error is derived and the impact of point correspondence geometric layout on mesh coordinate mapping error is analyzed. Experiments have been conducted based on the actual data of Chengdu Shuangliu International Airport. The results show that the method can automatically label aircrafts in video in an effective way.


INTRODUCTION
The movement of aircrafts is monitored through installing monitoring cameras in important areas such as runway, taxiway, gate position, and so in to help controllers to control the airport ground traffic in an effective way.Some large airports have a number of runways and the ground layout is complicated.There are often some dead angles invisible by visual inspection in the control tower.Then, the camera becomes an important means for controllers to intuitively grasp the ground movements.As a non-cooperative sensor, the monitoring video only provides the targets' image information and cannot provide the label information of targets.Controllers are also required to determine the label information of targets in video through viewing the flight plan, ATC monitoring screen, and so on.As a cooperative sensor, ADS-B (Automatic Dependent Surveillance-Broadcast) can provide target location, speed and other positioning information as well as the label information including the flight number.
In this paper, the label information of ADS-B is added into the video monitoring through data fusion to achieve automatic labeling of the video tracking targets.

SYSTEM SCHEME
The overall system scheme is as shown in Figure 1.Firstly, a frame of image is extracted from the video and then feature points are extracted from the image to find out the obvious point or line marks on the airport ground.Meanwhile, the accurate map coordinates of these point or line correspondences will be found out on the airport map.The homography matrix between the video image and airport map can be calculated with over four pairs of correspondences.Background differencing [1,2] and KLT algorithm are applied to the airport video to achieve detection and tracking of image targets, and the center of image target tracking is taken as the position coordinates of the aircraft image.The homography matrix is utilized to transform the map coordinates of aircraft in ADS-B monitoring data into the image coordinates in video.Finally, the target position data in ADS-B and video tracking data are subject to the data fusion.The K-nearest neighbor algorithm and others are applied to achieve the data association [8], thus correlating the label information including the flight number in ADS-B to the video to achieve the automatic labeling.

CAMERA CORRECTION
Camera correction is for achieving inter-conversion of the image coordinates of video tracking targets and the actual airport map coordinates and is a critical step for achieving automatic labeling in video.If the airport map is a projective plane and the video image is another projective plane, the projection transformation relationship between both planes can be described with a 3 3  u homography matrix H [9][10][11], as shown in Figure 2. The Second Research Institute of CAAC, Chengdu, Sichuan, China ABSTRACT: For the problem that the airport video monitoring could only provide the image information while the label information including the flight number is not provided, a new method for automatically labeling aircrafts in airport video through the fusion of video and ADS-B data has been proposed.First, the image coordinates of aircrafts will be obtained through the image tracking of video.Then, the homography matrix between two projection planes will be calculated with the four and above point and line correspondences selected from the airport map and video image, respectively to the map image coordinates into the map coordinates.Finally, the aircrafts in video can be automatically labeled through the fusion of image tracking data and ADS-B monitoring data.Because an image coordinate measurement error exists at the time of selecting points from the image, the resulting coordinate conversion error is derived and the impact of point correspondence geometric layout on mesh coordinate mapping error is analyzed.Experiments have been conducted based on the actual data of Chengdu Shuangliu International Airport.The results show that the method can automatically label aircrafts in video in an effective way.To put the map coordinates ( , )   x y and image coordinates ( , )   u v into the three-dimensional homogeneous column vectors ( , ,1) T X x y and ( , ,1) T X u v c

A pair of points
, the equation (1) can be written as follows: What is significant to homography matrix H is the ratio of matrix elements.Among the nine elements of H , there are 8 independent ratios.Therefore, homography matrix H has eight degrees of freedom.In the equation, homography matrix H multiplied by a nonzero scale factor will not change the projection transformation relationship, that is, homography matrix can only be determined with the difference of only one nonzero scale factor O . Two

( ) 0 h x h y h h x h y h v (4)
The way to write equations ( 3) and ( 4) into a matrix form is as follows: Wherein:

T h h h h h h h h h h
. It means that the correspondence of each pair of points can provide two independent linear equations.Then, four pairs of points should provide eight independent linear equations.Four 2 9 u matrixes A i can be superposed to constitute a 8 9 u matrix A .The sole limit is that any three points out of four point correspondences are non-collinear.As the rank of A is eight and the equation set 0

Ah
, there is a one-dimensional null space, thus obtaining the determined solution h with the difference of only one scale factor O .Once the homography matrix H is obtained, the inter-conversion between the image coordinates and map coordinates of any point can be achieved through equation (2).
When there are over four groups of point correspondence, 0 Ah is an over-determined equation set.
If the measured values of all point correspondences are accurate, the rank of A is still eight and A has a one-dimensional null space, for which exact solution of h can be obtained.However, in fact, there is error in the coordinate measurements of point correspondence due to the impact of noise, resulting in no accurate solution of the over-determined equation 0 Ah .Then, we can only define a cost function to obtain an approximate solution to let the cost function to have the minimum value.Generally, algebraic distance Ah is taken as the cost function.In order to ensure that h is not the vector 0, a norm condition Sometimes, it is difficult to select sufficient point correspondences on the airport map and video image.In addition to selection of point correspondence to calculate homography matrix, line correspondence can also be used to calculate homography matrix; the experiments in the literature by Wang [13] indicate that the homography matrix estimation method based on line correspondence has higher precision and better robustness than the point correspondence estimation.According to the duality principle of two-dimensional projective geometry, the theorem applicable to points is also applicable to lines, and points and lines are interchangeable.Therefore, in addition to point correspondence, the line correspondence can also be used to calculate homography matrix.
The linear equation on plane is 0 ax by c , wherein a , b and c are parameters of straight line.Therefore, it is the same with the homogeneous vector representation of points that straight lines can also be represented as homogeneous vectors ( , , ) T a b c .Assuming a pair of line correspondence l , c l on airport map plane and video image, then the point X on the straight line l and the point X c on the straight line lc can be respectively expressed as follows: It can be seen from equation ( 1) that X =HX c .The following equation can be obtained after putting in equation ( 7): Through comparing equations ( 7) and ( 8), the following equation can be obtained: The matrix as shown below can also be obtained after removing the O in the equation ( 10):

COORDINATE CONVERSION ERROR
The key for realizing aircraft automatic labeling in the video is to make the image coordinates of target and ADS-B survey coordinates realize mutual conversion possibly.The accuracy of coordinate conversion will directly influence the effects of automatic labeling.In this section, we will discuss the conversion error between the image coordinates and map coordinates.
The coordinates corresponding to point have measurement error, which results in that calculated homography matrix generally will not map object point X i on the airport map as the image point i X c in the image accurately, that is to say, error ( , ) exists between the calculated image point i X c and real image point X i c .Assuming that corresponding measurement on the airport map is accurate, this can be ensured by accuracy of airport map.We only discuss the coordinate conversion error due to corresponding coordinate measurement error of the points selected in the image.
For unreliability of homograph, it can be measured by covariance matrix 6 h of homography matrix H .
X c is derived as below: ; H is derived as below: , where h is nine-dimensional column vector composed by elements of H , Write down the j th row of H as jT h , and it can be obtained that Coordinate x and coordinate in the image make up 2n -dimensional measurement vector X .
The process for calculating an estimative converted covariance matrix is as following [9,10]: (1) Estimate converted Ĥ according to data of marching point and elements of Ĥ constitutes nine-dimensional vector ĥ ; (2) Calculate the value of Jacobian matrix / J X h w w c at ĥ , and the calculation formula is given in (b); Write Jacobian matrix in the form of partitioning, namely ( , , , , , ) 1 2 , and the following equation can be obtained: , where T X i indicates vector ( , ,1) of h Assuming that measurement of X i is error-free and measurement of X i c is mutually independent, namely, c is an independent gaussian random variable.If there is a standard deviation of m pixels at each coordinate direction, that is, covariance matrix Once covariance matrix of h is determined, the error of image point X c mapped in the image by any object point X on the map can be calculated, and covariance matrix of X c is as below: What we generally concern about is the distance from the image target calculated by homography matrix and the real image target, that is, root-mean-square error at position X c : ( ) It can be proved that [9,10], for identity mapping, if the points selected are uniformly distributed on a unit circle correspondingly, then That is to say, distribution of X c depends on radial distance of X c .The farther distance from the target to the center of circle is, the larger the error is.
Below we will use a checkerboard photo for test and carry out simulation verification for the above error calculation equation.Figure 3. is a checkerboard photo, where the points numbered as 1-6 in the figure are corresponding to the points selected for calculating homography matrix, and the blue points are the test points used for calculating coordinate conversion result.Assuming that corresponding world coordinates of the points on the checkerboard have no error, measurement error of image coordinate complies with the Gaussian distribution with mean value of 0-pixel and standard deviation of three pixels.
For point correspondence in the image, gaussian noise generated every time that complies with mean value of 0 and standard deviation of three is added to coordinates corresponding to point to calculate H , and then map world coordinate of blue point on the chessboard to the image .Repeat for 300 times and draw 3V ellipse distribution situation of subpoint.It can be seen that from Figure 4, projection scattering points are mostly located within two-dimensional Gaussian distribution 3V ellipse, which indicates that the error calculation formula is reliable.
A projection of test point coordinate projection error finally obtained by respectively selecting two sets of points numbered as 1,2,3,4 and 1,2,3,5 to correspondingly calculate homography matrix is shown in Figure 5.It can be seen that, four points are used to correspondingly calculate homography matrix, but different geometric distribution causes different projection error, which indicates that corresponding geometric distribution of point has important influence on projection error, and homographic point shall try to surround the area needing coordinate conversion if we want to reduce coordinate conversion error.The error compari- son between four and six point correspondences is shown in Figure 6, where the blue color indicates corresponding 3V ellipse of four points, and the red color indicates corresponding 3V ellipse of six points.
It shows that, the more point correspondences selected is, the smaller the error is.
An error distribution curve of homography matrix correspondingly calculated out by the points numbered as 1,2,3,4 is shown in Figure 7.And Figure 8 is an error distribution curve of homography matrix correspondingly calculated out by the points numbered as 1,2,3,5.It can be seen from Figure 7 and Figure 8 that, error distribution curve changes with corresponding geometric distribution of point, and the closer distance to point correspondence is, the smaller the error is.

EXPERIMENTAL RESULTS
A Prosillica GC1350C industrial camera with the resolution of 1360× 1024 produced by German AVT Company was used for the site video recording in the control tower of Chengdu Shuangliu International Airport on June 15, 2011.The ADS-B monitoring data  that is synchronous with the video is from the ADS-B ground station developed by The Second Research Institute of CAAC installed in the airport.
Figure 9 is a curve to calculate homography matrix and analyze error distribution by respectively selecting five relatively obvious mark points from the image and the airport map according to the intercepted one frame of video image.For point correspondence on the image and the map, manual selection is used and it is assumed that point correspondence coordinates on the map are accurate and point correspondence image error on the image complies with Gaussian distribution with five-pixel standard deviation.It can be seen from the figure that, the airport apron area surrounded by five points has relatively small error, the error of runway and taxi-way at the top left of the image is obviously increased because they are relatively far away from the point correspondence distribution area.Homograph is a projection conversion relationship between two planes, therefore, coordinate conversion between surface area in the image and the map can be realized only, and the air area in the image hasn't been in the airport plane, so such homograph relationship is not established and the error curve isn't given out.
Background differencing method is used to detect the image moving target.Gaussian Mixture Model [14,15] is used to establish a background model to extract background image and make difference with the original image to obtain a difference imaging, so as to detect the image moving target.KLT algorithm is used to track the image moving target [3][4][5][6][7].KLT is a tracking algorithm based on feature points, which conducts multiple iterations through building a Gaussian pyramid and predicts the withdrawn feature points, so as to achieve the purpose of tracking.
Assuming that measurement of image coordinate complies with Gaussian distribution for the convenience of error analysis, generally, such assumption is not verified [9,10].Meanwhile, measurement standard difference of homographic point on the image is also from experience and assumption.Thus more meaningful point relative to specific value of coordinate conversion error is the change trend of the error distribution curve.
The detection of moving objects in image utilizes background differencing.The background picture is extracted through establishing a background model using Gaussian Mixture Model [14,15], and is compared with the original image to obtain a difference image, thus detecting the moving objects in image.The tracking of moving objects in image applies KLT algorithm [3][4][5][6][7].KLT is a tracing algorithm based on feature points.Prediction of the feature points extracted is made through constructing a Gaussian pyramid and multiple iterations, achieving the purpose of tracking.
For the measurement of image coordinates, it is assumed to comply with the Gaussian distribution, so as to facilitate error analysis.This assumption has gener-ally not been validated [9,10].Meanwhile, the standard deviation of measurement of homographic points on image is also from experience and assumption.Therefore, compared to the specific value of coordinate conversion error, what is more significant is the variation trend of the error distribution curve.

CONCLUSION
(1) The four and above points are selected from the video image and the airport map to correspondingly calculate the homography matrix.The homography matrix is used to achieve conversion between image coordinates and geographic coordinates.The ADS-B data and video monitoring data integration is used to realize automatic labeling of aircraft in the monitoring video.
(2) The coordinate conversion error due to existing coordinate measurement error at the time of selecting points from the image is calculated, and the relationship between point-corresponding geometric distribution and error curve is analyzed.The error in the central area surrounded by point correspondences is relatively small and the error in the area far away from the point correspondences is relatively large.Theoretical derivation is also identical to Monte-Carlo simulation experiment.
(3) The ADS-B data and video monitoring data actually measured at Chengdu Shuangliu International Airport were used for experiments.The results show that the method applied in the paper can automatically label aircrafts in the airport video monitoring in an effective way.
Keywords: automatically labeling; video monitoring; ADS-B; data fusion; Homography Matrix; error analysis DOI: 10.1051/ C Owned by the authors, published by EDP Sciences, 2015

)
Figure 1.Overall System Scheme are n point correspondences and m line correspondences, a (2 2 ) 9 n m u matrix A can be obtained.The optimal solution h can be ob- tained from the over-determined equation set 0 Ah .

Figure 9 .
Figure 9. Error Distribution Curve of Airport Coordinate Conversion shows the automatic labeling effects achieved by video data and ADS-B data fusion.The track formed by the red circles in the picture indicates the positions of aircraft's ADS-B monitoring data mapped into the image; the blue rectangle frame indicates the image target detected and tracked.The blue point in the blue rectangle frame indicates the center of image target.The flight number of aircraft is marked on the rectangle frame.
Figure 11  shows that automatic labeling has been achieved for both flights CA4462 and HU7856 in the video.Due to the limited video recording time and limited visual angle of camera in the airport, and that not all aircrafts are installed with ADS-B airborne equipment, only two flights CA4462 and HU7856 are simultaneously recorded by video and ADS-B in the experiment.In order to further validate the reliability of coordinate transformation algorithm, a time frame not recorded

Figure 10 .Figure 11 .
Figure 10.Selection of Homographic Points from the Airport Video Image