A new UAV PTZ Controlling System with Target Localization

In recent years, with the continuous exposure of unmanned aerial vehicles in the local battlefield, as well as the rapid development of UAV technology, UAV's military application values, especially the advantages in the low-altitude reconnaissance have been the world's attention. Unmanned reconnaissance aircraft is becoming very promising new equipment in technology battlefield and an important information combat front. However, most of the current UAV controlling systems only provide manually PTZ controlling functions, which is inconvenient for the user to control the sensor to monitor targets on demand. This paper propose a new UAV PTZ controlling system, which enables user to keep target in the view filed by simply selecting a position from the digital map. The proposed system is also able to do real-time target localization for any pixel of the video. Experimental results show that our system is easy for PTZ controlling and the localization error is small.


Introduction
Unmanned Aerial Vehicle (UAV) [1,2], as new modern military equipment, with its unique advantage, is fully compatible with the modern war concepts of "no contact" and "zero casualties".UAVs can carry out tasks such as reconnaissance and surveillance, laser guidance, electronic jamming, communication relay, target location, combat assessment, and precision strike by carrying different equipment.UAVs can also be used in civil areas.It can be applied to field monitoring, weather detection, road inspection, exploration and mapping, flood monitoring, aerial photography, traffic management, power line inquiries, forest fire prevention and so on.
Optical sensor as one of the most common sensors carried by UAVs can be used for real-time collection and processing.The real-time battlefield video information, which is the most basic and most important information, plays an important role in flight controlling, battlefield commanding, reconnaissance, striking, and so on.
The sensor movement controlling, also known as PTZ (Pan/Tile/Zoom) controlling [3,4], is an important problem for continuous monitoring.At present, most UAVs only support the artificial control of the yaw, pitch and roll of the sensors, as shown in Figure 1, which is difficult to keep targets in the view field, especially when the UAV is moving.It's also inconvenient for the user to control the sensor to monitor multiple targets on demand.To solve this problem, we propose a new sensor controlling scheme, which allow users to control the sensor by directly selecting targets on the map.
In order to control sensors automatically, we need to know which place the sensor is now watching, which is known as target localization [5,6].The main goal of the target localization with the UAV output video is to match the pixel position of the target in the video image with the coordinates of the object in the real scene, such as GPS coordinates.The main method is based on the navigation information and equipment parameters to calculate the spatial coordinate system, which builds the conversion system between UAV carrier coordinate system and the geocentric coordinate system.At present, the localization technology based on UAV video images mainly focus on how to match the image between real-time images and the reference images [7][8][9][10], such as satellite images.The basic image matching algorithm is already very mature, and the key is how to match the images of different sources and how to improve the reliability and speed of matching.Based on the image matching, the algorithm uses multiple points on a pair of images to describe the correspondences [11,12].However, if the view field optical sensor contains a limited number of landmarks, the matching performance on the images cannot be guaranteed, and the algorithm will fail.In addition, the algorithm performs image matching is complex and time-consuming, which is not able to do real-time localization [13,14].This paper proposes a real-time target localization method based on UAV's video, which is able to calculate the localization of every pixel in the image on the fly.
The rest of this paper is organized as follows.In the next section, we describe the main components of proposed UAV PTZ controlling system.In section 3, we detail the real-time target localization method and the PTZ controlling method will be described in section 4. The prototype system and the experimental results are shown in section 5.

System Overview
The proposed system comprises a metadata processing module, an elevation data management module, a target localization module, a PTZ auxiliary control module and a GIS module, which is shown in Figure 2. The metadata processing module is responsible for receiving the UAV Flight data and sensor status data, and extracted useful information such as latitude, longitude, height, pitch angle, azimuth angle, roll angle of UAV and sensor, horizontal field angle, vertical field angle and video resolution.
The elevation data management module is responsible for storing and accessing the elevation data, converting the divided DEM data to the geocentric coordinate system, dynamically loading and buffering the target according to the UAV's flight trajectory.The target localization module calculates the angles of the center of view field according to the received metadata and performs the coordinate transformation, and calculates the intersection of view sight with the elevation data in the geocentric coordinate system.Thus the latitude, the latitude and the height of the center of the view field are obtained.
The GIS module displays the real-time view filed of sensor and enables user to select the position of target for monitoring, which is send to PTZ control module for further processing.
The working flow is described as follows.First, the metadata processing module receives the flight data and load data returned by the UAV in real time which is shown in Table 1.Then the cached metadata is sent to target localization module and PTZ control module.The target localization module uses the metadata and elevation data as input and outputs the geospatial location of the image pixels.And the PTZ control module use received metadata, elevation data and selected target position to calculate derivate angles and send controlling commands back UAV.The details of proposed target localization method and PTZ controlling method will be described in the next sections.

Proposed Target Localization Method
The received metadata information is buffered in Redis (a memory-based efficient key-value database) for quick access, and the cache invalidation time is set to 20 seconds.For each received video frame, the target localization module queries the metadata before and after the video frame from Redis according to its timestamp and interpolates to obtain the metadata information corresponding to the current frame, and then calculates the longitudes and latitudes of the video center and the four corners.The details are described as follows.
1. Calculate the distance between CCD and the lens which denotes as L=W/(2tan(ψh /2)), where W is the width of video frame, and ψh is the sensor horizontal view field angle; 2. Calculate the rotation matrix Rσ=R(Vc, Vp) between the visual sight Vc=< wc , Dc > and the target line of sight Vp=< wc , Pc>, where wc=(W/2,H/2,L), Dc=(W/2,H/2,0), Pc=(x,y,0), W is the height of video frame, H is the height of video frame and (x,y) is the pixel coordinates of target; 3. Calculate the rotation of UAV in geocentric system which denotes as Rlon and Rα, where Rα=R(<(0,0,0),(0,0,1)>,<α0, α1>),α0=T(αlon, αlat, αalt-100), α1=T(αlon, αlat, αalt) and function T transform longitude, latitude and height to geocentric coordinates; 4. Calculate the rotation matrix between UAV and the warp which records as Rβ based on the yaw, pitch and roll angles of UAV; 5. Calculate the rotation matrix between the sensor sight axis and the UAV body axis which notes as Rλ based on the yaw angle and pitch angle of sensor; 6. obtain the UAV flight area elevation data, convert it to the geocentric coordinate system, and denote the elevation surface as S; 7. Define a straight line in the geocentric coordinate system Vt=<(0,0,0),(0,0,-1)>, and then rotate this

Proposed PTZ Controlling Method
The PTZ auxiliary control module obtains the target elevation according to the latitude and longitude selected from GIS map, and then calculates the azimuth and pitch angle according to the UAV real-time metadata information.Finally, the module calculates the relative yaw and pitch angle which need to be changed and send commands to UAV for actions.The details are described as follows.
1. Select target in the GIS map, and get the longitude and latitude (Plon, Plat); 2. Get the elevation data Plat with the longitude and latitude of selected target, and transform to geocentric coordinates where the target is at P=( Px, Py, Pz); 3. Calculate the rotation of UAV in geocentric system which denotes as Rlon and Rα, where Rα=R(<(0,0,0),(0,0,1)>,<α0, α1>),α0=T(αlon, αlat, αalt-100), α1=T(αlon, αlat, αalt) and function T transform longitude, latitude and height to geocentric coordinates; 4. Calculate the rotation matrix between UAV and the warp which records as Rβ based on the yaw, pitch and roll angles of UAV; 5. Define a straight line in the geocentric coordinate system Vt=<(0,0,0),(0,0,-1)>, and then rotate this line with the rotation matrixes Vtr0= Vt* Rβ* Rlon * Rα; 6. Calculate the deviation azimuth angle Ky and the deviation pitch angle Kp between Vtr0 and β, where β is the geocentric coordinates of UAV; 7. Send the rotation command to UAV and rotate the sensor with Ky and Kp, the target will be displayed in the view field.

Experimental Results
Both target localization and PTZ controlling module need elevation data for calculation.In this paper, we use SRTM [15], which covers an area of more than 119 million square kilometres between 60 degrees north latitude and 56 degrees south latitude, covering more than 80% of the global land surface.The original data is more than 90G and cannot all be loaded into memory.In order to obtain the elevation information of the specified latitude and longitude in real time, the system divide the elevation data into the small pieces and dynamically load those pieces nearby the position of UAV.During the flying of UAV, the system will automatically load new pieces and unload historical pieces, thus using a limited memory space to ensure the 100% hitting rate of getting elevation information of specified latitude and longitude with limited flight speed. (a1) (b2) The target localization module can locate the target of any pixel in the video without laser ranging, and the time of single pixel positioning takes less than 50ms.In order to verify the effectiveness of the method, the method of the invention uses the UAV's historical metadata information and video data to locate and compares with the satellite map.The experimental results show that the positioning error of the method is up to 9 to 300 meters.Figure 3 shows the examples of target localization results, (a1) and (a2) are video frames from real-time video and selected target (red rectangle), (b1) and (b2) are localization results shown on Google map.
During the UAV flight, the proposed system can display the position of the view field on the satellite map in real time, and the existing UAV operating system, such as DJI, can only display the UAV and the direction of camera.Our system provides a more convenient way for aerial reconnaissance.We also evaluate performance the proposed PTZ controlling method.We manually select different targets from GIS map, and find the corresponding target in the video after controlling command are executed.The result is shown in Figure 3, where the rectangle denotes as the video area and the markers are the selected targets.We select 50 different targets during the flight and all of them were successfully monitored from the video after the automatic rotations.Figure 4 shows the results of PTZ controlling, where (a) is the selected target from GIS map, and (b) is the video frame after rotation.Our proposed PTZ controlling method successfully rotated the pan and the selected target from Google map is captured in the video.

Conclusion
UAV's military application values, especially the advantages in the low-altitude reconnaissance have been the world's attention.However, most of the current UAV controlling systems only provide manually PTZ controlling functions, which is inconvenient for the user to control the sensor to monitor targets on demand.This paper propose a new UAV PTZ controlling system, which enables user to keep target in the view filed by simply selecting a position from the digital map.The proposed system is also able to do real-time target localization for any pixel of the video.Experimental results show that our system is easy for PTZ controlling and the localization error is small.In the future work, we will improve the localization accuracy and we will focus on developing new PTZ controlling method which supports focusing on selected targets when the UAV is moving.

Fig. 1 .
Fig. 1.Example of UAV PTZ controlling for keeping target in the center of view field.
line with the rotation matrixes Vtr= Vt * Rλ* Rβ* Rlon * Rα* Rσ; 8. Calculate the intersection point of line Vtr and the elevation surface S, and the geocentric coordinates of the point is denoted as Ps =( Px, Py, Pz); 9. Transform Ps to longitude, latitude and altitude, Pα=(Plon, Plat, Palt), which is the target localization result.

Fig. 3 .
Fig. 3. Examples of target localization results, (a1) and (a2) are video frames from real-time video and selected target (red rectangle), (b1) and (b2) are localization results shown on Google map.

Fig. 3 .
Fig. 3. Results of PTZ controlling method, where the red rectangle is area of video and the blue markers are the positions of selected targets in video frames.

Fig. 4 .
Fig. 4. Result of PTZ controlling, where (a) is the selected target from GIS map, and (b) is the video frame after rotation.

Table 1 .
Descriptions of Metadata.