A Mission Performance Evaluation Approach for Civil UAS Applications

The Unmanned Aircraft Systems (UAS) mission fulfilment grade is determined by performance capabilities of the system elements, such as UAV flight performance, sensor parameters, energy consumption and communication abilities. The mission simulation and evaluation tool chain developed at the Institute of Aircraft Design allows to assess the system effectiveness in terms of civil and commercial UAS applications and by this to evaluate trade off studies regarding the compatibility between the air vehicle, the sensor payload and the mission. The presented approach for mission performance evaluation is based on the calculation of an overall mission performance index implemented in the UAS design and optimization processes.


Introduction
In this work an approach for UAS mission assessment and evaluation is presented.The methodology is based on deriving the corresponding mission metrics and the mission performance index (MPI).At the beginning of the presented approach the key mission performance parameters together with the mission requirements are defined.Depending on the mission goal the key performance parameters are weighted, normalized und implemented into the metric.In the second stage the UAS mission is simulated in the Mission Simulation and Evaluation Tool.During the mission simulation in the virtual operational environment the key evaluation criteria such as sensor coverage area, probabilities of objects detection, number of detected objects, slant range between the unmanned aerial vehicle (UAV) and search objects, communication range, obstacles detection in the line-of-sight, time of communication losses and others are obtained.Based on the derived criteria the MPI is calculated and the designed system in terms of overall mission performance is assessed.In order to compare different system configurations the evaluation methodology is implemented into the UAS design and optimization processes.As a result the UAS tailored to mission requirements is obtained.
The state of the art, the simulation and evaluation environment, as well as the mission evaluation metrics for civil UAS applications are presented and demonstrated on a simplified case study in this paper.

State of the art
Approaches for military UAS selection and evaluation according to mission requirements are already presented in several research groups.For example, a technique for military aircraft design evaluation is presented at the Georgia Technical University in the group of Aerospace Systems Design [1].Five most important attributes of a military aircraft, such as affordability, capability, operational safety, survivability and readiness are considered through an overall evaluation criteria.The approach results in a series of outcomes depending on the needs and preferences of the customer.The alternative concepts are compared through an evaluation matrix of overall evaluation criteria.Owing to the system attributes which are taken into account, these approaches are not suitable for a mission performance evaluation of a civil UAS.The application variety of the latter is rising and therefore a method for a system evaluation and selection according to the operational requirements is needed.
In this paper the presented UAS evaluation approach is based on the system simulation in the geospatial environment and on the calculation of the overall mission performance index.

Mission performance evaluation metric
UAS mission performance success depends on the performance capabilities of the system elements such as sensor camera parameters, flight altitude and speed, energy consumption and communication capabilities.Since the mission performance can be expressed in a number of criteria, the design process of an UAS consists of conflicting goals and objectives.For example, required ground resolution and desired image quality set the maximum altitude and speed for the UAV, sensor camera characteristics put limits on area coverage rate and give additional weight to the platform.At the same time bandwidth limits of the communications systems restrict the payload data transmission rate.
In order to assess and compare different UAS configurations an evaluation metric is introduced and implemented into the mission simulation and evaluation tool chain.

Key performance parameters
With technologies improvements UAS get a wider usage in civilian applications.Systems becoming more affordable and available and new classes of UAS consumers appear.According to key evaluation parameters the area of civil UAS applications can be divided into 3 groups [2]: 1. Mapping tasks, such as: crop performance, surveying of a construction site, environment monitoring, aerial photography and etc.For these type of missions important parameters are coverage area, quality of gathered data, cost, and noise.
2. Search and Track, such as: pipelines monitoring, traffic control, border patrol, wildlife monitoring, search and rescue missions, fishery protection, forestry monitoring and etc.In addition to coverage area and data quality, parameters like objects of interest detection probability, flight time to the search area, searching time and communication capabilities have to be taken into account during the mission evaluation process.
3. Delivery and Transport, such as: delivery of packages to customers, relay radio and wireless internet hot spots to restore service after a natural disaster.For these type of missions important parameters are operation time, UAV flight performance, payload capacity and costs.
According to Gundlach [3] a system requirements document can contain hundreds or thousands performance parameters that have to be met for compliance.The most important of them are called key performance parameters or top-level requirements and can include the following: -Total endurance In order to quantify UAS effectiveness the key performance parameters have to be defined.In the presented approach these parameters are obtained during the mission simulation and system overall assessment in the operational environment and are regarded as optimization goals in the UAS design process.The details of the design, simulation, optimization and evaluation tool chain are presented in the Section 4.

Mission performance index
Traditional system evaluation and optimization approaches focus on one or two system attributes independently [3].The introduction of the overall mission performance index (MPI) allows to take into account the correlation between the performance capabilities of the system elements, air vehicle design and system effectiveness.
In order to incorporate a mission evaluation method into a multidisciplinary design process the following requirements have to be fulfilled [4]: -MPI has to be quantified with a scalar value in order to be used as the result of an objective function for numerical optimization -Each contributing parameter should be normalized by a referenced value in order to facilitate comparability between mission evaluation results -Performance parameters should be normalized with reference values of the same physical unit -Each performance parameter is weighted by importance coefficients, which have to sum together to unity.
The goal of the UAS optimization process is to maximize the scalar MPI by varying aircraft geometry and payload sensor parameters using a genetic optimization algorithm.By that a UAS configuration is tailored to mission requirements.
The most commonly used method to handle more than one decision criteria and which allows to obtain an overall performance score is the Weighted Sum Model (WSM).In WSM method each attribute is multiplied with a weighting factor and added together.This approach is used as a base in MPI calculation.
According to the UAS performance attributes derived from the mission simulation and evaluation tool chain, the MPI is defined as [5]: The area coverage rate (ACR) is defined as covered area per mission time T: (2) where w swath is the ground sensor swath width and V is the airspeed during the mission.
The degradation factor ƒ deg (GSD) is introduced in form of an exponential function and it takes into account limits of the Ground Sample Distance (GSD) [6].The latter is used as a measure of a distance between the pixels projected on the ground at a slant range [3] and is calculated as: where FOV H is the horizontal field of view, H pix is the number of horizontal pixels of the camera and R the current altitude over ground of the aircraft.
The parameter E quantifies the amount of energy or fuel needed to fulfil the mission depending of the UAV engine type.
The object detection probability P det is determined according to the semi-empirical method called Johnson Criteria and time of object detection [5].Johnson Criteria calculates the probability Jp(N) of performing the respective task by comparing the number of sensor cycles N covered by a target to a reference value N 50 , which stands for 50% probability.According to Johnson the reference values N 50 is used with 0.75 for detection, 3.0 for recognition and 6.0 for identification.In an approach introduced by Schmieder and Weathersby the Johnson's criteria is adapted to different target scenarios by setting the respective N values [7].
Figure 1 depicts the dependence of detection probability and detection time.Based on the fact that in average a human needs 0,25s to notice visual changes [8], it is assumed that an operator needs minimum 2 second to detect an object of interest.At that moment the detection probability is considered as 70% and is rising further linearly.The 100% probability is achieved at a time of 10s [5].The Communication Abilities (CA) parameter contains information about time of communication losses Tcl and the possibility to store data on board during the losses.It can be represented as ratio between the mission time and time of communication losses: (5) User Operating Issues (UOI) attribute depending on the mission requirements can contain information about information about launch/landing options according to the system weight, how easy it is to use the system, if it needs any special handling or storage facilities and how environmental sensitive the systems is.For instance, for an agriculture application the most attractive UAS would be robust, hand-launched and belly-landed, not sensitive to environment conditions and of course easy to operate.The UOI attribute is introduced in form of a scalar value within the range from 0 to 1 [5].

Assignment of weights
In general weighting approaches can be divided into sumnormalized (coefficients sum to unity) and scoring models.In both cases the more important the criterion the higher the weighting coefficient.
Weights coefficients of criteria can be determined in several ways.One of the methods is so-called weight decision matrix, obtained from Analytic Hierarchy Process (AHP) method.The AHP method was developed mainly by Thomas L. Saaty [9].According to this approached each criteria is weighted in terms of all other criteria with relative importance indices from Tab. 1. From these values a matrix of weights is derived and the total criteria weights are calculated based on the use of eigenvectors.In order to reduce the large number of ranking decisions made by the decision maker a Multi-criterion Analysis of Preferences by means of Pairwise Actions and Criterion comparisons (MAPPAC) method, which is sometimes also referred as PAPRIKA (Potentially All Pairwise Rankings of All Possible Alternatives) can be used.In this method each criterion has a limited number of possible expressions, such as "poor", "medium", and "good".The criteria are pairwise compared and ranked [10].
Another technique called a Rank-Order Centroid [11] does not require the weights assignments made by a decision maker.The criteria are ranked from most important to least important.Weights can be assigned using one of the following formulas, where ri is the rank of the i-th objective and K is the total number of objectives: -Rank Sum Other possible ranking methods are presented in Buede [12].
The assignment of the weighting coefficients has to be adjusted to every mission case by the design engineer based on engineering insight and experience.Despite the fact that there are many methods for weight assignment, this process still stays subjective [5].

Mission simulation and evaluation
A tool chain is developed at the Institute of Aircraft Design, TUM, in order to take into account all elements of the system, e.g. to bring together aircraft design, payload, communication and other elements into one multidisciplinary design process and to get an effective UAS solution.

Overview of the environment for UAS design, mission simulation and evaluation
The overall structure of the tool chain is presented in Figure 2. It consists of the UAS design, mission simulation, visualized operational environment, mission evaluation and optimization parts.The UAS design environment creates a consistent design of an UAV taking into account general geometric data, masses, aerodynamic polars and propulsion system data.Simplified physical methods, especially for aerodynamics and structural weight estimation, are used where applicable.An overview of the UAS design process is presented in Figure 3 [4].The visualization part of the tool chain simulates the operational environment where the UAS performs a mission.
It receives information about the UAV position and orientation, as well as the orientation of the sensor installed on the platform from the simulation model.By means of high resolution texture data, elevation model based landscape, simulation of interaction of the UAS elements with each other and the environment the following information is obtained from the visualization part [12]: -Sensor ground swath -Detection probability of search objects -Slant range between UAV and search objects -Communication range and obstacles detection in the line sight -Time of communication losses -Height of the terrain These data are used in the sensor performance and mission effectiveness evaluation process which is based on the MPI.
In order to find the best suited UAS design by maximizing the MPI a genetic optimization algorithm is applied [4].By varying design variables, such as aircraft wing geometry and sensor performance parameters, a finite number of UAS configurations is generated, evaluated and ranked by the MPI.By using methods of randomized recombination, mutation and saving best individuals a new generation of UAS configurations is created, evaluated and ranked.This process is repeated until no further improvements of the MPI is achieved.

Mission design study
A design study for search and rescue mission is presented in order to verify the capabilities of the presented tool chain and performance evaluation metric.
The objective of the mission is to detect within the shortest time a lost boat in a certain area of the lake Chiemsee, close to Rosenheim, Germany.The area of interest is 16 km 2 .In order to take into account the uncertainty of the boat location three possible spots are defined at the beginning of the simulation.Therefore during the simulation it should be determined whether an object with an average size of 2.7 m length can be detected or recognized in one of these locations.
For the optimization process 4 design variables and their ranges are defined in Table 2.The possible camera types are presented in Table 3.The required GSD is defined as 40 cm in order to detect and recognise the objects of interest.Sensors with bigger resolution allow to fly with higher altitudes and therefore with bigger sensor field of view, while sensors with smaller resolution achieve the desired GSD with lower altitudes.At the same time, cameras with better resolution are more expensive and can be heavier.This trade off should be taken into account when choosing a sensor payload for an UAS.
For the presented mission design study there are two important evaluation parameters: object detection and mission time.Therefore the MPI for the optimization process is represented in the following way: (12) where P det is probability of the object detection in the i-th location and T m is the mission time.
The object detection is assumed as more important attribute of the mission success and weighting coefficients are taken as γ=0.7 and δ=0.3.
Four generations, with 5 individuals each, of UAS configurations were created during the simulation.Each individual, e.g.UAS configuration, was evaluated and ranked by the MPI.As a result of the optimization process the unmanned aerial system with camera 1 was designed according to the initial mission requirements.Figure 4 and Figure 5 presents the results of the optimization process.
The mission is fulfilled in 24 min at an altitude of 600 m over the water level and with a flight speed of 27 m/s.The boat is detected and recognized in all 3 possible locations.Results of objects detection, recognition and identification according to Johnson Criteria are presented in Figure 6.In order to show the flexibility of the presented approach for UAS design and mission evaluation, a bigger search area on the lake Chiemsee and different weighting coefficients assignment in Equation 12are chosen.The new search area is 41 km2 and γ=0.5, δ=0.5 The variation of the mission performance index during the optimization is presented in Figure 7.In order to obtain a converging process a bigger number of generations is necessary.The variation of the design variables for the new weighting scheme and search area is shown in Figure 8.The UAS configuration is designed with camera 3 and the platform is significantly bigger than in the first case.The mission is fulfilled in 58 minutes at an altitude of 960 m and with a flight speed of 32 m/s.

Conclusion
The presented tool chain allows to design, simulate, evaluate and optimize an UAS according to initial mission requirements.It is possible to perform within the simulation environment different UAS mission scenarios.By simulating the visualized operational environment it is possible to take sensor and communication requirements into account in the UAS design process.A genetic optimization algorithm is used to find an unmanned aircraft design that optimally fulfills a certain mission task.
The mission design study presents the flexibility of the tool chain and importance of chosen mission evaluation metric.Mission performance attributes selection and it's weighting assignment play a role in the UAS optimization process.
UAVs required for the mission (Fleet sizing for target coverage).Operational analysis represents the systems simulation in a geospatial simulation environments.It includes the following: -Target coverage -Payload effectiveness in operational environment -Communication analysis -Fleet sizing -Systems comparisons -Logistic analysis

Figure 1 .
Figure1.Dependence of detection probability and detection time[5] an undefined measure of the dispersion in the weights

Figure 2 .
Figure 2. Overall structure of the UAS design, mission simulation and evaluation tool chain [5].

Figure 3 .
Figure 3. Design process for calculation of a consistent aircraft dataset [4].

Figure 6 .
Figure 6.Results of objects detection, recognition and identification according to Johnson Criteria for γ=0.7, δ=0.3 and search area of 16 km 2 .

Table 1 .
Relative importance indices for weight determination in AHP.