Aerospace performance factor and its potential advances

The aviation safety community has traditionally measured safety performance through the lens of undesirable events – the historic recording, investigating and trending of the failures in the system. While this approach gives great insight into what has happened and why, it is of questionable value in determining what, where and when the next incident will occur. With the advent of the Aerospace Performance Factor or APF, aviation is taking a giant step toward a decision management solution that answers these questions and paves the way toward a revolution in safety performance measurement. The APF methodology is designed to expand the view of what information contributes to the aviation safety picture. The APF paints a broad landscape of organizational effectiveness by fusing weighted data from sources not traditionally included in the safety discussion, and then presenting the information in a way that assists aviation leaders in making better riskedbased decisions. This paper will discuss the APF from its to the progress of the initial and ongoing APF operational implementation, to its potential as a riskforecasting tool for the aviation industry.


Introduction
The need for a framework within which aviation stakeholders can manage safety has been recognized at an international level and in a growing number of domestic regulations. ICAO Annex 19 and its complementary domestic regulations are, however, scoped as general statements against which compliance can be tested rather than providing an evolutionary pathway which drives a culture of continual improvement.
The coming decades will present many challenges for aviation, and ANSPs in particular. These challenges will include increased traffic demands, unmanned aircraft, and environmental and security considerations. ANSP management must solve these while maintaining (and even improving) the current levels of safety. To accomplish this, ANSP management must continually strive to improve the ways they identify risks and manage safety. Across the industry, ANSPs are at different stages of SMS development. Some have very mature systems which are fully integrated into the operations. Others are starting to build formalized safety management practices and a culture which assures the priority of safety.
Based on the prediction of air traffic growth, the European Commission expects that the volume of traffic in EU will be doubled by 2020, and of course its main objective is to keep the same safety levels as they are nowadays [1].
A set of regulations for performance of air navigation service providers was established on the European level to ensure that safety within the European Union will be kept on a tolerable safety level.

Stakeholder requirements
It is important that the definition and purpose of each high-level safety key performance indicator (KPI) is documented and clearly communicated to all stakeholders. Figure 1 indicates the key principles for KPIs for different stakeholders, while recognizing the need to assist stakeholders in obtaining appropriate understanding of safety levels and outcomes.
A holistic approach to performance monitoring is an essential input to safety decision making. It is important to ensure that good safety performance is attributing to the efficient performance of the safety system and not simply to a lack of incidents. Very essential is that chosen metrics match the requirements of the stakeholders and decision-makers involved in safety improvements. Licu and Kelly (2007) Stakeholders in wider aviation industry and the general public require a relatively small number of indicators which can give an instant "feel" for the overall position regarding safety performance. Conversely, those involved in the management of the services concerned need a more detailed set of metrics on which base decisions regarding the management of the services and facilities being reviewed.
The pyramid figured below provides an outline of the key principles for KPIs for various stakeholders, where level of detail is much higher at the bottom of the pyramid than at the top. This was governing principle for the development and use of safety indicators. Fig. 1. Key principles on Stakeholders' requirements for safety performance. Source: [2] , 0100 (2018)

Safety key performance indicators
To ensure that safety levels are maintained or improved, systematic safety monitoring processes should evaluate, as a matter of routine, achieved safety performance in all safetyrelated operational activities. Safety performance indicators are used to analyze trends and detect unwanted degradation of safety levels, supporting the development of effective improvement plans. They can also be used to assess the extent to which political, strategic, regulatory and industry safety targets are being met.
Measures of safety performance have to be interactive to cover all aspects of the systems that they address and reflect both system failures (e.g. accident, incidents) and indicators of the proper functioning of critical system components [3].

Lagging indicator
'Metrics those measure safety events that have already occurred including those unwanted safety events you are trying to prevent'.
Lagging indicators are measures of safety occurrences, in particular the negative outcomes that the organization is aiming to prevent. Lagging indicators are mainly used for aggregate, long-term trending, either at a high level or for specific occurrence types or locations. Because they measure safety outcomes, they can be used to assess the effectiveness of safety measures, actions, or initiatives and are a way of validating the safety performance of the system. Also, trends in these indicators can be analyzed to determine if latent conditions exist in present systems that should be addressed.
Two types of lagging indicators are generally defined as: 1. Indicators for high severity negative outcomes, such as accidents or serious incidents. The low frequency of high severity negative outcomes means that aggregation (e.g., at industry segment level or regional level) may produce more meaningful analyzes. Example: number of runway excursions/1000 landings. 2. Indicators for lower level system failures and safety events that did not manifest themselves in serious incidents or accidents (including system failures and procedural deviations); however, safety analysis indicates there is the potential for them to lead to a serious incident or accident when combined with other safety events or conditions. Such indicators are sometimes referred to as 'precursor event' indicators. Example: number of unstabilized approaches/1000 landings.

Leading indicator
'Metrics that provide information on the current situation that may affect future performance'.
Leading indicators should measure both: things that have the potential to become or contribute to a negative outcome in the future ('negative' indicators), and things that contribute to safety ('positive' indicators). From a safety management perspective, it is important to provide sufficient focus on monitoring positive indicators to enable strengthening of those positive factors that make up your company's safety management capability.
Example: The percentage of changes to Standard Operating Procedures that have been subject to hazard identification and safety risk management.
Leading indicators may also be used to inform your management about the dynamics of your system and how it copes with any changes, including changes in its operating environment.
Example: The extent to which work is carried out in accordance with Standard Operating Procedures.
There is an assumed relationship between the two, which suggests that improved performance in a leading indicator will drive better performance in the lagging indicator [4].
It has already been established that an absence of safety incidents is not a true measure of the inherent safety risk within a system. It is important to view safety performance information in the context of the health of the safety management system. The proposed Leading Indicators measure the output of important elements of the safety management system to clarify that excellent safety performance is attributable to a safe system and not attributable to a lack of safety incidents by [5].
The concept of leading and lagging indicators has existed in domains outside of aviation for several years. In particular, economists use them to measure the health of an economy.
Safety performance measurement should ideally consider a combination of leading and lagging indicators. The main focus should be to measure and to act upon the presence of those systemic and operational attributes that enable effective safety management within your company and meanwhile, use lagging indicators to ensure that this safety management is effective. Lagging indicators, particularly indicators for lower level system failures, are useful to validate the effectiveness of specific safety actions and risk barriers or to support the analysis of information derived from your leading indicators.
Measurement of the maturity of ATM SMS is focused on the status of the development and implementation of safety management and safety oversight mechanisms within the EU region. In recognizing the importance of such a study, as it allowed the "temperature" of the safety system in the EU Region. It is ICAO's intent to carry out these measurements annually [6].

The Aerospace Performance Factor
The Aerospace Performance Factor, (APF), represents a new methodology in ATM lagging indicators that moves away from performance measurement that relies on a single or limited number, of measures. Instead, the Aerospace Performance Factor (APF) is an approach to the measurement of safety performance using multiple localized safety measures, e.g. reported incidents, weighted by subject matter expert judgment, and normalized against system operations into one value that can vary over time. Although this unique value gives the overall risk, the methodology allows it to be broken down into its component parts for the analysis of specific causal factors and lessons learnt [7].
The APF displays the resultant value with a system-wide view in a graphical format and can show macro changes in performance trends, based on actual data.
Originally developed by the United States Federal Aviation Administration (FAA) Air Traffic Organization (ATO) and US Navy as a safety metric, the methodology has been adopted by Europe where several air navigation service providers (ANSPs) worked together with EUROCONTROL. Accidents with direct and indirect ATM contribution are also represented in an aggregated manner but represented separately from other safety occurrences [8].
The following sections provide a brief overview of the development of the proposed APF.

Analytical Hierarchy Processing (AHP)
In compiling its APF, the working group used multiple localized safety measures, e.g. reported incidents, causal factors, or other operational factors or constraints etc. These were then weighted by subject matter expert judgment and normalized against system operations to form a hierarchy.
The weighting was achieved by the use of elements of Analytical Hierarchy Processing (AHP), a methodology used in multi-objective decision making analysis. The simplicity and power of the AHP has led to its rapid and widespread use across multiple domains beyond the process of making complex decisions. AHP enables a decision maker to portray the relationships between many facets of a complex problem and incorporates both quantitative and qualitative information including experience and intuition. Hence, the APF methodology can be applied to any organization that needs to assess risk, or performance, based on a complex set of metrics [7,8].

Building the hierarchy
All factors were obtained through the use of historical mishap reports of events and incidents within the system [7,8]. The elements chosen by the working group for the APF were grouped into 4 categories: -Air Incidents/Airborne Events (AI/AE); -Ground Incidents (GI); -Potential/Near Collision in the Air, and -ATM Specific Occurrences.
The elements were then placed in a hierarchy, using a mindmap that represents the logical relationships of the elements to each other. The completed structures of the elements are displayed in the figure below.

Weighting
Once the hierarchy was built, subject matter experts systematically evaluated the various elements by comparing them to one another two at a time. By making the comparison of the various elements the experts used their judgments about the elements' relative significance and a ratio scale was created. The AHP then converted these pair wise evaluations created by the subject matter experts and the weights were created.
The same calculations were then performed on each element of the hierarchy to calculate each element's relative contribution to the overall model. By calculating each element of the branches comprising a collection of the hierarchy and summing the score, the relative contribution of the element and branch to the overall model could be 'rolled-up' and overall scores could be determined. In so doing, all safety elements are 'rolled up' to create a cumulative value on the overall system [7].
The final weights for the EUROCONTROL ESARR 2 APF finalised by SAFREP Ad-Hoc Safety KPI group are as presented in Figure 4 [2].

Accidents APF
Thorough analysis of ATM accidents, the working group decided to include in the accident related KPI not only the accidents involving Commercial aircraft with direct ATM contribution, but also accidents with indirect ATM contribution, as this would also reveal valuable lessons to be learned for ATM and trigger safety improvements [9].
As may be expected, the number of accidents varies from year to year, but the overall trend is not increasing against a background of increasing traffic levels. It must be stressed that the calculations are based on a number of critical assumptions, including the rates of forecast traffic growth and the percentage contribution of ATM within the overall number of accidents. These assumptions are being further validated and improved, supported by increased levels of safety data reporting by states and the work carried out by the working group focused on developing Key Performance Indicators (KPIs) for safety [2].
Meeting the ECAC objective will demand an increasingly stringent safety performance over time with enhanced commitment by States, not only to provide comprehensive data but also to adopt, with an element of urgency, the outstanding action plans published by EUROCONTROL [9].

APF version 2.0 vision and development objectives
The APF is so useful due to its simplicity. One single line is enough in the APF tool to tell a complex story. This added value must be preserved in the future versions of the APF, by protecting it in the change process against the temptations to make things more complex, and by taking measures against the possible loss or reduction of meaning in the output data.
One substantial threat is the noise in data, i.e. variations of the APF line which do not correspond to variations of the measured parameter, which is safety. These artificial variations do not only lack meaning, but they in fact are hiding the actual evolution and the authentic phenomenon. All enhancements in version 2.0 of the APF must consider data noise cancelling where possible, noise reduction, or at least not incurring additional noise. These can and should be done in at least two ways: -noise filtering -numerator -denominator decoupling The numerator -denominator decoupling means adopting a data processing strategy that ensures separate data processing flows for the numerator and denominator before they are used in the division operation. They must remain relevant on their own, as well as independent from each other. This decoupling does not require the numerator and the denominator to be represented separately. They are still portrayed as a ratio quotient.
A strategic target for the APF 2.0 is the full scalability of the application. This means making the tool applicable at different levels, even in a network of users (if required), without the need for changes or adaptations. It should ensure easy and natural navigation from the ANSP divisional level up to the ANSP top management level.
The vision for the APF 2.0 is based on utility and efficiency, aiming at encouraging the adoption of the methodology by new users, and to consolidate the utility of the tool by existing users. The value of a tool such as the APF is given by its frequency of usage and the number of users/organisations.
The goal is to keep the efforts invested by the users into the APF tool at minimum, while enhancing the value of the results. The APF 2.0 is also evolving from just a safety reporting tool to a joint safety & decision-making tool. The purpose is to analyse a potential for further possible developmental enhancements for the current APF tool (V1.1) [7] and to advance the best solutions under the codename APF 2.0. The criteria for this analysis are the following: -consistency with the vision of the initial APF; -favourability among users; -practical utility; -likelihood of operational feasibility; -scientific soundness and consistency; -accuracy of the perception of safety; -strategic alignment with expected developments in aviation; -straightforward implementation;

Resilience engineering
Based on the work of James Reason [12] and Erik Hollnagel [11,13,14], resilience engineering is shifting the focus from what went wrong in an accident or incident, to the entire picture, i.e. the understanding the things that go right as well as go wrong. Its aim is not only to prevent things from going wrong, but also to ensure that things go right, i.e. to facilitate normal outcomes. Resilience engineering represents a new way of regarding what is safe or not. Conventional ways of managing risks are based on hindsight (i.e. on the analysis of what went wrong), on error emphasis and on the calculation of failure probabilities. Resilience engineering looks for ways to enhance the ability of organizations to create processes that are robust yet flexible and adaptive, to monitor and revise risk models, and to proactively use resources in the face of disruptions or ongoing production and economic pressures.

Solution 1
The improved APF will shift the perspective from "what went wrong?" to the whole picture, including "what went right?", to complement the "negative" safety analysis with the positive aspects concerning resilience engineering.
The new version of Risk Analysis Tool (RAT) already uses a neutralised and harmonised list of contributory factors to allow the shift in perception from Safety I to Safety II [11,15]. A factor introduced in RAT can be further subcategorized as being either mitigating or contributing. Mitigating factors are the resilience factors ( Figure 5).
These developments in the Tool Kit for ATM Occurrence Investigation (TOKAI) and RAT lead the way for the APF [16,17]. The APF 2.0 will enjoy full compatibility with TOKAI and RAT, and from this perspective, a hyper diagnostic feature for mitigating factors will be added alongside the hyper diagnostic for contributing factors. These mitigating factors or resilience factors are those described in the Neutralised Harmonised Contributory Factors taxonomy or possibly in other "positive" taxonomies -CAST/ICAO Common Taxonomy Team. This will allow organisations to evaluate and to improve their resilience by using the APF. Figures 6 and 7 illustrate both the existent and the proposed lines of hyper diagnostics for the incidents within the same time interval.

Mind-map aggregation
In the APF 2.0, full scalability is one of the strategic targets. This means that the tool is applicable at different organizational levels, even within a network of users (if so desired), without the need for changes or adaptations. It should ensure easy and natural navigation from the ANSP divisional level up to the ANSP top management level, and beyond.
The structure of an ANSP itself is divisional, and there are demands from the users to apply different mind maps for different divisions. Thus, a multi-level tree organisational structure should be considered, where a mind map may be built in two ways: A. By a panel of experts relevant to that hierarchy level, doing pairwise comparison or B. By automatically merging the mind-maps from the inferior hierarchical level, using a certain driver (e.g. a weighs vector) based on relative importance of the peers, or the relative activity of the peers etc.
There is no contradiction between the latter method (B) to aggregate or merge mind maps of several peers (ANSP divisions, ANSPs), and the AHP. Both ways should be made possible and available in the APF 2.0, for the users to choose from. Method (A) demands user resources but does not require anything in particular from the application. Method (B) is considered as a feature of the application and requires a specific implementation. One natural and effective method put forward in this document is illustrated in the following example.

Solution 9
Merging two or more mind maps automatically (method (B) above), by calculating weighted averages of the weighs in the mind maps using a driver which is relevant to that mind map family. This solution is illustrated in the example below. Let us consider two European ANSPs: ANSP 1 and ANSP2, each having its own mind map (See mind-maps below).  As expected, each ANSP has its own weight for the same child node (e.g. ATM Specific Occurrences -see Table 1). For the aggregation or merging of two or more mind maps, a certain driver should be adopted to weigh the relative importance of each participant in the merger. Table 2 illustrates the method, summarizing the weights of the two ANSPs at the first level of the mind map and presenting the results of the aggregated weights. The resulting aggregated mind map is shown in following figure 10.
The same method is applied to the weights of the lower levels of the mind map until the last diagnostic level. A numerical example of the aggregation of three ANSPs' mind maps can be found in Tables 3 and table 4.

Weights
Step 1 Develop ANSP specific mind maps Step 2 Define the driver for merging For this case we consider the traffic count as a driver. Let us consider that ANSP 1 has 60% more traffic count than ANSP 2. This means that the existing weights of ANSP 1 will be inflated by 30% and the ANSP 2's weights will be deflated by 30%.

Weights
Step 1 Develop ANSP specific mind maps Step 2 Define the driver for merging For this case we consider the traffic count as a driver. Let us consider that ANSP 1 has 60% more traffic count than ANSP 2, and 20% more than ANSP 3. This means that ANSP 1's weights are inflated by 40%, ANSP 2's weights are deflated by 50% and ANSP 3's weights are inflated by 10%.

Conclusions
The ultimate purpose of the APF is to provide decision makers with a picture of safety performance that yields actionable knowledge. The purpose of the APF is achieved through: -the provision of a trended view of safety performance over time, -the drilldown feature of the APF display which allows decision makers to identify likely reasons for excursions of the APF values outside of predetermined acceptable performance limits, and -a methodology that allows decisions makers to better exploit the resources using this knowledge. The APF vision for the future stands for the classic simplicity of the tool, more information and less noise, full scalability, utility, efficiency, and relevance as a decisionmaking tool. This paper is a brief analysis of the current APF and advances a number of solutions for its enhancement and development. Both proposed solutions are adding value , 0100 (2018) 2018 https://doi.org/10.1051/matecconf /201823601007 MATEC Web of Conferences 236 LOGI 7 without compromising the current features. When the new solutions imply transitions to alternative ways, the adoption of the new way remains optional to the user and the transition may be done smoothly and gradually. The APF tool proposes many other potential solutions of the use for safety benefit of different organizations in aviation field.
The aerospace performance factor represents a new methodology that moves away from performance measurement that relies on a single or limited number of measures.
Instead, the APF aims to aggregate multiple operational safety risks, expressed as the sum of reported incidents, in a weighted manner, into one value that can vary over time. Although this unique value gives the overall risk, the methodology allows it to be broken down into its component parts for the analysis of specific causal factors and lessons learnt.