Risks Associated with Aircraft Maintenance in European Commercial Air Transport

In the 2016 EASA Annual Safety Report, 1318 incidents were categorised as involving a safety issue related to aircraft maintenance. All the Key Risk Areas, or most frequent outcomes, for aircraft maintenance as a safety issue were shown to be related but associated Rulemaking Tasks have not yet appeared in the 2017-2021 European Plan for Aviation Safety. Categorising EU-wide incident reports appears to be difficult due to the use and construction of the ‘Event Types’, or contributing causes, taxonomy used in the reporting tool. Previously developed taxonomies are either limited in the number of categories due to limited information in reports, or they are aimed more at an investigation than straight reporting. As a result, these taxonomies are not suited for use in a reporting tool, and these taxonomies lack the detail for focused feedback to the maintenance industry. Thus, a two-level taxonomy has been developed within this study, using a high-level overview of the maintenance process as a basis for the first level and a variation of descriptions, both specific and more broad-spectrum, for the second level. A set of 3910 incident reports from the central repository were analysed, causing 2677 incidents to be excluded for a variety of reasons such as a lack of information (the narrative was empty in 1100 reports or 28%), or the use of a language other than English (814 incidents, 21%). The remaining 1232 incident reports were categorised using the new taxonomy. Of these, 361 incidents occurred during installation of components (29%) and 308 incidents were related to maintenance control (25%). The increased level of detail from the new taxonomy allows for identification of procedure types that are more susceptible to errors, such as installing or removing components or deferral of a defect so that an aircraft can be dispatched. This will hopefully lead to increased vigilance as well as further measures to increase safety.


INTRODUCTION
The remarkable success the commercial air transport industry has achieved is undeniable. The global, regional and national level safety and risk management initiatives are achieving results. In Europe, the Safety Risk Management process defined in EASA's Annual Safety Review (ASR) consists of five steps (Figure 1). The ASR also includes updated versions of the safety risk portfolios in different domains including "Commercial Air Transport (CAT) -Aeroplanes". Identification of safety issues requires a detailed analysis of statistics about accidents, incidents and occurrences reported by the industry. The European Central Repository (ECR) stores incident data supplied through the mandatory and voluntary reporting schemes (ECCAIRS, 2017). This data, along with an internal occurrence database, is used in yearly Annual Safety Reviews published by the European Aviation Safety Agency (EASA). In the 2017 ASR (EASA, 2017a), the safety risk portfolio for Commercial Air Transport (CAT) includes 'Aircraft Maintenance' as one of the safety issues. This is based on the fact that for the preceding 5 years, there are 1866 aircraft maintenance related incidents in the ECR, of which 9 were serious. Additionally, the review of accidents within the last 10 years indicates there are only 5 non-fatal accidents attributable to 'aircraft maintenance', and no fatal accidents exist. Unfortunately, the review does not provide any further detail about which areas/factors within Aircraft Maintenance cause these incidents.
The review also includes a list of the top safety issues within the CAT domain, but maintenance is not included. All the issues listed are operational, and these can be linked to some of the risk areas set out in the European Plan for Aviation Safety (EPAS) -2021(EASA, 2017b. The EPAS sets out the rulemaking and safety promotion tasks to mitigate all identified safety risks. Although one of these tasks is labelled as 'Design and Maintenance Improvements', currently there are no maintenance related actions clearly targeting the causes of previously mentioned accidents and incidents. In the 2016 ASR (EASA, 2016) the safety issue 'Aircraft Maintenance' was linked to all eight available key risk areas, including issues such as 'Airborne Conflict'. This led the authors to believe that although a lot of data is available within the ECR and EASA databases, more research could be applied to translate this into more specific maintenance-related safety issues, if necessary leading to both rulemaking and safety promotion tasks within EASA, and safety promoting tasks within the maintenance industry.

ANALYSING ECR INCIDENT REPORTS
A download from the ECR containing 7158 incidents related to aircraft maintenance was obtained from EASA in September 2016. The download covered incidents that occurred between January 2012 and December 2016 and only included a date and aircraft type without any further identifying data. The full narrative was included, along with two categorisations in separate worksheets: 1. Occurrence Category (Outcomes) 2. Event Types (Causal and contributory factors) As more than one categorisation could be selected per event, matching the basic data and narrative for an incident with the correct categorisations turned out to be problematic. The data did not include a common indicator that could be matched across the three worksheets. In some cases, multiple incidents occurred on the same day with the same aircraft type involved, leading to some uncertainty in the allocation of the correct categorisations. With the help of some programming and organising in Microsoft Excel a subset (3910 incidents out of the complete set of 7158 reports, the remaining 3248 reports have not yet been evaluated due to a lack of available time) of the data was evaluated and this led to these initial findings: 1. 22%, or 1556, of the 7158 incidents could not be evaluated as the narrative was empty. Within the subset of 3910 incidents, this category included 28% or 1100 incidents. 2. 21%, or 814, of the evaluated 3910 incidents included a narrative written in a language other than English. 3. Other incidents were excluded from further evaluation for issues such as a lack of information (8% or 303 incidents), or for being related to other aspects of aircraft operation such as ramp handling instead of continuing airworthiness (5% or 193 incidents). 4. In total, 68% or 2678 out of 3910 incidents, were excluded for various reasons.

EXISTING TAXONOMY FOR CAUSAL AND CONTRIBUTORY
The taxonomy used to allocate Event Types to the incident reports was partially reconstructed based on the complete set of 7158 reports in the download. This taxonomy uses four levels, covering every possible incident within the aviation industry, including but not restricted to: aerodrome related incidents, air traffic control problems, incidents that occurred during air transport flights, etcetera. Within this taxonomy, the subject of aircraft maintenance is not mentioned until the second level, and within this category, only eight subdivisions are available in the third level. The most useful level to categorise maintenance events is the fourth, with 53 subcategories, but in many cases, this is not used. Out of 14055 Event Type records (on average two are used per incident report), only 5046 or 36% use a category based on the level 2 subdivision Aircraft Maintenance, with only 2704 (19%) using this to the fourth level.
For the remaining incidents, the reporter has only used the first two or three levels of the taxonomy, which severely restricts the level of detail available from this categorisation. The taxonomy also includes a very extensive breakdown into ATA † chapters and related technical problems, which is used in 25% of these records with only 6,5% using this to four levels. Many of the other various categorisations are used to include related outcomes and operational issues relating to the event. Although useful in many ways, the level of detail available to analyse what went wrong during maintenance was lacking in the majority of cases. This led to the decision to use a different taxonomy and (re)categorise incidents based on the available narratives.

NEW TAXONOMY
Several taxonomies were evaluated for use in this project, such as: 1. CAA Paper 2009/05 (CAA, 2009) † Air Transport Association (ATA) -100 system: a six-digit numerical system that is used in aircraft maintenance documentation as a structured breakdown of the aircraft structure and its systems.

CAP 1367 (CAA, 2015)
3. MEDA (The Boeing Company, 2013) 4. HFACS-ME (Schmidt et al., 2000) According to Wiegman and Shappell (2001), the following points should be considered for a good taxonomy: 1. Reliability 2. Comprehensiveness 3. Diagnosticity 4. Usability 5. Validity To be able to use the results from categorisation to identify risks during the maintenance process, comprehensiveness was selected as the main priority. As a second priority, usability was selected so that the new taxonomy could be directly applied as part of a reporting tool. The issues with the existing taxonomy indicated room for improvement in this respect. The reliability aspect of the new taxonomy will be addressed during validity testing (see paragraph 6) and the diagnosticity aspect is one that is less relevant in this case as the taxonomy will be used to analyse the 'what' and not the 'why', due to the lack of detailed information in the reports. According to Wiegman and Shapell, the validity of the resulting taxonomy is determined by the extent to which it meets the other four criteria. Existing taxonomies are geared primarily towards either retroactive analysis of reports and because of this were limited in their comprehensiveness (CAA 2009/05 (CAA, 2009) and CAP 1367 (CAA, 2015)), or they are aimed at investigating the event itself to identify underlying causes using tools such as MEDA (The Boeing Company, 2013) or HFACS-ME (Schmidt et al., 2000) ‡ . To be able to provide both comprehensiveness and usability, a new taxonomy, based on a high-level overview of the maintenance process, was developed. Table 1 shows the first level of this taxonomy, including the origins of the category descriptions. The more comprehensive second level is included in the Appendix.
The first level descriptors evolved from a combination of previously mentioned taxonomies, existing structures in aircraft maintenance documentation and maintenance development and analysis of the various incidents used in this study. The category '1. Maintenance Control' encompasses all the tasks used in scheduling and organising maintenance. Categories 2, 3 and 4 are supporting structures that need to be in place before maintenance can be carried out and the remaining categories are based around a typical maintenance task as set out in aircraft maintenance manuals. This always starts with 'Job set-up', ends with 'Job close-up', and in between can be one or more of several different tasks (categories 6 to 12). Because of this, these categories will be immediately recognisable for aircraft maintenance engineers, technicians and other personnel. One significant compromise during the construction of this taxonomy is the use of both broadspectrum categories, such as '6.4 Incorrect procedure used or procedure applied incorrectly' (see Appendix), and more specific categories. This was done to allow for the use of reports with a limited narrative, as well as more comprehensive narratives. The main risk is that some reports will fit in a specific category, as well as one or more broad-spectrum categories. As the coding was done by a single researcher, selecting the most specific category for each incident, this research has attempted to avoid lumping a lot of incidents into a limited number of broad-spectrum categories. For use in a reporting situation, this deficiency will have to be remedied. One solution would be through highlighting the broad-spectrum categories in the list and instructing reporters to attempt to avoid these as much as possible. Within this research, 1232 reports have been categorised using the new taxonomy out of the subset of 3910 incidents, showing several significant issues.

RESULTS OF CATEGORISATION
The distribution of first level categories for the 1232 incidents are shown in Table 2. This shows that installation/removal issues, as well as maintenance control related issues are the two highest scoring categories. 13 Job close-up 130 11% § Maintenance Steering Group (MSG)-3 structure: A methodology used to develop scheduled maintenance tasks and intervals to satisfy regulatory demands on aircraft maintenance and reliability.

Percentage of categorised incidents
Total: 1232 100% The distribution of the level 2 descriptors ranked by the number and percentages of incidents shows that out of the 69 categories, the top 10 categories are headed by '10.7 Installation/removal incorrect' and '1.10 Defect deferred with incorrect procedure/reference/follow up' with 16.2% and 9.0% of the 1232 incidents respectively. The complete top 10 is shown in Table 3. Table 3 Top 10 of categorised incidents, ranked by level 2 descriptors (see Appendix for complete taxonomy and aviation specific abbreviations) The numbers in Table 3 show that after the first three lines, the percentages do not vary all that much. Extending this ranking over all the level 2 descriptors shows that between sixth place and twenty-eight place, the percentages vary between 4.1% and 1.0%, which indicates that a lot of incidents are spread quite evenly across the various categories.

VALIDATING THE METHOD
To validate this method of using a new taxonomy to categorise maintenance related incident reports, a survey was distributed to a group of over 20 Subject Matter Experts (SME). Completed surveys were received from six SMEs, with each participant categorising or excluding 25 sample incident narratives. This provided 138 data points (not counting excluded incidents), with the results as shown in Figures 2 and 3.
For 51% of the data points, the SMEs selected the same level 1 category as the one selected during this research. Out of this 51%, 61% of the data points also showed the same level 2 category. Looking at the complete set of data points, 31% of the points showed both level 1 and level 2 categories matching the ones selected during this research. The SMEs were asked to grade how well the category they selected described the incident in question, on a five-level Likert-type scale. They then also graded the Event Type classification that was included with the original reports on this same scale. Figure 4 shows the results of these two opinions.

No. Issue
Number of incidents   Figure 4 shows that for the original Event Type classification, the opinions vary greatly with most of them scoring between '1-Not At All' and '4-To a Considerable Degree'. The new taxonomy descriptions score mostly between '3-To a Moderate Degree' and '5-To a Great Degree' with 56% of 125 responses showing '4-To a Considerable Degree'.
Another comparison was created by coding the 69 level 2 descriptors of the new taxonomy to correspond to the three first level categories of the taxonomy described in the CAA 2009/05 paper. In this way, a relationship between the results from this study and the outcome of the CAA study could be shown (Table 4).  Table 4 shows that in both studies, most of the incidents are attributable to Incorrect Maintenance Action. Maintenance Control is the second highest ranking category with the remaining incidents classed as Incomplete Maintenance. A similar comparison was carried out with the data from the CAP 1367 study, but lack of information on the category definitions of the taxonomy used in this study prevented a clear outcome for this comparison.

MAIN CONCLUSIONS AND RECOMMENDATIONS
This project set out to show that maintenance-related Safety Issues could be derived from ECR incident reports on maintenance related events. This appears to be the case, but several related conclusions also emerged, and this project also clearly established several areas that are suitable for further research. ** For 6 (or 0.5%) of the evaluated incidents, the correct category within the CAA 2009/05 taxonomy could not be determined. • The existing Event Types taxonomy from the EASA database appears to be used inconsistently, with reporters not selecting the more detailed fourth level of the taxonomy in a majority of cases. This severely restricts the level of detail available from this data. • Available taxonomies from previous studies or investigation tools were either too restricted to properly describe the event clearly or they focused too much on the underlying human error related causality to be useful at the reporting stage. • The new taxonomy enabled several recurring topics to be identified, which appeared to align with results from earlier, related studies such as CAA 2009/05 or CAP 1367. • The inclusion of broad-spectrum categories in the new taxonomy did not hamper the identification of safety issues, although for use in a reporting situation instead of retroactive analysis, these categories may cause reporters to provide less detail than possible. • The usefulness of ECR incident reports may be improved by insisting on a detailed narrative for each report, and standardising on one single language. Also, the current four-level taxonomy used to include Event Type descriptors appears not to be used to its full capacity. • The aviation Maintenance, Repair and Overhaul (MRO) industry could increase safety by putting more focus on the correct use of procedures during the various installation and removal tasks in maintenance programmes. The procedures for deferring an acceptable defect could also be reviewed for clarity as this category is another high-scoring one. In both cases, the narratives for the related incidents might be used to investigate common factors further. • Further research into the distributions of incidents over time or the relationship between the number of incidents and specific aircraft types is also possible and might be used to narrow down recommendations to the MRO industry further. The limited fields available in this data download prevents analysis of variations between countries or regions. While these details are available in the ECR, only anonymised data was used during this study.