Safety differently: A case study in an Aviation Maintenance-Repair-Overhaul facility

. You This paper presents the findings from a ‘Safety Differently’ (SD) case study in aviation, and specifically in a maintenance, repair and overhaul (MRO) organisation in Southeast Asia. The goal of the case study was to apply a new method of safety intervention that is part of the Safety Differently toolkit and utilises a bottom-up approach. This research tested the extent to which these interventions could be embedded into a continuous improvement program in a highly controlled environment, namely an Aviation MRO. The interventions (called micro-experiments, ME) are considered as a flexible tool, which allows testing of process improvements in a safe to fail way, empowering the lower levels of the organisation, challenging safety related issues and revealing key areas in need of transformation. The ideas for the interventions considered in the case study were retrieved from interviews conducted with 50 mechanics, and include issues to address aviation safety and occupational health as well as quality. We elected to include all three categories in this study as the ME approach is applicable to all of these. This MRO case study showcases the benefits and limitations of the ME in aviation, revealing the conditions under which it may become useful. Future studies should further explore the role of complex and heavily controlled industries in similar bottom up approaches, so that interventions can become part of a continuous improvement plan.


Introduction
Safety Differently is a new, yet under conceptualised, approach to improving safety processes by using the experience, knowledge and creativity of the frontline employees [1].As part of the SD toolkit, micro-experiments (ME) have been introduced as a way to test the feasibility of interventions in a safe-to-fail manner.Maintaining aircraft to the highest safety standards requires skills of the mechanics within a Maintenance, Repair and Overhaul (MRO) organisation.Aircraft mechanics perform daily tasks, in demanding working conditions and whilst following procedures, designed to comply with safety and quality standards and regulations.Hence, given their subject-matter expertise, MRO personnel is a valuable source for the detection of issues experienced at the shop floor.Since it is important to continuously improve processes, this paper focuses on interventions utilising the most valuable source of an MRO organisation, its workforce.The MRO mechanics are key in sharing problems from the shop floor and help locate the potential interventions required to address issues affecting safety.Little is known about the results of the ME approaches and the added value to the continuous improvement of safety.Following the principles of Safety Differently (SD), we, therefore, explored bottom-up improvements through testing of the ME-based interventions.Our motivation stemmed from the lack of applications in highly controlled and complex risk industries, such as aviation.Aviation is an industry characterised by complexity, high severity risks, and consequently a strong regulatory framework.Any such ME case study or application in aviation is missing.
It was a challenge to examine the possibility of the ME application in an aviation organisation because the study must deal with strict regulations, whereas the methodology requires the elimination of constraints, possibly violating aviation rules.Our approach aimed to assess the benefits of ME in high risk industries and safety critical tasks [2] in two phases.The first phase involved data collection to a) identify gaps in the safety of the work processes and b) to design the subsequent potential interventions.The qualitative data was collected through a change-oriented research design, achieved by interviewing 50 mechanics who were the primary source according to the bottom-up approach of ME [3].The second phase was the experimentation of a limited number of appropriate interventions, in line with previous applications of the concept outside aviation.The interventions were designed for their smallscale application by using the organisation's active work area and employees as an in-situ laboratory [4].The project began in the base of the organisation with two university representatives who stayed in the MRO's base for months.The university agreed with the organisation about the project constraints, described in detail in 2.3, and the working progress of the project in the organisation's area and bays.In the following sections, we report the extent to which the SD-based methodology through the execution of ME could be used as part of a continuous improvement program within MRO organisations.

Safety Differently: concept and methodology
Sidney Dekker defines the Safety Differently (SD) concept as following: "Imagine a space, a space in which there are no rules, a space in which people can themselves determine the best course of action, a space in which people spontaneously negotiate and collaborate in order to create the safest outcomes for everyone, in this space a new type of humanity emerges, nobody is telling them, they figure it out on the spot, we call this emergent behaviour" [5].In a complex environment, where behaviour is difficult to predict, we cannot be sure that the interventions that we devise actually bring the results that we envisage, or that unintended effects are avoided.We need an approach that allows us to try different things, does not cause peril, and can be stopped if we don't get the results that we want.This is where the micro-experiments (ME) come in.
In an MRO organisation, ME can be applied if the organisational safety constraints are removed in order to accommodate the bottom-up research design, examining how the mechanics have to maintain safety in their tasks and operations.Typical constraints in an MRO setting mean that wide-scale interventions are not suitable, mainly due to the number of stakeholders involved, such as airlines and regulatory bodies.An intervention can act as a safe-to-fail, small scale project, which uses the organisation's workplaces and workforce [6].The subsections that follow discuss how the micro experiments are selected and how they were applied in the MRO organisation.

The ME concept
Captions Traditionally, aviation safety management is considered as a hard, positivist field of research [7].Safety Differently involves taking a more collaborative approach to safety.It involves getting the people who do the work to decide on rules and safe working practices.This is different from the traditional approach to safety where rules and policies are set by managers and other third parties before being communicated to (or imposed on) the people doing the job.ME fit into this approach as they are safe-to-fail interventions that are monitored for success or failure, and that have been generated by the work-floor.ME are different to ad-hoc trial-and-error interventions in that they are planned before they are executed, and carefully monitored to see whether the intended results are realised.ME are different to projects in that projects are not supposed to fail, whereas in micro-experiments are temporary and we allow for unexpected, even disappointing results.In a complex environment we cannot have the (misplaced) confidence that any intervention will surely deliver the predefined results as expected.We need to allow for the possibility for the intervention not to work without causing undue damage.In fact the experiment can be considered a success if we realize an intervention "fails" and that some other solution is required.We therefore need to be able to stop or slow the experiments.On the other hand, if all goes well the intervention is ready to be formally implemented and possibly enlarged on a wider (geographic or organisational) scale.Naturally we will create the ideas for the microexperiments in close collaboration with the task executors; many ideas will derive from them.
Micro-experiments generally require temporary safeguards because we will be deviating from existing processes, perhaps violating the constraints that are usually in place.Care needs to be taken to identify and only maintain those constraints that are absolutely vital to safeguard against unacceptable risks -we value the input of safety experts here.The criteria by which we determine whether an intervention is successful (or not) need to be determined ahead of time, although additional insights will occur as experience is gathered.Interventions that are deemed successful need to be incorporated in the next version of Work-as-Imagined.Even after formalisation, we remain vigilant about unintended effects, expecting the front line to report where exceptions are necessary.Multiple micro-experiments can run in parallel, preferably somewhat isolated from each other.Good targets for micro-experiments are procedures that are not delivering results as expected.Many organizations that we first engage with are susceptible to large and potentially embarrassing gaps between Work-as-Done and Work-as-Imagined, that are not straightforward to close.This is often a good starting point to try out micro-experiments.Other reasons to execute micro-experiments include adapting rules to changes in processes or equipment or for innovation.To further define the problem, we first need to identify the "job to be done": what operators really seeks to accomplish in the given circumstances.Successful micro-experiments help make the job safer and easier.

Methodology
Our methodology involved a sequential research design, comprising of two data collection phases.In the sections that follow, we initially describe the data collections phases for ME.Following this, we showcase the utilisation of previous findings in our data analysis.Before we present the results of the ME application, we describe the pre-coded themes and the analysis framework.Lastly, we discuss the limitations of these choices.

Research design phases
As outlined previously (cf.Section 1), the first phase involved the collection of data and the second phase regarded the interventions.The first phase included identifying and interviewing mechanics in semi-structured interviews.In order to identify the interviewees, the team requested to be present at the organisation's bays and observe the mechanics.Through this process, fifty (50) mechanics were purposively sampled and were interviewed individually.The main objective of this phase was to identify what gaps are prevalent between WAD and WAI.This data collection phase was supplemented with archival data from the MRO, including documentation from audits, investigations reports and safety reports, as well as injury reports and maintenance error notifications.Following this phase, the team identified a number of interventions, which formed the second phase of the study.The aim of the interventions as interventions was to close the gap(s) found through the findings from the first phase.During this phase, it was decided that the term interventions was changed to micro-projects following the organisation's requirement and in order to limit possible defensive behaviours that would affect results.

Coding and analysis framework
MRO organisations are typically optimised by lean approaches in order to create more value for their customers by utilising fewer resources [8].Our approach hence initially required an assessment of the MRO environment.Before our study commenced, the quality (and safety) department at the MRO, had initiated a safety project to assess their safety performance.Elements of the latter are observed in the organisation's vision for the safety project itself, striving for innovation and evolution, and finally, Genchi Genbutsu, going down to the shop floor to fact check one's ideas in the 'real world' [8].The findings of the project led to the establishment of a Safety Group (SG), encompassing the following pillars: • Technical Systems: includes all the tools, equipment, and all the physical items in the organisation, necessary to create value for the customer.• Management Infrastructure: includes the formal structures, the processes and the systems for controlling the resources, in association with the technical systems.• Mindset and Behaviour: include the employees' thoughts and feelings as well as individual and group behaviour at the workplace.
These three pillars formed our initial classification that we used to compare the SD results with the organisation's SG project, and hence led to a pre-coding of the issues found at the shop floor.All the sources used from the organisation's safety project was provided to the team by the SG.The coded shops, which were chosen for phase one observations and consequent sampling and data collection (Table 1).In the second phase, the researchers collected data, mainly from the interviews in which the front-line mechanics assigned.These technicians answered the questions posed to them and communicated their ideas for possible interventions.The findings from the interviews and observations led us to a master coding according to shop floor and their bays.In addition, the MRO's project used numerical values to score and classify findings following the Safety Risk Management (SRM) manual, which meets the compliance requirements according to ICAO's Safety Management Manual (SMM) and the data driven systems to proactively manage safety, Safety Management Systems (SMS).The MRO uses numerical values for the risk and taxonomies according to SMM standards [9], based on the Severity (minor to hazardous) and the Probability (extremely improbable to frequent).The Risk Indicator (R is calculated by multiplying the probability (P) with the Severity (S) of an occurrence (S), forming the R = P x S equation.The result of this equation rated from low to high risk and the values for the possibility and the severity explained detailed.The SRM was used during the project to score and classify the findings.

Research rigour
The team took a series of steps to minimise bias, gain access, and to overall ensure rigour in the research process.The mechanics were interviewed to gain information based on their expertise and bypass the organisational-level hurdles.By doing so, the feeling of empowerment of the mechanics increased, placing emphasis on the value of the technical and operational experience of the mechanics.In addition, we described the interventions to the organisation management to gain internal approval, allow execution of the interventions and to peer-review the results.Following this, and regarding the micro-projects, acceptance by management was ensured to reach consensus on the interventions proposed by the mechanics.After suitable interventions were collected, the gathered intel has to be verified by the rest of the mechanics, since implementing an idea that only suits one mechanic is not a valid microexperiment.Besides validating with the mechanics, it is also important to validate the opportunities of improvement with the quality management representatives to make sure they agree on the findings and see the same promising interventions for current tasks or processes.In case there were multiple possible interventions, the mechanics scored them on an efforteffect diagram (Figure 1) together with the Quality Manager.All the proposed interventions were compared with each other in assuming the effort required to achieve each intervention and its expected effectiveness.An example is shown in Figure 1, where intervention "1" was preferred because it could yield the most effective result with the least amount of effort when compared to intervention "2".In general, interventions mapped on the top-right area of the matrix were preferred over the rest.[10] To gain further approval, the micro-experiment approach was described to management in detail, covering points such as the process, success criteria, constraints and anticipated interventions as well as the area and possibilities of amplification.Regarding the execution of the interventions, each micro-experiment had a customised period.The main factor was the duration of the task each intervention targeted.One researcher was continuously present during the execution of each experiment to observe the intervention, function as the point of contact for concerns, questions, ideas, and, most importantly, collect the views of the mechanics on each intervention.When the predetermined timeframe expired, the experiment was concluded by the researcher, the intervention was suspended, and the task was brought to its previous state (e.g.tools, procedures).A final template was created with the approval intervention when it validated from the quality management department.The decision on whether to amplify each successful intervention into an improvement plan laid with the organisation's management team.

Constraints
When the project was being planned, and following the principles of ME, the team observed that there are several limitations when applying ME in an aviation organisation.The chosen MRO organisation allowed the researchers time to organise and plan the project locally, but the quality manager and the safety group required the following constraints to be applied to the ME's: • Any actions must be according to the regulations • Any intervention must be according the procedures • Any interaction cannot create new risks • Any intervention is according to the organisation's risk assessment • Any intervention is not a permanent solution • Any intervention must comply with requirements set by organisation's clients • Any intervention is only applied to the predetermined bay with predetermined group • Any intervention will run for fixed period of time • Any intervention can only be executed with the present of at least one of the university's representative on site.• Any intervention can only be executed in agreement with Quality Manager, VP, the bay manager involved and the project team.
The limitations deriving from the application of ME in a highly controlled industry quickly became the first important finding of the case study.It is clear that the first three constraints (any actions must be according to the regulations, any intervention must be according the procedures, and any interaction cannot create new risks) do not allow any innovation to take place.These, as well as the required permission to execute the ME's, severely impeded progress at the site, and needed executive circumvention to allow the ME's to be applied.Although in the ME concept the interventions are described as a bottom up process, the validation, approval and decision processes in aviation are a top down process with assumptions and issues including not merely safety aspects but also extending to other departments like security and quality.The discussion about the limitations follows the presentation of our results.

Results
This section presents the findings from phases one and two of the sequential research design applied following the principles of the ME concept.Table 2 below provides an overview of the amount of data and of the sources involved in phase one of the study (i.e.interviews and archival data).Interview findings were initially categorised in single or multiple concerns.The categories emerged from the narratives of the research as presented in Table 2.When a finding was a single concern, it meant one mechanic only raised it; such issues were either considered minimal, according to the QM and the SG, as general remarks or seen as irrelevant to the organisation.Valid concerns were findings reported by multiple mechanics, and issues on which the organisation agreed, but had its intervention planned, processed or implemented (11 in total).In addition, findings were categorised according to the bay coding (cf.Section 2.2.2) as presented in Table 3.As shown below, the findings were also categorised according to in-and on-the aircraft (in A/C, on A/C), and were summarised for each of the shops.Bay 6 is not included in the study.The shops that are not necessarily linked to all bays.For example, the landing gear shop (LDG), is linked only with the bay 5 and bay 8.In total, sixty-nine findings (69) were recorded.However, ten of these findings were finally selected as interventions, through the validation and approval process as described previously (cf.Section 2).The accepted findings as interventions are described in detail in 3.2.

WAD -WAI analysis
During the analysis of the interview data, it was observed that the mechanics were hiding the gaps between how they should do their job (WAI) versus the way they perform their tasks (WAD).Hence, we observed that the MRO organisation manifests signs of a problematic safety climate.These limitations are further discussed in Section 4.

Problem description
Intervention description The mechanics are not allowed to wear their safety shoes during final maintenance in a cleaned cabin, prepared for delivery to the customer.However, mechanics often make last-minute repairs in the cabin, in which the use of safety shoes is recommended.
Three types of covers were tested and a silicon, flexible shoe cover was selected due to ease of use, grip provided and antistatic properties.The experiment proved useful and QM extended the shoes throughout the organisation by adding them to the personal protection equipment (PPE).
Following installation the seat covers are often damaged and repairing them usually means delays in customer acceptance and is costly.
For the business class seat covers, the intervention is a fabric, reusable seat cover.
For small tools, such as a drill bit, a expendables form needs to be filled in at the tool room booth, making the tool keeper wait for the completion of the form.
The filling in of the form elsewhere can increase the throughput of the booth and therefore reduce the current queue time.
Under the wing of Airbus aircrafts, a strong presence of kerosene fumes can be noticed.
Fume-removing fans could be used to circulate the air.A risk, however, is that the fumes move to other work areas.
Current methods of working on the fuselage or wings of the aircraft can be dangerous to mechanics and can damage the aircraft.
Butterfly Suction Cups: A new system, in the shape of a butterfly, was proposed as an improved and safe alternative.
The safety shoes at are currently used by mechanics are uncomfortable and are of low quality.
The introduction of a new type/brand of safety shoes has been proposed.
The extreme hot climate working conditions in Southeast Asia.
To address the tropical working conditions, lighter shirts made from a dry-fit fabric, have been proposed for the mechanics.
When a tool is missing from one toolroom mechanics often walk to the other toolroom a few 100 meters away with timeconsuming.
A transportation means such as a bicycle can significantly reduce the time needed to get a tool from a different tool room.
Everyday mechanics have to retrieve their standard toolboxes from the toolroom, which creates long queues at the start and finish of the shifts.
The Relocation of the toolboxes back in the shop area was suggested, and only using the toolroom as for original and limited equipment.
Inside the cabin, mechanics use stairs made from foam and tape, these are unstable.
A more firm and secure stair was proposed as a safer equipment for the mechanics.

Discussion and Recommendations
The SD principles guided the application of the Micro Experiments in a bottom up process to identify the problems and the solutions for safety related issues.The research objectives focused on exploring whether the application of the ME approach could trigger interventions for improved safety processes in an aviation organisation.Our two-phases methodology following the principles of the ME in an aviation organisation, revealed the benefits and the limitations of the approach in a high controlled and operationally risky environment.Below we discuss the key findings from the ME case study at the MRO organisation noting our contribution.

Occupational Health, Process Safety and Quality overlaps
One of the critical realisations during the project was the relationship between safety and quality management.Through the research, the team realised that the constraints and the limitations in an aviation organisation are strong barriers for the ME application as described and applied elsewhere.The lack of case studies in aviation meant that the research would enter an under-explored field, which created challenges and sometimes misunderstandings between the researchers and the management of the organisation.We therefore report that the ME assumptions are challenged by the organisation's quality and safety department, which descriptions of ME do not account for.For example, the MRO's quality department attempted to assess the safety findings and observations as quality markers.Our findings here highlight the problems in the quality vs safety and occupational health misperceptions, and how these create confusion in the safety management process and the identification of suitable interventions for the organisation.Secondly, the safety department initially raised constraints that paralysed the ME and needed to be lifted by top-level intervention for progress to be made Albeit, the revelation shows that the ME approach can reveal deeper organisational issues intervening in the safety processes, whether managed or performed, and supports the benefits deriving from bottom-up approaches [3].Because of these benefits, mid-project interventions were able to take place.Initially, the team was able to recognise the lack of ME fundamental knowledge and thinking in the MRO organisation.Secondly, the team was able to address this issue by filling the knowledge gap with ME workshops and additional visits on-site the MRO.The intermediate actions aided to have most of the interventions accepted from the organisation, yet, as reported in Section 3, with quality and occupational health aspects, and not a clear safety orientation.The same misconception was identified in the interviews where the participants wanted to discuss and proposed quality and not safety interventions like the cover of the cabin seats during the work of the technicians in the aircrafts' cabin.This finding and change process underpins the added value of ME and the intervention possibilities in aviation organisations.

Insights following the SD principles
Using the ME approach to inquiry, the findings from interviews in interventions achieved a variety of results and insights into the adoption of the boader SD principles.Concerns such as middle management isolation, fear of change, and not wanting to take responsibility, revealed problems of safety bureaucracy and showcased the possible issues these can induce during a (research) project and in the safety processes of an MRO organisation without directly impacting the ME.The mechanics also reported that there are even cases in which they were demoted after they made critical remarks about the organisation and/or shared innovative ideas.There are clear indicators of the presence of these types of bureaucracies within the organisation.Our ME findings hence support studies highlighting the importance of safety climate underpinning aviation MRO practice [11].Importantly, our study showed that, when the organisation shows value and appreciation towards the work of the mechanics, their opinions and ideas, the attitude towards responsibility changed.Specifically, the MRO mechanics were appreciative of the responsibilities and chances are given to them to be part of a change-based project.It showed them the importance of bottom-up suggestions and challenging of organisation procedures.The interventions provided the flexibility required to fit and try a wide selection of interventions.The first completed interventions showed the amount of effort mechanics put in their ideas and the results they can achieve from this input.The fact that the shoe cover intervention (Table 5) will be amplified organisation wide is a first step towards the continuous improvement.

Evaluation of the MRO project
During the planning and execution phases of the project, several success factors and limitations were revealed.Success factors include the MRO's availability and willingness at a high level to run the project in their working area.The decisions the team took in the latter environment, aided in achieving research rigour and insights for the transferability of the results [12].In detail, the team followed the data to make decisions instead of aiming to satisfy standards not relevant and not appropriate for the project (i.e.external and internal validity, reliability and objectivity).In addition, the researchers' professional background in aviation organisations helped establish a common communication channel with the MRO managers, gaining access into their MRO systems.Hence, familiarity with the culture of the participants aided reflexivity [13].As a bottom-up process underpinned the project, the developing cooperation between managers and mechanics became evident.This success factor showed the power of the ME approach and the advantages over traditional intervention approaches (ad-hoc or projects).
Limitations of the project include the chosen context of the case study, hence addressing our main objective.In detail, in the highly controlled MRO environment, the more inexperienced members were not able to recognise possible safety-related issues, which would have been useful for the project process.Moreover, the inexperienced members unintentionally behaved like auditors, adding to a lack of trust already in the MRO, and revealing problems of safety climate.Similarly, the managers were unprepared for this type of project.Our findings hence support past aviation MRO safety studies, where safety climate is seminal to developments.Additional limitations were the context of safety bureaucracy, which surfaced organisational problems such as the lack of a positive safety culture, the lack of trust and the lack of safety knowledge in the middle and the low-level employees.For example, during interviews, it became evident that most employees are confused between the failure analysis and incident investigation.

Conclusion and future directions
Safety Differently in aviation organisations is an underexplored field.Micro experiments are not safety recommendations, but they comprise a tool to explore the perceptions of shopfloor employees that guide actions and behaviours.In order to formulate interventions, incident reports can be used, although it should be noted that incident reports do not trigger them.Incident reports can be used as a tool to identify focus areas, assess the status of the safety recommendations and close the loop.In addition, it is recommended that, although the micro experiment is a bottom-up process, the organisation is assessed before any safety differently project can begin.Our case study application revealed the bottle neck situation which forms our contribution to further ME applications in a similar aviation organisation.Our recommendation is hence to initially assess the organisation's safety climate and then to apply the framework for the Safety differently project.The assessment of safety climate is fundamental for the SD -and therefore ME -implementation in aviation.It is a way to plan the researcher's actions before the application of any method and tool of SD concept.The preparation of the safety program participants is an additional necessary action, which includes top-down training and identification of the setting the micro experiment takes place, so that daily working load is not affected.They are no potential risks for the organisation's operations.Moreover, it is recommended that the organisation prioritises the organisation's problems, using quality tools and methods (e.g.Effect -Effort, Pareto, Fishbone diagrams).
In any aviation organisation which has many working tasks, the term 'micro experiments' is not a familiar one and could, therefore, threaten their acceptance.It is therefore suggested that the term changes to 'micro-projects interventions', which is a less threatening term to aviators.Future research should hence focus on tactics for the smoother application of the micro-project interventions can give opportunities in organisations to prepare, plan and organise better large-scale projects in the future.With the use of interviews at the shop floor, new findings and interventions can be collected, executed and possibly amplified throughout the organisation.However, mechanics will have to be empowered to share their concerns and ideas openly.Addressing safety bureaucracy problems will not only transform the intervention to an improvement plan, but it can help enhance wider safety practices (e.g.safety promotion, reporting).Safety Differently and ME can, therefore, be used to continuously improve following a bottom-up approach.

Table 2 .
Number of times that concern is raised and/or observed.

Table 3 .
Numbers of findings per bay.