Implementation of advanced technologies into Aeronautic integrated maintenance concept - Use of virtual reality in ground-floor training maintenance execution

. Aviation Maintenance industry, Repair and Overhaul (MRO) procedures need to keep up with the technological evolution and evolve from the 2D support to the 3D. The available manuals for learning and training MRO tasks rely much on old 2D drawings and lists of maintenance steps to be performed sequentially. However, these are complex actions that require and would benefit greatly from a 3D insight in order to be quickly and comprehensible absorbed. Virtual Reality (VR) apps are potentially a suitable option to turn these procedures closer to reality and, thus, improving competences and skills. Amongst the several maintenance optimization developments of the AIRMES project, which is cradled in the EU Clean Sky 2 Joint Undertaking programme, the above concept is applied to maintenance execution by developing a VR app to help practitioners in the process of carrying out specific maintenance activities as removing and positioning components into aircraft structures. The VR app runs on a mobile platform that uses a smartphone and a portable motion capture device coupled with a head mounted device allowing the practitioners to learn and to train onsite on how to proceed with the maintenance operations. The practitioners will be in an immersive and interactive environment where both the host aircraft structure section with the target component and auxiliary/peripheral systems parts are displayed and in which the 3D component can be removed by virtual hands that emulate, through the motion capture device, the hands of the user. The system developed provides a high-level training and reliable information to the technician on the maintenance operations for a dedicated situation and facilitate the identification and execution of the procedure to be applied, improving the time for repair.


Introduction
The increased offer of low-cost virtual reality (VR) technologies, Augmented reality (AR) and Mixed Reality Interfaces (MR) is enabling users and researchers to take the next step in technological innovation regarding human communication and interaction.
Hardware such as Oculus Rift, the HTC Vive and the Sony PlayStation VR enable 3D, stereoscopic head-tracker displays, hand/body tracking and binaural sound, providing an immersive, multi-sensory experience.
Immersion, perception to be present in an environment, and interaction with that environment are the three mandatory features of VR systems.
User immersion can be obtained through output devices which allow the user to "be" in the virtual environment. See, hear, can be ensured by glasses or helmets or HMD or CAVE systems, while touch feeling and force can be simulated by haptic output devices, tricking body senses to provide a more real virtual experience. Regarding user interaction with virtual environment is mostly ensured by input devices that launch continuous and discrete commands or movements to the environment via simple joystick or keyboard to a glove allowing capturing finger movements or a tracker able to capture movements in the physical world and translate them in the virtual environment. [1] Systems that use VR for maintenance purposes have been available since the 1970's [2][3][4][5][6], with numerous recent applications [7,8]. VR Maintenance (VRM) stands for the employment of VR technology (both software and hardware) to simulate several stages of the maintenance cycle. Disassembly, repair, replacement and installation tasks are available for training in a virtual platform, where the user is able to simulate all the steps of an MRO process. VRM is now a valid option to outdated procedures, reducing prototyping time and associated resources [2].
However, within the aerospace industry, VRM systems are still at an embryonic stage. Systems like the one to assist the maintenance of the F-16 aircraft's engines [9] are being tested by the Portuguese Air Force. VR can also be used for visual inspection purposes [10], or to train disassembly sequences [11]. Although there are not many examples worldwide, the interest on use of VRM is increasing [12].
Digital systems like VR can be enriched by 2D information on the material and processes required to execute a certain maintenance operation, thereby evolving towards interactive Decision Support Systems (DSS). A DSS is likely the best strategy available to assist operators in deciding the best solution for a complex problem for which algorithms are not available. Like VRM procedures, DSS that help the operator to select the best management practice are available within the aircraft industry. Peng et al. [13] have proposed a DSS for air material inventory. A short-term planning methodology of the line maintenance activities of an airline operator [14], at the airports, and [15] a three-stage methodology that can be used to select personnel rosters.
Critically, a solution that merges VR and DSS into one single platform is still not available. The potential in enriching VR systems with (contextualized) 2D information has not yet been reached.
The application of this technology to civil aviation MRO will address one of the several activities that contribute to technically-induced operational disruptions which are present in around 5.8% of all European flights and with an estimated cost of 2.8 billion € per year.

Technology brick
The average user has nowadays the technology that enables him to experience the virtual reality world enjoying things that are not accessible in real life, mitigating the drawbacks from gamifying training such as cost, hardware and custom software that have a big up-front investment when setting up which can be prohibitive to some companies.
VR Maintenance (VRM) stands for the employment of VR technology (both software and hardware) to simulate several stages of the maintenance cycle. Disassembly, repair, replacement and installation tasks are available for training in a virtual platform, where the user can simulate all the steps of a Maintenance Repair and Operations (MRO) process.
In AIRMES, a VR-based prototype was developed allowing the maintenance operator to recall the maintenance procedure on a stand-alone system that provides all the required information, both virtual and descriptive.

Use Case, specifications of operational scenarios and requirements.
The selected Use Case (UC) for not only the VR application but for the other developments of project AIRMES (for more information access project link on chapter 8), was the possible fixes for a malfunctioning/defect Wing Anti-Ice Valve (WAIV) on the right side wing ( Figure  1).
The choice took into account best use cases that should focus on the Minimum Equipment List (MEL) maintenance procedures, as these are the most stressful situations and is one of the best example of "running against the clock" and possible optimization of time and costs. Due to the architecture of the aircraft de-icing system, although there is a WAIV on each wing, the right side WAIV is a key device as any problem with it renders de wing de-icing system inoperable.
Facing an operating issue with WAIVs there are two deferred scenarios:  If the Left Hand side (LH) WAIV is defective, it can only be dispatched per MEL in CLOSED POSITION (if no icing conditions are foreseen). In OPEN position, no dispatch is granted in the MEL, thus the aircraft is grounded until the WAIV is replaced.  If the Right Hand side (RH) WAIV is defective, dispatch can be granted either in OPEN position (icing conditions foreseen) or in CLOSED position (no icing conditions foreseen).
The VR system was developed to aid the technicians to remove the WAIV (Wing Anti-Ice Valve). The removal procedure of this equipment is not frequently carried out hence maintenance technicians do not show a high level of proficiency performing this task. Moreover, the removal of the WAIV, requires unfastening and unplugging some cables and ducts that due to their difficult access or very fragile components require very precise actions to be performed by the technician.
The range of relevant users, other stakeholders and equipment already in use by them were identified as well as possible maintenance task location (Hangar, ramp, stands, etc.) and associated conditions such as light, temperature, humidity, etc., handleability (working with gloves, oil, etc.), Workspace (confined or free space), Safety issues (ATEX environment, EM discharges), Intervention Maximum time (to define autonomypossible charging between tasks, number of tasks with one charge), Operation identification means (FIN, Part Number, AMM tasks).

VR Devices
The VR devices chose for this application is off-the-shelf/commercially available solutions thus tackling cost issue and operating requirements mentioned above, moreover the equipment was dimensioned to the requirements and specification thus optimizing the cost/benefit ratio and becoming a major advantage for the user/company.
Regarding input devices, the tracker LEAP motion was chosen to detect user hand gesture used to manipulate the virtual environment objects. This device ensures smooth hand and finger tracking even against complex backgrounds and extreme lighting conditions, it also is an extensively tested device providing great stability and reliability, including finger fidelity allowing detailed functions.
As a processing/output device the choice was a mid-high level Samsung smartphone as this equipment already is part of the essential gear of the user to perform his work. All users make use of the smartphone to receive information from maintenance central services so if a smartphone could be used for processing and visualization it would be a plus reducing furthermore the initial hardware cost.
Lastly a simple VR helmet/goggles would be the 3D enabler platform by having install on it the smartphone and the motion tracker. No special goggles were necessary, just compatibility with the chosen smartphone was observed. Regarding software the choice relapsed on Unity Real-Time Development Platform (game engine). This game engine is used to create 3D or 2D, VR and AR contents for games, simulations or other types of experiences. Unit also supports more than 25 platforms and has been adopted by a large number of developers of several industries besides the gaming one, such as film, automotive, architecture, engineering and construction.

Development
The VR system is composed of several off-the-shelf/commercially available solutions, working together in order to enable the user to be immersed in a VR environment which also provides the interaction with the virtual objects, in this case the WAIV and other peripheral systems.
During the project, we have successfully achieved the hardware integration of the motion detector with the smartphone and through the VR application, we provided to the user a VR mobile prototype, usable anywhere and interactable without pointers, remotes or other equipment -using only the technician's hand.

Data Gathering
In order to create a realistic representation of the procedure, it was necessary to collect all the available information on the installation and removal of the Wing Anti-Ice Control Valve:  Text Manuals;  Honeywell CMM Valve Anti-Icing Wing Part No SAS911-002A;  AIRBUS AMM 30-11-51 PB 401 -Removal/Installation of the Wing Anti-Ice Control Valve;  Videos;  Images;  Verbal descriptions and replacement procedure carried out by the technicians.
Valuable information was obtained directly from the maintenance hangar where the full replacement procedure of the WAIV was seen and questions answered by the involved technicians. (Figure 2) With this information, it was possible to dig further on the multiple steps of the procedure, as well as the difficulties involved in the process.

3D Modeling
For the first prototype, further development and detail of 3D components were carried ( Some of the surrounding components were also added, so the environment would become more realistic.
At this stage, the 3D model was only textured with basic colours. An additional baking of an Ambient Occlusion map was added to increase the 3D model sense of depth. Then, the 3D model was exported for further integration in the Unity3D Game Engine.
Lastly for the second prototype (mobile application) the first prototype model was further developed, texture of the parts was improved, on more real 3D models and on mobile interaction capability.

Interaction
First approach to the interaction between user, hardware and WAIV virtual reality model was carried out by developing only the WAIV 3D model. This was conducted firstly by testing very basic manipulation concepts (only WAIV manipulation) and using very light models in order to simplify the task at hands, no 3D models of the interior and exterior of wing section were created at this point, in Unity 5.0. Using the LEAP device (motion detector system) to handle the model it was possible to pinpoint possible handling problems, limits of operability and other aspects to be considered, before moving to a more complex model and VR environment.
Next step was to create a Unity3D project with integration of a version control system with a remote repository via Unity3D Modules.
Having a completer and more complex VR environment, enabled us to add other interaction features based in multiple hand and finger gestures, such as gesture to grab, hold and release the WAIV and gesture to rotate the virtual world. Through experimentation and combination of multiple hand and finger gestures, the following interactions were implemented:  Gesture to rotate the virtual world (hand palms facing each other and pointing upwards ( Figure 4)  Gesture to grab, hold and release the anti-ice valve (grabbing gesture on both hands while near the valve 3D model) (Figure 4). A large portion of the work developed focused on the interaction capabilities of the system. In this way a fully interactive system was developed propped by a smartphone, a motion controller and a laptop.
Further developments were carried out in order to achieve the second prototype that would run entirely on a mobile device (smartphone), these developments included modification and enhancement of 3D parts, both WAIV (Wing Anti-ice Valve) and surrounding environments and parts (auxiliary system of cables, sleeve that feeds the valve and plane wing section) and technology development/transfer to the smartphone/mobile interaction system (head based visual and motion capture apparatus), with more parts capable of interaction.
With the VR application running on the mobile device following developments focused on improving user virtual interaction and continuous refinement of 3D models and integration of the maintenance procedure steps together with the respective parts highlight. (Figure 5)

Prototype demonstration
During the project several demonstrations, both internal and external, were carried out.
Internally several demonstrations took place at the end user facilities (TAP ME) to perform Verification and Validation (V&V) of each prototype stage. In these demonstrations not only the responsible for the project were present but also several maintenance technicians that shared their opinions and improvement suggestions for this application and future ones.
A final demonstration, in which all the AIRMES project solutions were tested, was performed at TAP with the presence of AIRBUS and Clean Sky 2 representatives.
Regarding external demonstrations, AIRMES project and the VR prototype were invited by Clean Sky 2 to be present at their stand in ILA Berlin 2018, Farnborough International Airshow 2018 and MRO Europe 2018 (TAP stand) providing visibility of the research and development projects that are funded by the EU and the impact of these new developments in the aeronautics sector.

Conclusions
European Civil Aviation loses around 2.8 billion € per year due to technically-induced delays that affect 5.8% of all European flights. MRO optimization, either through activities planning, prognostics, enhanced knowledge databases or mobile tools, will tackle this issue and even if they provide a small increase in the maintenance readiness thus leading to a reduction of operational disruptions, this will translate in several millions saved.
The developed VR-based prototype allowed the maintenance operator to recall the entire maintenance procedure on a mobile stand-alone system, providing not only the required descriptive information but also the possibility to interact with the virtual model, thus enhancing and enabling an in-depth comprehension of the procedure and the equipment under intervention.
The feedback received after trials and demonstrations performed at TAP was very positive: TAP engineers reported that this technology would increase the comprehension of the removal process and will boost the readiness of the maintenance technicians. The adoption resistance of this technology was also very low as the system was relatively nonintrusive, very intuitive and was clearly seen by the technicians was an added value instead of just one more inconsequential step in the maintenance activity.
We even use some equipment already owned by the end user, to achieve the best "bang for buck" and avoid an overkill system that with a much higher price would create a barrier for the deployment of this tool. Basing our prototype in already owed equipment (smartphone) will allow to do a more seamless deployment of this technology both human and cost related.
In the future and taking into account the above-mentioned facts, it is expected to have this type of system implemented, as a complementary procedure to the standard one.

Acknowledgment
We thank the support, technical insight and expertise from the TAP ME team that worked in this project, and greatly improved the outcome of this VR application.
Specially to Joel Ferreira (Project coordinator) and Luis Oliveira (Work Package coordinator) for providing, the best they could, with all information requested and access to TAP technicians parts and aircrafts as well as a great work coordinating all project activities.