Collaborative Augmented worker and Artificial Intelligence in Zero defect Manufacturing environment

New technologies are bringing together shopfloor elements which were separated before. The human worker, the IOT system and the Artificial Intelligence can co-operate together in order to better prevent and solve any possible problem which can lead to a product defect. Collaboration is enabled by a platform connecting the worker via Augmented Reality with data from the industrial IOT and the suggestions from the Artificial Intelligence.


INTRODUCTION
The industry 4.0 paradigm is carrying several technologies in the factory. The H2020 QU4LITY † project is investigating the role of the human worker in new highly automated zero-defect manufacturing environments and which is the level and way of collaboration with the Artificial Intelligence in controlling production parameters, quality of products manufactured and collaborative real-time troubleshooting of emerging production misalignment.
The first technological trend analysed is the eXtended reality ‡ technologies (encompassing Virtual reality, augmented reality and mixed reality) which are changing the way in which people perceive the digital world having this fully digitalised and connected in to digital twin for fidelity in replicated environments or connected to the physical element of the factory (machineries, equipment/robots, products, etc.). Augmented Reality is quickly progressing, Gartner in the 2018 version of the famous Hype cycle § 2018 moved it after at the bottom of the "Trough of disillusionment" and ready to get more mature getting back to expectation and really moving to a more maturity to be widely adopted. AR is progressing in maturity of wearable devices like the Microsoft HoloLens ** which will have a new major upgrade on the second half of 2019.
The second technological trend is Artificial intelligence which is one of the most disruptive classes of technologies, which is expected its Widespread in 2019 and will become more widely available due to cloud computing, open source and the "maker" community. The technology is changing the way in which data is managed passing from a passive collection of data, to active reacting IOT system analytics and moving towards more and more sophisticated intelligent systems which understand the context in the factory and predicting possible defects and providing prescriptions to the factory itself including both equipment and workers.
Several explorative research activities have explored the conjunction of these technologies talking about AI impact on work labour, † † augmented intelligence ‡ ‡ and augmented worker § § The QU4LITY project address the thematic of zero-defect manufacturing like aerospace from different perspectives. The focus of this paper is to define a system by which the augmented worker wearing an Augmented Reality connected headset can collaborate with the shopfloor Artificial Intelligence to better and timely support the goal of zero product. The solution should leverage on technology experimenting in a real environment on real tasks the augmented worker approach and measure results tailoring the system on the specific environment.

DIGITALISATION
As visible in the below picture the analytics capabilities are moving from a static representation of data to a more automatization of the decision-making process decreasing the human input. The positioning of the project is in supporting the level of diagnostic, predictive and prescriptive supporting the collaboration between the human worker and the system to the decision support phase. Major goals are in fact to: • To support human input in decision making like in troubleshooting activities in which is necessary to have information collected by the human to select the right procedure to execute. • to support human actions (e.g.: maintenance, different machinery setup, physical operations on products, etc.) when the decision is defined. Actions can be done: • By the human alone executing a pre-defined procedure • In which both human and the system (MES, machineries and analytics services itself) collaborates in the activities • The autonomous system actions are not taken into considerations ** https://www.microsoft.com/en-us/hololens † † Toward understanding the impact of artificial intelligence on labor, Morgan R.et alt. PNAS April 2, 2019 ‡ ‡ Thinking with machines: Intelligence augmentation, evolutionary epistemology, and semiotic, PeterSkagestad § § White paper on augmented worker https://www.augmentir.com/rise-of-the-augmentedworker The main changes are the shift from the AS-IS paper and desktop-based scenario to the TO-BE paper-less digital scenario. In the AS-IS the desk-oriented operator uses PC to read the output analytics provided from the machineries in order to take the right decisions; additional human input is collected in asynchronous way in respect with actions; the operator uses the pre-defined paper documentation to execute operations. In TO-BE every operation in which the operator has an active role can benefit from the smart solution support that leverages Augmented Reality technology; within • Human input and decision: the augmented operator become part of the system and integrated by handheld devices (tablet) and/or wearable devices (HoloLens) with analytic services (predictive maintenance, prescriptive analytics, dashboards, etc.) and troubleshooting activities that can be triggered by the system. The human collaborates in collecting more information from the system, receiving request forms & more operations (checks, calibrations, setup, etc.). The output of the process is the decision of what to execute. • Action: fully electronic digital logbook is available by handheld devices (tablet) and/or wearable devices (HoloLens). The augmented operator sees both digital instructions and augmented layer that guides him to the operations step-by-step which minimise the execution time and errors. Parametric procedures can adapt to the actual real-time status of the system for more precise operations; system simulations can be connected as well if-cases evaluation before the execution.
All the information collected during TO-BE augmented activities are fed back to the system. ***

Figure 2: AS-IS Desk-oriented Worker (left), TO-BE Augmented Worker (right)
Hence, the TO-BE transformation supports several use cases including the execution of operations like predictive maintenance and co-troubleshooting between operator and analytic system, the machinery set up procedures based on information received from diagnostic, the machinery maintenance procedures based on diagnostics received, etc.
In more general terms, the envisioned advantages of operational support via AR in this scenario is to (i) Increase the efficiency and efficacy of operations (e.g.: decrease intervention time), (ii) Increase visibility on processes (everything executed is log and can be provided as input to "data mining and analytics), (iii) decrease the operator input by providing a digital decisional support to expedite the operation needed to be taken into action and (iv) keep safety at high standard.

TECHNICAL SOLUTION
A set of procedures will be identified and digitalised. The Operation Support system will be connected to the analytics services (diagnostic, alarming, etc.) and some rules will be created to support the troubleshooting phase and semi-automatic selection of procedures based on analytics services. The operator will have a wearable device (HoloLens) or a handheld device (tablet) on which the application runs.The application running on the operator device guides him step by step in the execution of the procedure steps. The guidance is provided by two customisable elements: • An Augmented Reality overlay showing information on top of the physical items. Typical communication approach is to show: text & arrows pointing out items to find or to reach; static 3D showing where to place an object; dynamic 3D to show how to execute an operation; other elements • A floating window reporting instruction and any other useful information; the floating windows should allow to move around steps and can contain one or a combination of: 3D animations; 2D static element (text and/or pictures); Videos; Question & Answer if a different routing is needed (e.g.: is green light on?)

Figure 3: Application in Operation
The underlying procedure to be executed collaboratively can span from simple flow procedure to very complex troubleshooting. In the image below a visualisation of a complex procedure is provided in which the different steps are defined to be executed by the IOT system providing values, Artificial Intelligence to provide automatic assisted decision with certain confidence degrees and human operations dealing mainly with manual operations. Human operations are visualised with a brown circle around (example).

PROPOSED SOLUTION
The architecture solution provides an explanatory about the process shall be taken into consideration to maintain the expected result. In the Execution field, the field operator (line, maintenance, manager operator, etc.) will be guided thoroughly to execute the necessary operation, As well the outputted analytics will be displayed in the window tap in HoloLens and any variation of the parameters, the analytics output graph will change accordingly in real-time based on the data gathered. This won't benefit only the operator in taking the right decisions to execute actions, it gives as well full autonomy to the operator while walking freely to analyse and predict in case troubleshooting procedure is needed. Data is exchanged among the actors using a publish-subscribe mechanism.
The solution is based on Pacelab WEAVR † † † is a powerful software suite to streamline the authoring, management and execution of extended reality (XR) applications for operations, maintenance and training. Pacelab WEAVR enables companies to create, deploy and execute a wide array of virtual, augmented and mixed reality solutions from the same set of technical data, creating a seamless and consistent experience from basic familiarization exercises to advanced, interactive field support. The solution is agnostic in respect with Hardware; devices supported are, for example iOS and Android tablets/smartphones, HTC VIVE, Oculus, HoloLens, etc. The option proposed is to deploy the application on both Microsoft HoloLens (as wearable device) and android tablet (as handheld device). About HoloLens if commercially available and mature enough for an industrial environment the second version of the HoloLens announced during the World Mobile Congress on February 24, 2019 will be used. PACE is Microsoft Mixed Reality Partner (MRPP) so has access to newest software releases and direct support from Microsoft Mixed Reality specialists.

CONCLUSION AND FUTURE WORK
The designed solution has been detailed and planned to be instantiated in two industrial use cases and one laboratory experiment. The project will deploy and tailor the solution to the specific environments. It is expected that the experimentations will run for three months in all three environments and data about the evaluation will be collected. The KPI are linked to the time to operate in different conditions, time to access to the correct material and correctness of the operations.