An Investigation of Mixed Reality Technology for Onsite Construction Assembly

. The construction industry often falls short of other industries in terms of innovation; overall productivity within the construction industry has remained nearly flat for fifty years. As such, the industry is frequently exploring new technologies with the potential to increase its work efficiency as well as productivity. Mixed Reality (MR), which allows for the blending of real and virtual worlds, is one such technology that has a capability to improve productivity. The aim of this research study is to te st MR’s potential as an onsite assembly tool in construction. The study more specifically determined if holographic BIM models, spatially anchored in 1:1 ratio over active construction sites, could provide trades’ sufficient information to assemble and install their work without using paper plans. A controlled experiment was designed to test if the electrical and plumbing components of a mock-bathroom could be installed only with MR. A total of 5 sub-groups, each consisting of two participants, took part in this experiment and the time is taken by each sub-group (i.e. the MR group) to complete the assembly was recorded. This was then compared with the time taken by a similar number of sub-groups (i.e. the Control group) to complete the same assembly using paper plans only. The analysis indicates that on average the MR group took approximately 9% less time as compared to the Control group. Qualitative analysis via a post-exercise questionnaire found several technological limitations that the MR must overcome before it sees widespread implementation as a tool for guiding construction assembly. The post-exercise questionnaire also gathered participants’ perceptions about MR potential in construction. This paper highlights the benefits and limitations of the MR technology in construction along with recommendations for improvement as well as future research.


Introduction
The Construction industry is frequently subject to claims of inefficiency. Studies focusing on construction efficiency have documented 25 to 50 percent waste in coordinating labor and in managing, moving, and installing materials [1]. This might not come as a surprise to most in the field; in 2016 the World Economic Forum claimed that overall productivity in the Engineering and Construction (E&C) sector has remained nearly flat for 50 years [2]. In an industry that accounts for about 6% of global GDP, that seemingly stagnant pace of innovation permeates further than the industry itself. Society as a whole could benefit economically, socially, and environmentally if the construction industry caught up to other industry sectors. Many studies in the last 10 years have focused on what the construction industry can do to pick up the pace [3,4].
Building Information Modeling (BIM) is a glimmering exception to the traditionally bleak outlook on E&C innovation. The use of BIM in construction projects has increased significantly in recent years. According to research by McGraw-Hill Construction, the percentage of companies using BIM jumped from 17% in 2007, to 49% in 2009, to 71% in 2012 [5]. In 2018, that number is likely even higher. If this is, in fact, the case, the statement could be made that most construction companies develop and use some form of digital 3D models of their projects. If inefficiency stems from failing to make the best use of accessible resources, and a resource accessible to most AEC entities is BIM, it would be imprudent not to explore ways in which entities can achieve more from their BIM models. Mixed Reality (MR) technology has the potential to offer construction professionals more out of their BIM Models [6].
Mixed Reality (MR), as defined by Paul Milgram in 1994, is the merging of real and virtual worlds somewhere along the "Virtuality Continuum" (see Figure 1) [7]. It essentially allows for digital objects to exist and interact with real objects in real-time. Similar to basic computing in the 60s, MR has traditionally been reserved for entities holding both specialized equipment and personnel. However, recent advancements in this field, such as the Microsoft HoloLens, have simplified MR hardware to the point where almost anyone can take advantage; no specialized knowledge is necessary to perform most tasks. More so, applications such as SketchUp Viewer ® and 3D Viewer ® have made it exceptionally simple to transform BIM models into holograms [6]. With almost no additional effort, these holographic BIM models can be scaled, moved, and further manipulated to meet the user's needs. Models can even be placed in 1:1 ratios, allowing the user to walk through and view their holographic BIM models in the same way as a person walking through a constructed building. These features open the door to countless additional ways in which entities can further utilize their BIM models via MR, one of which is guiding construction assembly.
Numerous prior research studies support MR as a tool capable of aiding assembly. Richardson et al. [8] conducted a between-subjects experiment, comparing assembly effectiveness between traditional model-based instructions (MBI), the same MBI on a moveable tablet, and AR work instructions. He found that users completed the assembly in 33% less time while using the AR work instructions. Chryssolouris et al. [9] evaluated the use of AR assembly instructions in manufacturing and found that they contributed to reduced product development times and cost savings while simultaneously improving first-time quality and market response time. An experiment conducted by Chalhoub et al. [10] showed that MR can be effective for communicating design for electrical prefabrication.
Though MR has been proven useful for assembly, much of the existing research is theorybased and largely detached from the settings and scopes of work that would be experienced in real construction. The aim of this research study is to implement MR technology for construction assembly with two specific objectives, (1) to test MR's ability to communicate design details to installers for onsite assembly; and (2) to compare assembly efficiencies using MR and traditional paper plans. The research also investigated the technological limitations that MR needs to overcome in order to see its widespread adoption in the construction industry. This study is limited to the use of Microsoft (MS) HoloLens as an MR platform for construction components assembly. The following sections highlight the experiment setup, data analysis and its interpretation, followed by conclusions and recommendations for future research. Figure 1 explains various forms of real and digital environments using a virtuality continuum developed by Milgram [7]. At one extreme, on the left, is the case in which nothing is known about the remote world being displayed. This describes the real world as it is seen with one's own eyes, absent any virtual computer-generated images. The other extreme, on the right, refers to a world that the user perceives as completely virtual. This is essentially a world in which the user is 100% immersed in a virtual environment. In between "Real Environment" and "Virtual Environment" are the three terms which the continuum was created to define: augmented virtuality, augmented reality, and mixed reality. Mixed Reality (MR), sometimes referred to as hybrid reality, is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real-time. Mixed reality takes place not only in the physical world or the virtual world, but is a mix of reality and virtual reality, encompassing both augmented reality and augmented virtuality via immersive technology [11].

Fig. 1. Milgram's Virtuality Continuum [7]
Currently, not many companies use the term "mixed reality" to market their hardware; rather, they use the more specific term "augmented reality". The most well-known device that does market itself as MR is Microsoft HoloLens.

Microsoft HoloLens
The MS HoloLens is a self-contained head-mounted display, depicted below in Figure 2. Specialized components such as a HoloLens Processing Unit (HPU) and advanced sensors enable holographic computing. This enables HoloLens users to engage with digital content and interact with holograms in the world around them [12].
Advanced sensors within the HoloLens continuously capture data pertaining to the user's activity and the surrounding environment. They can see, map, and understand the physical places, spaces, and objects around the user. The device understands gestures and maps the world around the user, all in real-time. In addition, the HoloLens has a see-through holographic high-definition lens and an advanced optical projection system. These enable the HoloLens to generate multi-dimensional full-colour images with very low latency, allowing the user to see holograms in their own world. The headband is designed to distribute the weight around the crown of the user's head, saving the ears and nose from undue pressure. Though the HoloLens has more computing power than the average laptop, it is passively cooled without fans. Lack of wires and external cameras, as well as the lack of phone or PC connection requirements, allows the user to move freely [12].

Mixed Reality (MR) Applications in Design and Construction
MR technology has long been explored as a potential asset to the design and construction industries. In 2011, Dunston and Wang published a hierarchical taxonomy of AEC operations for MR applications. They found that MR-based technology is suitable for informationintensive tasks, which in the AEC industry, often deal with the information translation usually between a paper-based source and the work. In these tasks, MR can augment a human's ability to access information and documentation in the course of performing the work and enhance the individual's decision-making ability [13]. Table 1 outlines various applications of MR in the design and construction industries.

Progress Monitoring
To overlay the as-planned building model of a structure over the actual as-built structure to determine the percent completion of the project.

Onsite Clash Detections
To allow trades to perform clash detections on the construction site itself by integrating holographic models with real objects.

Information Overlay
To superimpose useful information onto real objects in the surrounding environment or to add comments to their surrounding real environment.
Several designs and/or construction-related Apps are available in the Microsoft store to use with the HoloLens. Some of the most useful Apps are: − SketchUp Viewer for HoloLens: SketchUp Viewer allows users to quickly and easily import a SketchUp model to the HoloLens for visualization as a hologram. − ARchitect: ARchitect offers a sample home for a walkthrough. A unique feature in this app allows the user to see how the home looks in varying sun-lit conditions. − BIM Holoview: BIM Holoview projects 3D Revit and Navisworks models into the physical world as holograms. − HoloLive 3D: HoloLive 3D allows users to bring BIM models from native design software and place them in the real world. This app provides advanced measuring, scaling, and other tools that allow for a highly immersive experience.
A mixed-methods research design, consisting of quantitative and qualitative measures, was employed to carry out this research study. Figure 3 outlines the research methodology.

Fig. 3. Outline of Research Methodology
For quantitative data collection, a "between-groups" experimental design was employed through the creation of an exercise entitled "Electrical and Plumbing (E&P) Assembly Exercise". The E&P Assembly Exercise was created to determine the time taken by 2-member groups to install electrical and plumbing components into a mock-bathroom using "MR" and "Paper plans". The exercise required the installation of 85 components, most of which consisted of pre-cut PVC conduit and fittings. Participants were required to lay the plumbing and electrical components into their proper positions inside of a preassembled 5'x10' mock-bathroom. Each of the 85 components was labelled to guide the installation. Figure 4 shows the mock-bathroom as well as E&P assembly model.
The participants were divided into two main groups namely the "MR group" and the "Control group". Each group consisted of 2 persons. Participants of both groups were selected by convenience and were limited to willing student volunteers enrolled at Auburn University's McWhorter School of Building Science. There were a total of 5 MR sub-groups and 5 Control sub-groups. The MR sub-groups used the holographic model to install all components in place whereas the Control sub-groups used traditional paper plans. All subgroups were filmed, monitored, and timed through their completion of the exercise. Additionally, participants were given a pre-exercise questionnaire to collect demographics and a post-exercise questionnaire (for MR group only) to collect qualitative data on their perceptions of the MR technology and/or HoloLens.

Fig. 4. Mock-bathroom and Electrical & Plumbing Assembly Model
As mentioned above, the quantitative data was gathered by measuring the sub-groups' completion times of the assembly. The exercise timer was turned on promptly at the start of each group and turned off directly upon completion when participants placed the last remaining piece into its appropriate position. This data was later used to compare the efficiency between the two groups. An additional timer, nicknamed the "discrepancy timer", was kept in addition to the primary timer. The purpose of this time was to measure any uncontrollable errors experienced by the participants. An example of a discrepancy would be if a participant found they were missing a piece required for assembly. Prior to beginning the activity, participants were instructed to promptly inform the instructor upon experiencing any uncontrollable error; in such cases, the discrepancy timer would then run from the moment the error was noted to the moment the error was resolved. Discrepancies were predicted to be much more common in the MR group, as pilot testers experienced problems such as the device turning off, the application closing, or the E&P model jumping out of place. These times were recorded in addition to the total time it took the groups to complete the exercise. Figure 5 shows an MR group completing the assembly.

Pre-Exercise Questionnaire
A pre-exercise questionnaire was given to all test sub-groups prior to commencing the exercise. A total of 6 questions were designed to collect demographic and relevant prior experience data from the participants. Table 2 shows the summarized information collected through this questionnaire. The results indicate that all sub-groups have more or less the same composition and possess very similar construction and/or E&P assembly experience. Hence it is reasonable to compare quantitative and qualitative data collected from all sub-groups.

Experimental Data
The E&P assembly completion times for all subgroups are reported in Table 3. The description of the recorded times is as follows: T1: Assembly time in minutes T2: Discrepancy time in minutes T: Net time taken to complete the assembly (in minutes) = T1 -T2 A comparison of the net average assembly time for both main groups indicates that the MR group took approximately 9% less time to complete the assembly as compared to the control group. The 9% improvement in productivity, though small, is significant because it shows that the MR technology has the potential to boost work efficiency and productivity of the construction trades. Due to the small data size, it was decided not to perform any inferential statistical tests on the collected data.

Post-Exercise Questionnaire
The purpose of the post-exercise questionnaire was to gauge the participant's perception of MR technology in construction. As such, it was only administered to the MR sub-groups. The results of the post-exercise questionnaire are summarized in Table 4. Based on the analysis of the post-exercise questionnaire, the following conclusions can be drawn: 1.
MR in general, and HoloLens in particular, are found to be an effective tool for communicating design details to electrical and plumbing installers or other construction trades. Seventy-seven percent (77%) of participants found it to be "extremely effective" and the remaining 23% found it to be "very effective". No participants reported that the HoloLens was moderately or slightly effective, or not at all effective.

2.
If asked to complete the same exercise over again, 100% of participants predicted they would be more effective using the MS HoloLens rather than paper plans. 3.
Two-thirds of participants felt there was some degree of the learning curve associated with using the HoloLens, whereas the remaining one-third felt that no learning curve was required and the HoloLens was immediately effective in practice. The following technological limitations of the HoloLens were recorded: Spatial anchoring: Spatial anchoring refers to the HoloLens's ability to lock a hologram to a designated location in the real world. Seventy-three percent (73%) of participants reported that the model moved out of position. This makes the improvement of spatial anchoring technology one of the most prominent needs for the HoloLens. Without sufficient spatial anchoring, the Microsoft HoloLens loses the ability to depict assemblies in their accurate position to installers.
Model brightness: Approximately 27% of participants claimed that at some point or another, the model was not bright enough to see. In one instance, the discrepancy timer was even initiated due to the participants' inability to see the model. After reviewing the film of the groups who experienced problems with model brightness, it was found that all cases occurred on days that were exceptionally sunny, with bright light coming through the windows. As construction assembly predominantly takes place outdoors, the HoloLens needs to improve upon its ability to display holograms in direct sunlight.
Application reliability: The HoloLens turning off by itself or application crashing was reported by 27% of participants. Amongst the discrepancies, the timer was initiated twice because there was an error with the application and it had to be restarted. In these instances, the SketchUp model flipped upside down as the result of a glitch. As the HoloLens is new, and developers are still getting used to developing MR applications, this limitation is not surprising. But for implementation of MR into construction assembly tasks to occur on a large scale, these applications need to become more reliable.

Conclusions and Recommendations
This research demonstrates that MR technology can successfully communicate design details for construction assembly, specifically validating that the MS HoloLens is a tool capable of guiding electrical and plumbing component installation. Additionally, the elaborate nature of the Electrical and Plumbing Assembly exercise used in this research serves a strong proof of concept for construction assembly via MR. A 9% productivity gain is promising for a new technology that is still in its infancy. A number of technological improvements are required if MR is to see widespread adoption for assembly purposes. The most prominent need for improvement is spatial anchoring technology, as successful assembly would require the model to hold its position without jumping out of place. This is especially true in the realm of construction, where installation of components frequently requires accuracy to be within 1/16 of an inch from perfect. Other shortcomings the HoloLens must overcome before widespread adoption are occluded visuals in sun-lit conditions, as well as the inability to wear a hardhat with the HoloLens.
This study is a pilot test of MR technology and has several limitations which must be improved in future studies. One of these limitations is the small sample size, prohibiting inferential statistical testing from being carried out. One of the greatest advantages MR provides is its ability to display information at the exact spot it is required; in construction assembly, this equates to a reduction in time spent thinking about component layout, as the installer would automatically know the exact spot where the components are to be assembled. But in this research, the holes in the walls and subfloor, through which the components were installed, were pre-drilled for participants. This method took away the greatest advantage of the HoloLens, as the pre-drilled holes allowed the control groups to have a very good idea of the layout. In a scenario where the assemblers had to drill their own holes, the control groups using paper plans would not have this advantage, and they would have to spend additional time measuring out the intended layout. We predict this would work heavily in favour of the groups using MR. In addition, the 85-parts required to be assembled by the exercise were labelled in somewhat of sequential order. For example, C1, C2, C3, C4, and C5 all had to be installed in order. Although not all areas of the assembly were labelled in a sequential order that was straightforward, a better way to design the exercise would have been to label the parts completely non-sequentially. Similar to the pre-drilled hole bias, sequential part labelling took away one of the main advantages of MR for assembly.
For future research, the following measures are recommended: (1) Test MR technology against other forms of design visualization such as the 3D BIM models; (2) Expand groups' to non-construction students (i.e. novice users) as well as professionals to determine how each group is benefited from this technology; (3) Synchronize two or more HoloLenses so that all group members can see the exact same hologram; and (4) Try out more Apps to compare their effectiveness in relation to SketchUp App.