Pragmatic science? Reflections on the academic-industry interactions in a European aviation research programme

In this article we explore the types of science produced and disseminated in human factors research in the cognitive domain. We reflect on the methods and techniques used in the European, Horizon 2020 ‘Future Sky Safety’ Project: Human Performance Envelope (P6). This project has adopted multiple research paradigms, successfully engaging academia and research organizations throughout its four-year duration. We discuss the challenges of conducting practically focused work that can also be brought to publication in peer-reviewed journals. Finally, we frame our research contributions within this project using a recent model of practitioner engagement.


Introduction
Our starting position is that human factors and ergonomics are about the observation and characterization of the work that people do. If we then create theory or predict taskperformance, so much the better. In this article we reflect on the challenges and opportunities of developing practically relevant research that is also suitable for publication in peer-reviewed journals.
Human factors is an applied discipline. Interest is rarely awoken without at least a passing allusion to some work or task, even when this task is highly abstracted. In cognitive ergonomics, we find it difficult to characterize any part of human factors as 'pure'. The closest we imagine are some of the human error taxonomies, but arguably, even these have their roots in cognitive psychology. Certainly, useful and used concepts that sometimes parade as theory can often be traced back to simpler concepts in cognitive psychology [1]. Such concepts are none-the-worse for this when used with care, if this ensures that due consideration is given to human capabilities and performance limitations.

Knowledge production and dissemination
In the discipline of human factors, cognitive work is observed and measured as an object of study. Using an explicated method we generate knowledge. Then in academia at least, the gold-standard would be written publication in a high-quality, peer-reviewed journal. Other routes for dissemination would be through white-papers, the grey literature which would encompass magazines or trade-journals or conferences. In this article we write from the perspective of academia. We frame this discussion using the technology readiness levels (TRLs) first suggested by NASA [3] (See Table 1). TRLs define different levels of maturity of a technology from the very first idea to a manufactured technology being used for an intended purpose in an intended context. Traditionally, low TRL (1-3), fundamental knowledge is predominately generated in universities. Using Gibbon's characterization of mode 1 knowledge, this fundamental knowledge is then 'thrown' at industry to use [2]. Industry is then expected to take this knowledge forward, creating an impact narrative on the way for use by universities to secure more funding for fundamental research. In the meantime, fundamental knowledge can be disseminated in peer reviewed journals, on which further fundamental knowledge can be based.
However, human factors is an applied discipline. As such, we question whether this division into higher and lower TRL is useful, healthy or sustainable: human factors dies in ivory towers. Indeed, human factors research conducted by academics can start to look like operating at a higher TRL. Certainly at Cranfield University we have worked on projects at TRLs 4 -6, traditionally the preserve of other types of research organizations. As in the current project, the academic community can work with the practitioner community to prescribe research and research designs that are focused on work and users within the system as a whole. This brings us closer to the mode 2 knowledge production styles which emphasize knowledge co-creation in context: with industry. This also means that our research may look different to the dominant paradigms employed in the peer-reviewed literature. Table 1 Technology readiness levels (TRLs) and their descriptions.

TRL Description 9
Actual system "flight proven" through successful mission operations. 8 Actual system completed and "flight qualified" through test and demonstration. 7 System prototype demonstration in a space environment. 6 System/subsystem model or prototype demonstration in a relevant environment. 5 Component and/or breadboard validation in relevant environment. 4 Component and/or breadboard validation in laboratory environment. 3 Analytical and experimental critical function and/or characteristic proof-of-concept. 2 Technology concept and/or application formulated. 1 Basic principles observed and reported.

Categorising knowledge
Co-creation of knowledge between academia and industry certainly a noble goal. However, this co-creation can be problematic when set against the requirement for academics to publish in the peer-reviewed literature. It can also be problematic when funded research programmes demand return-on-investment in specific timescales. We also suspect that some journals are less inclined to publish work from research paradigms that move beyond the positivist. A useful framework in which to consider the different types of articles in journals is provided by the Journal of Occupational and Organizational Psychology [4] shown in Table 2. This framework has roots in Anderson et al. [5] who characterize research across two dimensions: practical relevance and methodological rigor. With the addition of a third dimension: theoretical/ conceptual content, de Jonge [4] characterizes the resulting eight categories. These categories provide a basis for the assessment of articles by handling editors, and some criteria to drive review. The alliteration based on 'p' is playful, but we do note an implicit, negative bias for research which is characterized as having 'low practical relevance', whatever this is taken to mean. Who, after-all would care to be dammed with the faint praise of being 'proficient'? Especially when set against the excellence narrative pursued in academia more widely.

Future Sky Safety: Human Performance Envelope
Two authors (JN and BK) have been working Project 6 'Human Performance Envelope' since 2014. This project is part of the larger, European Future Sky Safety programme of research funded through Horizon 2020. This long programme of work has moved through many methods at points during its evolution. From experimental approaches to understand how new technologies work, to focused, qualitative studies to understand how operators use the new technologies which have been made available through industry. The execution and evolution of this programme of research activities has underlined the requirement to be fully open to a variety of research paradigms while being cognizant of their limitations. However, as with most things in life the reality is more demanding. On reflection, many discussions throughout the project have focused in the number of participants (n) in any given activity. Of course there are pressures and limitations on the number of participants when considering highly trained experts versus normal populations. Using the positivist narrative, higher n means higher reliability and higher explanatory power. Both implicitly and explicitly, this has been a guiding light in many discussions. However, on reflection all the partners have worked pragmatically to ensure that activities are meaningful, even when lower number of participants are employed. It is notable that the studies that have been published in the peer-reviewed literature so far have been predominantly those which use an experimental paradigm with a high n. Their carriage, we suspect, eased by the dominant position of the positivist paradigm in science more broadly. We suspect that this bias is extant in human factors. In the best traditions of the mixed methods approach (Table 3) we claim to have been pragmatic throughout while being clear about the limitations and advantages of the selected designs. Specific activities that have been conducted are listed in Table 4. The deliverables are publically available at the Future Sky Safety website and the specific deliverable number is given in the table. The range of activities conducted included positivist, higher n experimental work (3) to more focused context specific studies (6) and most things in between. The research programme demonstrates what can be achieved when industry and academia join forces to co-create knowledge. This knowledge is not always amenable to publication in journals, but overall the power of the combination can make real inroads into developing new technology or ideas. This has been done throughout the project when considering the new cockpit concepts and the real-time physiological measurement of pilots to assess state.  Experimental measurement, eye-tracking interviews to support cognitive task analytic techniques.

7
New cockpit concept simulations (6.6) 10 Experimental measurement, eye-tracking interviews to support cognitive task analytic techniques.
Marcos and Denyer [7] suggest that engaged practitioners can perform four invaluable functions in the knowledge production (or research) process. As contributors they help not only to identify organizational problems that require investigation, but also to refine the subsequent research questions that drive the investigation. Their contextual knowledge illuminates what is happening in the particular research setting and ensures the focus remains on current and topical issues. As brokers they facilitate access to appropriate research settings through their relationships with key gate-keepers. They also encourage the cooperation of respondents in the data collection processes. Once the data have been collected and analysed, they play an important evaluator role. Here they can challenge and shape the interpretation of the data. Using their practical knowledge, they can draw attention to topics that may be minimized or overlooked. They can also identify the important findings differentiating them from the statistically significant ones. Finally, they can play an advisor role, helping researchers identify appropriate dissemination routes for the project findings that reach relevant practitioner and policy audiences. Depending on their external status in the practice and policy community they may be able to facilitate engagement with key industry bodies and regulatory authorities.
Returning to the Human Performance Envelope programme of work, we claim that the practitioners represented in the group have performed all of these important functions. The practitioner community has contributed to proposing and defining important organizational problems. For example Lufthansa was a key contributor to developing the complex failure mode with which to engage the pilot not-flying. Access to high fidelity simulators and the pilot community has been brokered by DLR, Lufthansa and Thales without whom, the quality of the contextual environment would have been lacking. Following the collection of data, all practitioners involved in the project spent time with the academic partners to evaluate the data both in terms of the scientific merits and also the relevance to current commercial issues of interest to them. Finally, the practitioner community has been vital in ensuring the dissemination of work outside of the peer-reviewed journal space. This has included regulators and airlines bringing the research rapidly back round to the practitioner and academic community, accelerating impact.

Concluding thoughts
It is particularly the case that in aviation, context is very important [8]. Aviation is complex and undertaken within large complex systems by highly trained operators from pilots to ground handling staff at airports. In this light, it is important for the academic and practitioner community to work together, avoiding the production of popularist or puerile research. We believe that this balance has been achieved, not always perfectly within the Human performance Envelope programme of work. In this programme we have successfully employed the four ideas of practitioner engagement identified by Marcos and Denyer. The scientific paradigm of pragmatism has been adopted, implicitly and explicitly, through the programme. This has led to research outputs that we hope are potent, prescient and pragmatic across the peer-reviewed literature and the wider aviation community. We would add one final quality that we always hope that research would have regardless of paradigm: to be interesting. Notwithstanding our regret that we could not find an adjective beginning with the letter 'p', we trust that the co-creation of knowledge in this project has this quality.