The study of human behaviour in a laboratory set-up with the use of innovative technology

Virtual reality technologies known since the 1990s have virtually disappeared. They have been replaced by their new editions characterised by a very faithful reproduction of details. The laboratory described in the article was created to study the intensity of various sensations experienced by a person in virtual worlds. A number of different scenarios have been developed that are designed to trigger specific reactions, to produce symptoms similar to those observed in arachnophobia, acrophobia or claustrophobia. The person examined can be equipped with a series of non-invasive sensors placed on the body in such a way that they do not interfere with immersion in VR. The laboratory instruments enable the acquisition and synchronisation of many signals. Body movement data is recorded by means of Kinect. Involuntary hand movements are measured with 5DT gloves. In addition, body temperature, ambient temperature and skin moisture are continuously monitored. Apart from recording the image from VR goggles, it is also possible to record the entire session on camera in 4K resolution in order to interpret facial expressions. The results are then analysed in detail and checked for patterns. The article describes both the test set-up itself as well as several test scenarios and presents the results of pilot studies.


Introduction
Practical applications of VR technology were already the subject of interest of scientists at the end of the twentieth century [1]. The authors focus their interests on the aspect of supporting physically handicapped people, whose gestures -through software, VR and a hand armed in a gauntlet -can be mapped to computer actions, audio messages or even devices. Currently, the scope of the term VR covers a much larger range and is primarily understood as a means for creating a photorealistic digital world, which can be penetrated by using, in the simplest solution, specialised glasses. VR technology can also be successfully used in other applications, e.g. for the preparation of museum exhibitions [2], interactive presentation of the operation of various devices [3], or for creating a serious game for anti-stress therapy [4].
A person's immersion into the world generated by the computer and possible interactions with it causes the person to lose the sense of reality. These VR properties have contributed to the emergence of the concept of treating people suffering from various phobias by stimulating threats in an environment widely regarded as safe.
Eichenberg describes the achievements of researchers in the use of VR (in the first decade of the 21st century) as an effective method of treatment of many mental disorders [5]. He presents the limitations of this technology: the costs of equipment purchase and staff training, objections to the technology by many therapist communities and contraindications for certain groups of patients, e.g. people with post-traumatic stress disorder (PTSD). The positive aspects of VR include the effectiveness of the method, long-term cost reduction, lower logistics costs, the ability to control settings and increase the motivation of people for therapy, especially in young patients.
Economical aspects of VR application in phobia therapy are discussed in [6]. The solution presented there consisted in creating software for Android devices based on Google cardboard and VR simulations, addressing the three most recognised phobias (acrophobia, phasmophobia, nyctophobia). Such a kit is a low-cost solution and can be used without delay for therapeutic purposes.
Article [7] describes a study of people's behaviour in the face of the stress ahead of an academic exam, employing for this purpose text (TX), sound (AU), video (VD) and VR with the introduction of the so-called technological breaks (which interfered with the immersion of a person in the digital world). In order to evaluate the effectiveness of each technology, the respondents filled the Post Media Questionnaire (PMQ) and the Slater-Usoh-Steed Presence Questionnaire (SUS). In addition, objective measures of the emotional state of the subjects were used: Electrocardiogram (ECG), Thoracic Respiration Signal (RSP) and Facial corrugator supercilii muscle Electromyography (EMG) were recorded. An important result of these studies showed that VR is an effective therapeutic method if it works flawlessly, and the immersion into the generated digital world is not disturbed. In [8] arachnophobia was examined, to which a questionnaire before and after VR sessions was used. To determine the level of anxiety, the Hamilton scale was adopted, denoting drug 4 levels (from absent to very strong). The study on people with claustrophobia, which in the future can be used as a virtual reality exposure therapy (VRET), is presented in [9]. A method for creating virtual 3D worlds for the preparation of various research scenarios in which the following characteristics were variable (openness of space, order and colour) is shown.
The described experiments are usually carried out with the subjects in the sitting position, and in order to assess the behaviour of people who are in VR immersion, different types of polling were used, for example PMQ and SUS, and sometimes physiological indicators such as ECG, RSP and EMG.
The aim of the article is to present the test set-up created for studying people's behaviour in VR environment and to verify its suitability on the basis of pilot studies. The set-up records many heterogeneous signals such as physiological (pulse, humidity body temperature), kinematic (body movement, hand gestures, facial expressions) or environmental (temperature and humidity of the environment). The information obtained from the interview and two questionnaires (before and after the test) supplements the information collected from tests. The set-up will be used for further research, the aim of which is to better understand the origin, course and real scale of phobias impacting the human organism.

Prototyping the laboratory set-up
Activities regarding the test set-up included the following stages:

Development of functional requirements
The set-up for human reaction test was designed to allow the collection of data in a heterogeneous field of origin (physics, physiognomy, environment) as well as in the time domain. Physiological data do not have to be saved as often as physical data, but much more often than environmental data.
We managed to establish a functional requirement for the data acquisition system about the need to sort data from different sources relative to time. The data are saved in independent paths in a multi-threaded process in order to provide unrestrained access.

Selection of devices and peripherals
The infrared camera, capable of identifying individual parts of the human body, collected physical data, whereas a regular camera was implemented for documentation and verification purposes.
To collect physiological data, a pulsometer and a humidity sensor were used, and a thermometer to collect environmental data.
Sensors responsible for collecting physical and physiological data were connected to a special workstation that controlled the devices, their start and stop time, and managed the recording process. To prepare the graphics station, the authors used the documentation of technical aspects of the necessary hardware and software, methods for creating photorealistic virtual environments, models and applications designed for the medical treatment of patients suffering from various phobias described in [10]. A separate workstation acted as a graphics station for generating VR projections. The authors decided on this kind of hardware separation, anticipating the possibility of interference, disturbances or simply delays in the recording of sensor data. A similar test set-up of heterogeneous signals presented in the paper [11] did not generate any jitter in the operation of sensors, however there, (1) instead of VR devices, movement recording cameras were used, and (2) more expensive and more precise equipment was used to construct this station. Furthermore, the authors were afraid of delays in VR projection, which might result in the occurrence of VR sickness.

Construction of the test set-up
A schematic diagram of the set-up constructed in the Laboratory of Intelligent Systems Programming and Computer 3D Technology (Lab3D), the virtual and augmented reality section, is shown in Figure 1. Figure 2 shows an early prototype of the set-up. The VR projection equipment used served to build and maintain immersion in the virtual world. The following were used to build a prototype test set-up: VR glasses (P1, resolution 960x1080 px per eye, refresh rate 75 Hz) with a hand movement sensor (P2, 200 Hz), 3-D surround sound set (P3) -for VR experience increased suggestiveness [12], computer set for VR image generation (P4, P6, octa-core processor, 48 GB RAM, graphics card: memory interface 256-bit, 1600GHz, 8 GB memory).
The VR session was used to create a participant profile based on physiological data obtained with the help of measuring equipment. Part of the equipment was wearable. Data collected from the participant during the experiment was: contraction of the fingers recorded through fibre optic gloves (P7, minimum sampling rate 75 Hz, one sensor per finger, sensor resolution 12-bit A/D), sweating recorded via the humidity sensor (P8, spot measurement before and after the experiment), pulse -recorded by means of a heart rate monitor (P9, sampling rate 1 Hz), limb movement recorded by the infrared camera (10, 640x480 px, IR depth-finding 30 Hz, recording 144 channel). Data from many sources were heterogeneous due to differences in the frequency of the recording, and discrepancies in the start and end time of registration of each of the measuring devices. The process of data synchronisation concerned the use of hardware solutions. To synchronise data after the participant's session ended, the following elements were used: high-resolution recording from cameras capturing the entire experiment from the front and participant's profile (P11, P12, 30 fps sampling, 1920x1080 px), recording from a projector emitting structural light (P13, sampling 60 Hz, 1920x1080 px), time recording from a digital clock (P14, accuracy 0.01 s).

Preparation of software
For data acquisition it was necessary to develop an original solution that would enable: remote control of sensors (start-up and calibration), management of the recording process (start and end), and data parsing to form a stack of data records. Part of the equipment was connected by means of these control applications, operated from the level of highly efficient workstations (P5, P6), including: A1 application for collecting data from gloves (P14), A2 applications for collecting data from the heart rate monitor (P9), A3 application for collecting data from an infrared camera (P10). Measurement applications were run on a separate machine (P5), in order to evenly distribute the computational load.

Development of test scenarios
In order to identify the biggest fears occurring in the test group, a preliminary survey was conducted. A randomly chosen group of students from cities and rural areas were asked about their anxieties and fears. A survey was conducted on a group of 20 students (8 women and 12 men) from the age of 21 to 28 (median 23). The survey introduced a scoring scale of anxiety scores from 1 (no fear) to 5 (severe fear). The results obtained are shown in Figure 3.
The questionnaire also included a reference question "Are you afraid of wolves?" -in European culture, the wolf is treated like an archetype of evil, which everyone is afraid of. On a scale from 1 to 5, the average value of fear is 3. The survey shows that out of 13 listed threats, value 4greater than the average -was accorded to the reference question (wolves) and to the fear of a swarm of hornets. Six different hazards obtained the value of 3 (average): arachnophobia, acrophobia, ophidiophobia, cynophobia, apiophobia, entomophobia. Claustrophobia threats and fear of frogs and toads aroused minimal fear -value 1.
Based on these answers, a pilot study scenario was proposed, which was consulted with a psychologist. A scenario for acrophobia was decided. The created world presented in VR faithfully reproduced the existing place in nature [13]. The scenario used was about the passage over a suspension glass bridge, placed over a deep canyon in a mountain setting (Fig. 4). The VR projection was accompanied by a 3D sound effect.
The scenario is open and nonlinear. The subject can move freely around the virtual world and decide for themselves what and when they want to see. For study purposes, the control over the user is taken over by the expert and he is the script's narrator (Haron).
Several important steps can be distinguished for measuring purposes and identifying anxiety in the scenario. They are characterised in Table 1.  Apart from the questionnaire, this was to be the second, verbal form of profiling the study participants. They were asked, among other things, about the familiarity with computer technologies, VR technologies, interest in electronic games, films. The answers collected during the interviews were intended to help the authors verify the significance of visual and auditory stimuli relative to the interests, experiences and general knowledge of the participants.

Preparation of the method of performance evaluation
Based on the position data of the body parts (x, y, z coordinates) in space and time, the reference points representing the mean position of each part of the body were determined. Then, for each time sample, the distance of points of the body parts from their averaged positions was calculated. In this way, the authors wanted to obtain information on the tendency of the body parts to move (cm) during the experiment.
Glove data after passing the initial selection were normalised to simplify the reading (0 -full flexion of the finger, 1 -full finger extension). Based on the data from the heart rate monitor, the deviation of the measured values from the values in the resting state (base value) was determined -the pulse value measured at the beginning of the test.
Subsequently, all the data were sorted against the timestamp. In this way, on the basis of many heterogeneous signals, a numerical record of all changes in the physics and physiognomy of the participant in the experiment during the VR session was obtained, omitting all irrelevant or incorrect frames.
After comparing the numerical results with the video recording of the reference camera and the recording from the VR glasses, it was possible to clearly identify the visual factors causing or contributing to changes in the participant's physics and physiognomy. Further assessment of the significance of stimuli was based on the identification of the moment when there was a simultaneous and significant change in physical and physiological factors.

Testing
Based on the procedures presented in Section 2, pilot studies were carried out. A single study session for each participant consisted of the following activities:    The recording session of a single participant lasted about 300 seconds.

Analysis of results
To analyse the results of the pilot studies, the values belonging to the four groups of data were selected: hand movements, torso movement, flexion of the fingers and pulse. Hand movement and torso movement was recorded by the Kinect device (Fig. 1, P10), the finger flexion movement was recorded using a 5DT digital glove, and the pulse was recorded by a Mi Band 2 device. Figure 8 shows a fragment of the study of one of the participants. The fragment presents graphs of the interpolated data in 10 seconds. During this time, the participant finished phase 2 (Fig. 5.) of the experiment, went through phase 3 (Fig. 6.) and started phase 4 (Fig. 7.).
The obtained results were grouped according to the relationship (A, B, C, D) and shown below in Figure 8. Fig. 8A -after a few seconds of the subject's stable posture, between 8-9 seconds there is a clear change to Spine base, which means it can bend over (this can be verified in the video image). Fig. 8B -after the recording starts, the object moves (stabilises) 0-1, subsequently, between 2-3 seconds there is a very fast hand movement. The next phase (3-5 sec.) shows a clear movement of the arm passing into the nervous movement of the thumb (6-8 sec.). The next stage (8-10) -a clear raise (8-9 sec.) and lowering (9-10 sec.) of the object`s hand. Fig. 8C. -shows that the biggest finger reactions refer to the thumb and the middle finger. These fingers react most quickly to the change of scene visible in VR. Based on these graphs, one may wonder if it is worth analysing the movement of other fingers thus reducing the number of analysed values. Fig. 8D -presents the increase in the pulse value calculated from the subject's pulse at the beginning of the task. The increase in the number means that the person reacts to images shown in VR.
The results presented in Fig. 8 (a, b, c, d) are detailed and their individual interpretation does not give an overall result. It is necessary to correlate them and draw aggregate conclusions. The combination of information about the movement of the whole character, turnover, hand movement, finger movement and pulse becomes the basis for proper inference. After a holistic analysis, the application is in Phase 2: the test object in Phase 2 improves goggle glasses, looks around (Spin Base, Middle, Shoulder) adjusts glasses (Hand Centre, Hand Tip) and its excitement (Pulse) is clearly growing. In Phase 3 the tested object looks around (Spin Base, Middle, Neck) but also starts to get nervous (Thumb, Pulse), Phase 4 (in which the test object notices defect on the glass, crack) causes nervousness (Pulse), nervous behaviour (Spin Base, Hand Tip, Thumb, Wrist, Hand Center).

Conclusions
The set-up constructed with the use of various sensors and devices allows for effective, simultaneous collection of many signals during one research session, bringing no discomfort to the participant of the experiment.
During the test session and during the initial processing of the collected data, no disturbances were detected between the registered signals.
Separation of the 3D VR scene generation function from the data acquisition function by using two computer units was the right solution because the subjects did not comment on the appearance of discontinuities in VR, which is very important. The multi-channel recording of all data is extremely useful as it allows resigning from the analysis of signals that contribute little or no information on the behaviour of the subject, or are a duplication (strong correlation) of information from other signals, Fig. 8. part C.
The use of numerous heterogeneous signals allows for the correct and in-depth interpretation of the variability of individual recorded values.