Problems of using haptic feedback to control manipulator tools

The paper reviews the machine operators support technologies used in carrying out work tasks. The limitations of the current design solutions have been identified. The construction of control systems using haptic feedback was discussed and the research carried out on this technology was reviewed. The concept of hydraulic manipulator tool and its haptic feedback control system for monitoring loads during working movements is presented. The concept of the test stand for testing haptic feedback control systems and the preliminary experiment plan are presented.


Introduction
Construction machines and robots are advanced technical devices that, thanks to their design and attachments, help people in performing various types of tasks. The efficiency and effectiveness of a given machine is influenced by a number of its parameters and features, e.g. the type and structure of the drive, kinematics of working attachments called arms or manipulators too, used tools or working accessories and used control systems. There are numerous cases of research works aimed at improving the parameters of the machine and developing new technologies to be used in them [1][2][3][4][5][6][7][8][9][10].
The attachments and manipulators of these machines currently in use are designed to perform many different tasks. Grasping and picking up objects, soil excavation or cutting wires are only a few examples of the use of working tools. Their control systems are made in an open system structure and are usually based on the use of various types of joysticks and control devices using levers and buttons assigned to the movements of the manipulator links and tasks such as opening/closing the gripper or bucket. The challenge of control system designers is to find a compromise between intuitiveness and ergonomic handling and the functionality of a given manipulator. Commonly used design solutions are not free from drawbacks. Controlling the machine manipulator when it is in the operator's direct observation field is a relatively easy task, but when the working tool is outside the visibility zone, e.g. when the manipulator is controlled in teleoperation mode, the feedback signals that the operator receives have limitations. In such a situation, learning to control requires specialized training and practical exercises of the operator.
Emerging new technologies allow us to modernize the existing approach to the working tool control systems of construction machines in order to support the perception of their operators and reduce their learning time.

Limitations of machinery working tools
The example of a common problem of working tool control is the lack of sufficient feedback when operating hydraulic excavator manipulator, especially under the ground surface. In the era of highly developed underground infrastructure of cities, in particular power installations, gas pipelines and telecommunications lines, it is difficult to plan earthworks so that the excavation corridor does not interfere with the installation. There are numerous cases of damage to the installation during earthwork, which was caused by the lack of marking of the installation on surveying maps, and as a result, the costs incurred repairs of the installation is high.
Another example are the robot manipulators used, e.g. for operations to identify and neutralize improvised explosive devices. Such tasks are most often carried out in teleoperation mode, where the only feedback that goes to the operator is the image from the cameras deployed on the platform structure and its manipulator. The accuracy required of the operator, when performing such tasks, usually results in a long lead time of the task, which due to the nature of the object being manipulated is generally limited [11].
In order to overcome the constraints associated with these structures, modern machines are equipped with additional, deliberately designed, signaling systems to increase the situational and traction awareness of the operator using, e.g. warning acoustic signals, light signals using LEDs, based on information from inclinometric sensors deployed on the manipulator. Feedback information is provided on the operator panel, example of which is shown in Figure 1 [12].

Technologies to support the perception of machine operators
Frequently, the signals in the form of images, light or sound are insufficient. Especially in the case of tasks carried out in teleoperation mode or in the process of excavating soil, under whose surface there are obstacles. During this type of task, the operator is not able to directly observe the effector of the working attachment, which significantly increases the risk of making a mistake by him and leading to a collision with an obstacle and damage to the machine. In order to improve the perception of the operator, various technical solutions and technologies were investigated with the use different types of sensors.
There are studies of systems designed to support the work of hydraulic excavator operators, incl. a device scanning the area around the working machine using laser scanners. This solution is designed to locate and recognize objects around the working machine [13]. Another operator support technology is the distance sensing system to detect obstacles in the path of the working tool [14]. Some other example is the load monitoring system using a force sensor fixed between the bucket and the last part of the manipulator and a camera to observe the manipulator's motion. This system measures the forces affecting work equipment and transmits this information to the operator in the form of a numerical value on the camera image [15].
The development of control systems, sensor technologies and control devices has led to the start of research into the expediency and effectiveness of the use of haptic feedback in the control of technical devices which, by force or vibration, transmit additional information to the operator.

The essence of haptic feedback and its applications
Control systems using haptic feedback are capable of generating force or vibration of variable amplitude and frequency on the controllers as feedback from the actuator. In order to implement haptic feedback in the control system, sensors (e.g. forces, pressures, displacements) and components capable of generating haptic effects in the control device (e.g. servo mechanisms, actuators, vibrating motors) are necessary (e.g. servo mechanisms, actuators, vibrating motors) [16]. A number of proprietary control devices have been developed for haptic feedback in control systems of various types of devices, from mobile unmanned platform manipulators to heavy welding machine manipulators. These solutions differ in kinematics, the number of degrees of freedom, or the way haptic effects are generated. Despite the differences, the main task of all the solutions developed is to provide the operator with feedback on the inertia of the device they control [17,18,19,20,21,22].
Haptic feedback technology is used in mobile touch screens and vehicle control consoles (vibration effect), surgical robots (master-slave) and graphic design (virtual carving and modeling). Attempts were also made to use the haptic feedback control system to train manipulator operators in augmented reality [16,23].
Various studies have been conducted to determine the effectiveness of haptic feedback in improving operator perception when performing tasks. During the research [24], the operator's response time was compared using three types of feedback: visual, acoustic and haptic. The subjects were tasked with pressing the button when receiving the information of the particular type. The results of the studies initially indicated that the response time using haptic feedback alone is slightly longer than for the other two types of information. On the other hand, research [15] aimed at comparing the impact of different technologies to support operator perception when manipulating a small wheeled robot on a track with obstacles set has shown that the presence of haptic feedback in the control system significantly reduces the number of errors made during teleoperation compared to the use of stereoscopic image and high-resolution image. The technical solutions of surgical robots and manipulators for micromanipulation and micro-assembly of electronic systems are known, in which haptic feedback enabled teleoperation tasks to be performed on very small objects [26].
The test stand for the haptic feedback control system is also presented in [27]. Two axial fixed actuators were used, the piston rods of which were directed towards each other. One of them was a load, while the other simulated work tools, controlled by an electrohydraulic system using a haptic joystick, the distinguishing feature of which is the use of a magneto-rheological brake to generate a haptic effect. The subjects tested were tasked with slowly moving the actuator piston rod (working tool) until they believed that there had been contact with the obstacle. The test was performed in the non-haptic variant and with haptic feedback, in the latter case recording a much smaller error in indicating the point of application of the load.
With regard to machine manipulators, haptic control systems are most often made in a master-slave system. In this system, the operator performs movements with the levers of the control device (master -controller), which are scaled accordingly and transferred to the actuator (slave -e.g. manipulator) [28]. An illustrative diagram of such a solution is given in Figure 2.

Fig.2.
Example of the implementation of a control system with haptic feedback in a master-slave system [28].
Master-slave control can be carried out by mapping the controller work area to the manipulator's work area and controlling a point with known coordinates (so-called position control) or by giving the direction and speed of the actuator accessories -so-called speed control. Position control is used in cases where the movements of the controller lever are significantly greater than those of the manipulator's executive member, where its mass and inertia do not affect the accuracy of the movements. Speed control, on the other hand, is used when the movements of the controller lever are significantly smaller than the manipulator's operating movements and its mass and inertia have a direct effect on the delay of movements when position control is used [29].

Haptic feedback in machine manipulators implementation
The literature describes research of haptic feedback technology in machine manipulators. A common part of these studies is the willingness of their authors to determine the properties and appropriateness of using haptic feedback to obtain information about the inertia and repositioning of the controlled device, and the point that differs from each of them is how this feedback is implemented. The haptic control system, used in teleoperation mode described in [30], involves the use of a laser scanner to measure the bucket distance from the ground surface on an ongoing basis. Reducing this distance to a certain minimum value results in feedback being sent to the operator using a vibrating motor attached to his right hand. The purpose of the haptic feedback in this structure was to provide the operator with information on the start of excavating soil. The diagram of the implementation is shown in Figure 3.   Fig.3. Diagram of implementation of haptic control manipulator of a hydraulic excavator according to [30].
Another haptic feedback control was used in [31], where this feedback was intended to deliver information about the inertia of the manipulator of an 18-ton hydraulic excavator. The control strategy proposed copying the movements of the control device to the excavator manipulator with the correct gear ratio. The paper describes testing using the programmed trajectory of the working tool movement. When moving the controller levers, the operator does not experience haptic effects only if it follows the programmed movement track of the work tool. Each time it crosses the virtual walls, the controller will provide the operator with feedback in the form of vibration or stiffening of the controller levers relative to each other. An example of a programmable motion trajectory and the actual response on a test object are shown in Figure 4 and Figure 5.  A similar solution was used in [32] research in which haptic feedback was intended to provide information on the position and inertia of the movement of a hydraulic manipulator used for service tasks on power lines. The programmed motion trajectories for different types of tasks of such a design solution are shown in Figure 6. The backhoe-loader manipulator research is shown in [28,33]. The primary purpose of the haptic feedback in these research, as in the previous studies, was to enable the manipulator to feel dynamic loads and inertia. The main idea of the presented method is to map the workspace from controller to manipulator, and then compare the repositioning of both devices. The offset error is perceived as an unforeseen load on the working tool, which is indicated by the controller with the appropriate haptic effect. The implementation scheme is shown in Figure 2. Linear and angular displacement sensors in the manipulator joints, as well as pressure transmitters, were used to implement the feedback.
The test results described in the above publications define haptic feedback as a technology that improves the perception and accuracy of the machine operator. However, most of the available papers describe the use of programmed working tool motion trajectories and pressure and position transmitters to implement feedback, whose main task is to facilitate the performance of repetitive and predictable tasks and to provide the operator with information about the dynamic loads and inertia of the manipulator during motion.
There is a noticeable lack of research to gain knowledge on the use of haptic feedback to monitor loads and forces during working movements, e.g. load on the cutting edge of the bucket during excavating, with feedback in the form of a haptic effect on the controller.

Concept of the test stand
The construction of the proposed test stand is intended to provide knowledge during the research on the possibility of using haptic feedback to monitor loads on tools during objectgrasping tasks and ripping the ground with a hydraulic manipulator ripper. The tasks are intended to provide an answer to the question whether haptic feedback is capable of providing information that the operator understands about loads during such works. The developed concept assumes the use of a commercial controller, equipped with three servo mechanisms, which allow to generate various types of haptic effects. The signal from the control device, properly processed using the software on the PC, is transmitted to the Danfoss Plus+1 microcontroller. The microcontroller is designed to process commands from the control device to the hydraulic manifold open/close signal, which controls the hydraulic actuators that drive the manipulator and its attachments.
The manipulator's work tool is a proprietary gripper solution that will be used to perform object-picking up tasks. One of the jaws is equipped with a ripper tooth, which is intended to allow loosening of the ground to the IV category of excavating difficulties. Two hydraulic actuators are responsible for the drive of the gripper, which through the linkage system set both jaws in symmetrical motion. The 3D model of the proposed solution is shown in Figure  7. The gripper jaws are equipped with force sensors which, thanks to the appropriate fixation, read the actual value of the gripping force and, for the jaw with a ripper tooth, the loads on the cutting edge. The location of the force sensors is shown in Figure 8. Additional use of the tensometers affixed to the gripper jaws and pressure transmitters in the drive actuators is envisaged in order to compare the effectiveness of the feedback using three different sensors during the test process. The signal track for the assumed construction of the test stand is shown in Figure 9. In order to facilitate the intended task of ripping the ground, it is possible to disassemble one of the gripper jaws in order to increase the available range of motion of the ripper tooth. Working tool after this modification are shown in Figure 10. The preliminary plan of the experiment involves conducting haptic feedback control system tests during the tasks of taking on different types of objects and ground ripping. It is assumed to perform tests in different variants, e.g. picking up objects of different geometry and made of different susceptibility materials. It is also planned to determine the ability of the haptic feedback control system to transmit information on objects beneath the surface of the ground, in the form of e.g. electrical wires or metal bars.

Conclusion
The use of haptic feedback in the control of mobile equipment manipulators is a topic explored by many researchers, and each of them presents a different approach to how such a system is implemented. A number of control device concepts and test stands have been developed to test the properties and effectiveness of such feedback. Those publications confirm the expediency of the use of haptic feedback in the performance of repetitive tasks in which it is possible to predefine the path of movement and to anticipate obstacles in the path of effector.
The proposed solution of the manipulator's working tool and test stand will allow to supplement the lack of knowledge in the area of research on this technology, on how to use it to monitor loads of working attachment during operation. After completion of the development process and construction of the test stand, a series of tests is planned, the characteristics of which correspond to the tasks actually carried out with the use of this type of equipment (e.g. taking up objects, loosening the soil). This research can contribute to the development of a technique for detecting objects beneath the surface of the ground, e.g. during earthworks. Confirming the effectiveness of haptic feedback in the performing of tasks under the ground surface or in teleoperation mode can make a significant contribution to the development of manipulators of hydraulic excavators, backhoe loaders, as well as manipulators of robots and other unmanned land platforms.