Analysis of the advancements in real-life performance of highly automated vehicles\' with regard to the road traffic safety

The article presents the analysis of the performance of the vehicles equipped with automated driving systems (ADS) which were tested in real-life road conditions from 2015 to 2017 in the state of California. It aims at the effort to assess the impact on the road safety the continuous technological advancements in driving automation might have, based on of the first large-scale, real-life test deployments. Vehicle manufacturers and other stakeholders testing the highly automated vehicles in California are obliged to issue yearly reports which provide an insight on the test scale as well as the technology maturity. The so-called 'disengagement reports' highlight the range and number of control takeovers between the ADS and driver, which are made either based on driver's decision or information provided by the vehicle itself. The analysis of these reports allowed to investigate the development made in automated driving technology throughout the years of tests, as well as the direct or indirect influence of the external factors (e.g. various weather conditions) on the ADS performance. The results show that there is still a significant gap in reliability and safety between human drivers and highly automated vehicles which has been yet steadily decreasing due to technology advancements made while driving in the specific infrastructure and traffic conditions of California.


Introduction
The upcoming transition towards driving automation is predicted to significantly change many paradigms of both road transport and automotive. Introduction of automated driving will not only greatly influence the legal regulations, but also provide a space for the emerging of new business models and deepen the importance of safety in human-machine interaction systems. Before reaching the full autonomy, the intermediate stages allowing the vehicle to operate independently in certain conditions (levels 3-4 of SAE J3016 taxonomy) are expected to be certified for public and private use. One of the greatest concerns in this area is how the safety of road users will be affected by implementing highly automated vehicles, both in micro scale (efficient handling of automation functions) and macro scale (change in the rate of road casualties).
Highly automated vehicles have been tested in recent years throughout the world by both the vehicle manufacturers and software developing companies. Yet, there are certain limitations deriving from the national and international legislative acts which do not allow to take tests in real traffic conditions. The recent amendments of 1968 Vienna Convention on Road Traffic, the major international act regulating the principles of road traffic, are targeted specifically at driving automation and point out that 'every driver shall at all times be able to control his vehicle (...)' and 'every driver of a vehicle shall at all times have his vehicle under control so as to be able to exercise due and proper care and to be at all times in a position to perform all the manoeuvres required of him' [1]. Despite of these advancements, the regulatory framework of the Convention's signatories for type approval procedures of highly automated vehicles has not yet been harmonized by the UNECE WP29 (United Nations' working party responsible for the vehicle regulations). Therefore, the vehicles with high automation functions and/or with no driver on board are still not allowed to be tested in real traffic in the majority of states, despite the efforts taken by the EU to modernize the European type approval framework legislation [2]. Yet, countries like China and the USA did not sign the Convention which allowed them conduct less strict regulatory policy in this matter and introduce the proving grounds for the automated driving technology. Since 2011, a total of 33 U.S. states (i.e. Nevada, Arizona, Oregon, Michigan, California) authorized the tests of automated and autonomous vehicles on their territories [3].

The analysis
It is estimated that around 94% of road accidents involve some level of human error [4]. It is therefore crucial to investigate how different scenarios of connected and automated driving technology development can enhance the road safety. Taking into account the fact that the implementation of automated vehicles will also depend on policy-making, it is essential to analyze these factors as well.
Taken analysis was aimed at the detailed investigation of the disengagement reports issued to the Department of Motor Vehicles of the California state by the testing parties from 2015 to 2017 [5,6]. The reports are publicly available and can be found on the Department's website. The analysis on one hand allowed to indicate the technological advancements made by the vehicle manufacturers, while on the other hand revealed the factors which significantly decrease the performance of the on-board automated driving systems.

The structure of disengagement reports
All manufacturers willing to deploy the automated driving tests in the state of California are obliged to obey the California Autonomous Vehicle Testing Regulations approved by the Office of Administrative Law. Testing regulations establish an Autonomous Vehicle Tester Program and are steadily updated due to the needs of testing the higher levels of driving automation -the recent update from 26th February 2018 allows to provide fully driverless tests. Still, this article focuses at the tests made until 2017 in which an on-board driver assistance was compulsory.
The regulations require every testing party to submit an annual report (the so-called 'disengagement report') which summarizes the number and reasons behind the disengagements resulting in taking over the control by the driver. According  In fact, the reports issued by different vehicle manufacturers are rather loosely structured and are not consistent with each other. The reason behind is that there is no specific template to be filled. Therefore, it provides certain difficulties in comparing the specific automated driving technology performance results. To provide an example, Waymo company provided the following division of disengagements causes: -disengage for a recklessly behaving road user, -disengage for hardware discrepancy, -disengage for unwanted maneuver of the vehicle, -disengage for a perception discrepancy, -disengage for incorrect behavior prediction of other traffic participants, -disengage for software discrepancy, while Delphi Automotive listed the following issues behind the disengagements: -heavy pedestrian traffic, -other driver unexpected behavior, -emergency vehicles, -complete lane change, -cyclist, -traffic light detection, -construction zones. As one can see, the different reasoning behind the disengagements could be caused not only by the lack of cause terms' harmonization, but also due to different naming used by the companies. For the sake of a detailed analysis it is therefore important to provide further harmonization of the disengagement reports.

Method
The research was taken in different areas believed to affect each other in preliminary phases of automated vehicles' deployment. Firstly, it was investigated to what extent the driving automation could affect the road safety during the mixed traffic stage of technology implementation. This was made based on the biggest, yearly updated database of autonomous vehicle testing performance -socalled 'disengagement reports' which reveal the number of takeovers made by human testers due to automation systems' inefficiency or operator's safety concerns. These reports reveal not only the technological state-of-the-art, but also the level of trust given so far to the vehicles by both the OEMs and operators. The investigation was based on the data collected from years 2015-2017 from the world's 2nd biggest automated driving testing site -the state of California.
Secondly, based on the findings from the driving automation technology progress revealed in disengagement reports, the safety-targeted policy adoption was proposed. The analysis was made based on RAND Model of Automated Vehicle Safety. This model confronts the proposed future changes in road travels, automated vehicles' policy adoption scenarios and the safety level acceptable for their deployment [7].

Fig. 1.
Example of a take-over request information from a commercially available passenger vehicle. Source: https://but.ly/2uY8CQk.
Since the efficient performance of SAE L3-4 vehicles will still depend on human performance, the aforementioned analyzes were also compromised with the research aiming at finding the most efficient HMI-based communication strategies in take-over request context. The research was taken in scope of aDrive project which was led by the Motor Transport Institute. The research scenarios were covered in driving simulation conditions, on a representative sample of 30 active drivers who owned B licence holders for at least two years [8].

Results
A total of 409 automated vehicles, owned by 19 companies were tested in California from 2015 to 2017. They covered a total 2,597,112 km. The analysis was based on the data gathered from 5 companies which continued the tests during all three years. The total size of their fleet changed from 69 in 2015 to 85 in 2017. Coverage of their vehicles accounted for 90% of all kilometres driven in autonomous mode.
Since the disengagement is an event during which the operator takes over the control of the vehicle due to either automated driving system malfunction (e.g. unwanted maneuver of the vehicle) or road safety caution, lack of his/her intervention can be identified as the basis of a possible road accident. Therefore, the number of occurred disengagements was compared to a number of road injuries and fatalities in the EU and US in 2015. Using this criteria, it turned out that, depending on the tested solution, that driving automation technology is still from 2 to 3 orders of magnitude less reliable than human driver. Yet, a significant decreasing trend in this area can be observed. Since driving automation strongly relies on machine learning algorithms fed by collected real-life data, obtaining additional datasets were confirmed to be crucial. During the course of testing, it turned out that the increase (or at least lack of decrease) of fleet size and covered distance significantly affect the efficiency of driving in autonomous mode.  It was calculated that an autonomous vehicle would have to drive 442.5 million failurefree kilometres to demonstrate the performance of a fatality rate relevant to human driver's with C=95% confidence level. Yet, this number drops to 1.450.000 million kilometers in case of reported injuries rate. Therefore, it was estimated that even in the case of automated vehicles being adopted in less than 10 years from now as only half as efficient as human drivers, around 9% of road casualties could be saved.
The results of driving simulation-based human-machine interface research revealed that the efficiency of driver performance in L3-L4 automated driving scenarios can be shaped by targeting different senses with a set of stimuli. It turned out that the mixture of visual & auditory information provides driver to become 2 times quicker during the takeover request than other stimuli.

Conclusions
The conducted investigation was of a preliminary nature. Still, it suggests that, despite a still evident gap between the efficiency in driving performance between the highly automated vehicles and human drivers, the early, large-scale deployment of self-driving vehicles might actually be more efficient in a longer term, than the continuous, timeextended process of fleet replacement. Results of the continuous testing conducted in California so far reveal that the facilitations provided for their upscaling could enable a quicker path towards more efficient autonomous vehicles, despite the likeliness of initial decrease in road safety. The reason behind this lies in the noticeably high progress made by the driving automation systems' performance which is based on the machine learning algorithms, advancing when being fed with large amounts of data. The policy makers are then recommended to proceed a quicker path to allow large-scale test deployments, not waiting for the automated vehicles to become safer than humans. Additionally, stress of the decision makers should be put on the standardization & certification of dedicated HMI systems which will play a crucial role in maintaining the role of a driver in complicated traffic situations. Still, it needs to be underlined that the analyzed tests were made in specific road conditions, including the road infrastructure and cultural conditions. Additionally, since the dataset obtained from the disengagement reports is limited, more in depth analyzes need to be made as soon as more detailed information is provided. Therefore, it is necessary to continue further large-scale tests in different, local traffic ecosystems in order to ensure if the highly automated vehicles will be able to efficiently perform in various infrastructure, traffic and environmental conditions. aDrive system was financed and supported by the project aDrive -PBS3/B6/28/2015, funded within Applied Research Program of the Polish National Centre for Research and Development.