Real time emergency auto parking system in driver lethargic state for accident preventing

This paper is presenting a safety driving and accident preventing system which uses a vision sensor to detect driver drowsiness and lethargic states. The system notifies the driver in dangerous situations. Moreover, in case the driver is unable to conduct safe driving, emergency parking system is to be activated. The system comprises two stages. First is a drowsiness detection stage which uses a smartphone or a tablet computer as a processing unit. The second stage is the vehicle emergency parking control system which uses a microcontroller unit (MCU). The MCU is connected to an alarm system, hazard lights and a vehicle control interface. The experiment results showed realistic real time responses. Drowsiness detection processing time average is about 480 ms / frame. Alarming system is responding perfectly within 500 ms. Simulation results illustrate the effectiveness of the developed schemes for the auto parking system in real time. The average time from drowsiness detection to fully parking, if the vehicle is moving at the speed of 100 km/s, is about 15s.


Introduction
Traffic accidents are serious global problem.Studies show that sleep and fatigue are involved in around 10 percent of accidents [1,2].According to National Highway Traffic Safety Administration (NHTSA), annually from 2009 to 2013, on average there were over 72,000 police-reported crashes involving drowsy drivers.Therefore, it is suggested by many road safety related entities to develop driver alarm and assistance systems to reduce the number of accidents.For example, Malaysian Institute of Road Safety (MIROS) introduced what is called "research pillars".Pillar 3 is "Safer Vehicles" [3].
Driver fatigue detection has been studied during the last two decades by many researchers.The causes of fatigue can be physical, physiological, or psychological [4].Unfortunately, there is no standard fatigue measurement, because the direct measures are few.A recent comprehensive review on various commercial and academic literatures is available in [5].A quick summary is presented in this introduction.Most of the research methodologies can be divided into four categories: Methodologies which are focusing on the driver's current state, driver performance, a combination of the driver's current state and driver performance and methodologies focusing on vehicle's current state [5].Furthermore, few commercial products are already available in markets.For example, the Lumeway eye alert Distracted Driving and Fatigue Warning Systems, Toyota driver attention monitor, DADSTM (Driver Alertness Detection System TM ), Fatigue Monitor, and many other systems.Few systems use methods depending on inputs from devices which are not visual sensors, like [6].But most of the algorithms are depending on visual information.
The outcome of most of the systems is an audible alarm.Few number of researchers suggested different ways for overcoming drowsiness.For example, Singh H. et al. [7] proposed a system which gives warning output in form of sound and seat belt vibration.The alarm is deactivated manually.In 2009, Mercedes-Benz unveiled a system called "Attention Assist" [8].The system monitors the driver's fatigue level and issues a visual and audible alarm.The significant feature in this system is the linking with the vehicle's navigation system.This allows the system to tell the driver where coffee is available.In 2016, the latest version of "Attention Assist" not only makes the driver aware of behaviour that may be influenced by drowsiness, but also helps driver to correct minor errors [8].
The practical use and efficiency of the devices in preventing accidents are still under inspection.United Kingdom royal society for the prevention of accidents (RoSPA) published a literature review on driver fatigue and road accidents [9].The study investigated number of technical devices.The study concluded that such devices may prove beneficial, but there are concerns that drivers would rely on them instead of managing themselves for safety.The study raised the question: "Drivers are normally aware that they are sleepy, so why is a device necessary to tell them so?" Therefore, more efficient actions should take place to achieve the main goal of preventing and reducing accidents.In this paper a new emergency auto parking system is proposed.To the best of our knowledge, there are studies for autonomous cruise control system [10], but no studies are available for parking on road side in case of emergency.The proposed system is activated when the driver is unable to conduct safe driving.The purpose of this system is to automatically control the speed and the steering of a vehicle to park it safely on the emergency lane.
Smart systems for vehicle parking are not new field of research.They are attracting a great deal of attention recently.Parking systems in general deals with two categories: Garage-parking problem and parallel-parking problem [11].Figure 1 (a) and (b) shows the difference between garage-parking and parallel-parking.The target of such systems is to help the driver to park the vehicle in a limited parking space [12].The proposed system in this paper can be categorized to another category which is emergency parking.Figure 1 (c) shows the emergency stopping lane which is the destination for vehicle parking in emergency cases.

Emergency parking system
Figure 2 shows the flowchart of the proposed system.The system comprises two stages.The first stage is drowsiness detection, which uses the built-in camera of a smartphone.The video frames captured by the camera are processed by an Android application (App).The output of this stage is received by the second stage, namely, the vehicle emergency parking control system.The second stage uses a microcontroller unit (MCU).The MCU is connected to an alarm system, hazard lights and vehicle control interface.

Drowsiness detection
There is no standard fatigue measurement system.Most measures are of the outcomes of fatigue rather than of fatigue itself [5].Moreover, there is no study which links the effect of fatigue and driver's ability to drive.Consequently, no robust model or algorithm for risk assessment based on fatigue is currently available.In the proposed system, fatigue is evaluated by monitoring drowsiness effect on driver's eyes.A smartphone camera is used to capture driver face.An Android App is used to process the video frames captured by the camera.
Smartphones and tablet computers are increasingly becoming popular among researchers and developers.Researchers are recently using these devices for powerful controlling applications, like robot operating [13].Image processing and pattern recognition applications performance on smartphone platforms are still slower than the performance on personal computers.However, the performance is reaching acceptable levels [14].Therefore, smartphone can be considered as efficient processing unit for fatigue detection.
To monitor drowsiness effect on driver's eyes, the first step is face detection.Face is detected by applying Viola and Jones method [15].The method works by applying masks which are rectangles contain a small number of Haar wavelets.Haar descriptor for one mask defines a weak classifier.A set of weak classifiers is used to decide whether there is a face in the window occupied by the masks.Slide sets of masks through the image; decide whether a given window contains a face or not.The Haar feature-based cascade classifiers using an AdaBoost classifier are applied on driver face images of the video stream.AdaBoost, which is an adaptive machine-learning meta-algorithm, cascades weak classifiers to build a strong classifier.The cascade function is trained using a set of positive images (images of faces) and negative images (images without faces).
After detecting the face, the region which includes the face, is extracted.The next step is eye detection.The area of the face that contains the eyes is analyzed using the Haar cascade algorithm again.To detect eyes, the cascade function is trained using two classes: Images of open eyes, and images of closed eyes.Eyes data are used to calculate eye blinking frequency.The blinking frequency parameter is expressed as the percentage of number of images with closed eyes to the total number of images at a fixed interval of time.If the blinking frequency is greater than 80%, the driver is considered asleep and the state is considered as a state of emergency.Driver state data are sent to the second stage via the Bluetooth module.

Parking system prototype and simulation
The diagram of the second stage processing unit is shown in Figure 3.The unit receives eye closure data from the smartphone via Bluetooth module.If closed eyes are detected and the state is evaluated as state of emergency, the MCU fires the alarm signal.Hazard lights and the automatic control system of the vehicle are activated if the driver does not respond within 3 seconds.The driver may respond by manually pushing a button that is connected to the second stage processing unit to turn off the alarm and the automatic vehicle control system.A prototype of the system is built using Arduino MCU, which is connected to four light-emitting diodes (LEDs), a buzzer, a push-button switch, and a motor as shown in Figure 4.
As long as the driver is not responding, the second stage processing unit will continuously determine braking and speed-reduction scenarios based on vehicle speed and the distance of any other vehicle in its rear.To determine braking and speed-reduction scenarios, parameters such as steering angle, throttle input, and forward velocity are considered as shown in Figure 5.The control system of braking, speed reducing, and steering control is simulated using MATLAB-Simulink.The simulation is performed according to a proportional-derivative (PD) controller and the mathematical model shown in Figure 6 and Equations ( 1) and (2).
where T u is the steering angle input, v u is the throttle input, t x is the current lane position according to road width, ref Steering angle input is taken into account to make sure that the vehicle forward velocity gradually decreases according to distance of the controlled vehicle from the emergency lane.
Based on the gain coefficients of the PD controller, For example, in case the desired location according to the lane width is the emergency lane, the steering angle input, T u changes from "0" to a negative value.When the steering angle input, T u is changed, the desired vehicle forward velocity, y ref v is reduced accordingly due to the negative value of the steering angle input until the value of the steering angle input changes back to "0".The braking scenario is based on reduction of velocity of the vehicle wherein the steering angle input value, T u is equal to "0".

Results and discussion
The proposed system was tested on several video sequences captured for different people.The temporal sampling rates are 30 frames per second.When the system detects closed eyes and state of emergency, alarm system is fired.If the driver does not respond within three seconds, the electrical motor is started in the prototype.
Face and eye correct detection rate was 95%.After detecting the face, eyes are extracted from the face to determine eye closure according to the described algorithm.An example of detecting face and eyes is shown in Figure 7. Drowsiness detection processing time average is about 480 ms / frame.After drowsiness detection the data are sent via Bluetooth module to the second stage.The second stage processing unit triggers the alarm signal within 500 ms.
Further tests were done using the MATLAB simulation.The MATLAB simulation results illustrated the effectiveness of the developed schemes for the automatic emergency parking system.It showed real-time response for the testing cases.Figure 8 shows snapshots of the vehicle parking simulation system.The system is taking in consideration the surrounding vehicles.Other vehicle positions and speed data are fed to the controller to allow it to control the vehicle safely.In reality, such data are collected using a Hall Effect speed sensor and a rear ultrasonic or laser distance sensor.Simulation results illustrate the effectiveness of the developed schemes for the auto parking system in real time.The average time from drowsiness detection to fully parking, if the vehicle is moving at the speed of 100 km/s, is about 15 s.The Processing times for the entire emergency parking system actions are summarized in Table 1.The complete process requires about 19 seconds to park the vehicle safely on the emergency lane.This system is activated in case the driver is unconscious or unable to conduct safe driving.
The successfully simulated system is to be applied on the all-terrain vehicle (ATV) shown in Figure 9.The main constraint for applying it on the prototype ATV is designing a

Conclusions
A safety driving assistance system is presented.The system notifies the driver in dangerous situations and activates an emergency parking system in case the driver is unable to conduct safe driving.The alarm system responds perfectly.The simulation results illustrate the effectiveness of the developed schemes for the automatic emergency parking system.The complete process requires about 19 seconds to park the vehicle safely on the emergency lane.This time is considered acceptable for preventing, or at least reducing, the sleep and fatigue related accidents.
Next step is to design a mechanical automatic steering system to apply the model on the ATV.The effectiveness of the detection method and the risk evaluation method in relation to fatigue are not discussed.The study is focusing on eye closure as an important parameter for fatigue detection.More robust evaluation methods for detecting driver fatigue and linking it with its effect on driver's ability to drive is needed.Moreover, it is recommended to improve the classification method by detecting three or more levels of eye closure: open, half open, and closed.Other parameters like gaze duration, looking direction, yawning, blinking frequency, and vehicle movement are suggested to be involved in fatigue and risk evaluation.

x
is the desired lane position according to road width, y t v is the current vehicle forward velocity, and y ref v is the desired vehicle forwarding velocity.The steering angle input, T u relates to the desired vehicle position according to the lane width, ref x ; the current vehicle position according to lane width, t x ; and the desired vehicle sideways velocity, x ref v .The lane width x relates to average travelling speed of vehicle on each lane of the road.The lane width x may include slow lane, fast lane and emergency lane.The throttle input, v u is related to the steering angle input, T u , current vehicle forward velocity, y t v and final vehicle forward velocity, y ref v .When the steering angle input, T u is changed towards the emergency lane, the vehicle forward velocity, y t v gradually decreases until the vehicle arrives at the desired location which is the emergency lane.The current vehicle forward velocity y t v is to be determined by a speed sensor.Furthermore, the desired vehicle forward velocity, y ref v relates to changes of steering angle input, T u wherein if the vehicle is moving straight with no changes of steering angle input, the desired vehicle forward velocity is equal to "0".The value of the desired vehicle forward velocity y ref v is targeted to be "0" which means the controlled vehicle stops on the emergency lane.
according to vehicle physical specifications, the steering angle input, T u and throttle input, v u are computed, wherein the steering angle value of "0" indicates that the vehicle is in a DOI: 10.1051/ , negative values indicate that the vehicle is moving to the left, and positive values indicate that the vehicle is moving to the right.

Fig. 9 .
Fig. 9.The prototype vehicle with the speed sensor.

Table 1 .
Processing times for the entire emergency parking system actions.