Extending Driving Vision Based on Image Mosaic Technique

. Car cameras have been used extensively to assist driving by make driving visible. However, due to the limitation of the Angle of View (AoV), the dead zone still exists, which is a primary origin of car accidents. In this paper, we introduce a system to extend the vision of drivers to 360 degrees. Our system consists of four wide-angle cameras, which are mounted at different sides of a car. Although the AoV of each camera is within 180 degrees, relying on the image mosaic technique, our system can seamlessly integrate 4-channel videos into a panorama video. The panorama video enable drivers to observe everywhere around a car as far as three meters from a top view. We performed experiments in a laboratory environment. Preliminary results show that our system can eliminate vision dead zone completely. Additionally, the real-time performance of our system can satisfy requirements for practical use.


Introduction
In order to reduce car accidents, many electronic devices have been introduced to assist driving, such as parking sensors and Rear-View Car Cameras (RVCCs) [1].RVCCs facilitate driving enormously by making it visible.However, the Angle of View (AoV) of existing RVCC is less than 180 degrees, that is, there exists dead zone outside our view, which is a primary origin of traffic accidents.To resolve the problem, we design a vehicle panorama system, which can extend the vision of drivers to 360 degrees around a car.
Our system comprises four wide-angle cameras, each of which has an AoV near to 180 degrees.The cameras are mounted at different sides around a car.Obviously, the cameras can capture 4-channel videos around a car.Via switching among the videos, drivers can observe everywhere around a car.However, switching videos may cause distraction from driving.Additionally, the experience is bad to observe four distinct videos, especially when videos overlapped with each other.In order to mitigate the problem, we seamlessly integrate the 4-channel videos into a panorama video.The image processing algorithms used by our system are camera calibration, image distortion correction and image mosaic.The output of our system includes a panorama video and a side video.The panorama video enables drivers to observe the environment around a car as far as three meters from a top view.The side video can be switched among the four cameras automatically.In order to investigate the effect of our system, we conduct experiments based on a mimical car and good results are achieved.
The contributions of this paper are: • Hardware design of a driving assist system based on DM6437, which can extend driving vision to 360 degrees around a car; • Image processing algorithms used by our system; • The investigation of the effect of our system.The rest of this paper is organized as follows: Section 2 presents the general view of our system.Section 3 introduces our image mosaic technique.Section 4 reports our preliminary experimental results.Section 5 concludes this paper.

General view of our system
The hardware of our system mainly consists of the following components: CCD Cameras, TVP5158, DM6437, DDR2 RAM, NOR Flash, Video Output Connector and CH376.Its functioning is illustrated in Fig. 1.As we can see, it captures video images through four wide-angle cameras mounted at different sides of a car.The video decoder TVP5158 digitizes and decodes all popular base-band analog video formats into digital video output [2].The main processor DM6437 [3][4][5][6][7] takes the digital videos as input and processes images based on OpenCV [8].The storage of our system comprises a 16M NOR Flash and a 128M DDR2 RAM.The former is used to save programs and configuration information.The latter is used to cache videos.Additionally, relying on the file management controller CH376, our system can upgrade software through USB flash drives or SD cards.
Based on the above design, our system can provide drivers with a panorama image around a car from a virtual view point as illustrated in Fig. 1.The virtual view point is above the centre of a car.Its height can be assign a value between zero and ten meters.By raising the virtual view point, we can broaden the panorama view to a maximum extent of three meters around a car.

Image processing
Our system processes images based on the open sourced computer vision library OpenCV.In this section, we elaborate the key image processing algorithms used by our system, inclusive of camera calibration [9,10], image distortion correction [11,12] and image mosaic [13,14].

Camera calibration
Camera calibration is a necessary step for many image processing tasks, such as 3D computer vision and image distortion correction.Through camera calibration, we can achieve the following metric information about a camera from 2D images: • The extrinsic parameters (R, t) of a camera, where R is a 3 × 3 rotation matrix, t is a 3D translation vector; • The intrinsic matrix A of a camera, which is given by The extrinsic parameters and intrinsic matrix relate the world coordinate system to the camera coordinate system.Let m and M ෩ be a 2D and a 3D point respectively (homogeneous coordinates).The relationship between a 3D point M ෩ and its image projection m is given by where s is an arbitrary scale factor.The distortion coefficients are essential for the image distortion correction.
We calibrate cameras using the method proposed by Zhang [10], which only requires the camera to observe a planar pattern shown at a few different orientations.As shown in Fig. 2, our system utilizes a chessboard for calibration.It contains a pattern of 7 × 5 squares, so there are 24 internal corners (i.e., points where the black squares touch each other.)The size of the chessboard is 12.9 × 9.2 centimeters and that of each square is 1.9 × 1.9 centimeters.We print the chessboard with a high-quality printer and then put it on a plastic board.Before calibrating a camera, we hold the plastic board by hand and place it under the camera as far as its image fills in the whole screen.Our system takes eight gray images of the chessboard as input of the calibration algorithm (the gray image can be obtained by extracting the Y channel of an image in format of YCbCr-422.)Since the hand will inevitably shake, the eight images must be under different orientations.
The working principle of the calibration algorithm is shown in Fig. 3, which mainly consists of three OpenCV functions, i.e., cvFindChessboardCorners(), cvFindCornerSubPix() and cvCalibrateCamera2().The first function attempts to determine whether the input image is a view of the chessboard pattern and locate the internal chessboard corners.A non-zero value is returned if all of the corners are found and ordered.Otherwise, it returns zero.What should be noted is that the coordinates detected by cvFindChessboardCorners() are approximate.To determine their position more accurately, we should subsequently call the function cvFindCornerSubPix() which can refine the results obtained by cvFindChessboardCorners() and achieve the sub-pixel location of corners.After that, we compute the world coordinates of the corners based on the metric information of the chessboard.Finally, we take the camera coordinates achieved by cvFindCornerSubPix() and the computed world coordinates as input and call the function cvCalibrateCamera2() to estimate the intrinsic camera parameters, distortion coefficients and extrinsic parameters for each of the chessboard images.

Image distortion correction
As shown in Fig. 4, serious distortion exists in the images captured by our system.To facilitate image mosaic, we should correct them.We perform the image distortion correction based on the OpenCV function cvUndistort2().Let's assume that G' is the corrected image of G. cvUndistort2() corrects image G by mapping each pixel of G' to a pixel in G.The mapping is calculated based on the intrinsic matrix A and distortion coefficients k1, k2, p1 and p2, which are achieved by the above camera calibration.The corrected image output by cvUndistort2() has the same size and type as the input image.What should be noted is that the function cvUndistort2() only works with gray images.

Image mosaic
Our system performs the task of image mosaic based on a two-phase approach.The first phase is carried out only once, in which we calculate and save the image projection parameters into the Flash.The second phase is executed in each frame, which integrates the 4-channel images into a whole based on the projection parameters achieved in the first phase.Below we will elaborate our approach in detail.
As we can see from Fig. 5 (a), Ow-XwYwZw is the world coordinate system.Its origin is located on the ground, at the center of the car.We assume that there exists a virtual camera, which is h meters high above the center of the car.Its camera coordinate system is Oc-XcYcZc.The goal of our image mosaic technique is to project images captured by four cameras into the image Iv, which is output by the virtual camera.Let I1, I2, I3, I4 be the corrected images captured by the front, rear, left and right cameras respectively.In order to project these images to Iv, we should achieve the homography matrixes Hi, i = 1…4, which are between Ii and Iv, respectively.As shown in Fig. 5 (a), we compute the homography matrixes based on four chessboards, which are the same as that used for camera calibration, except that they have different sizes.The size of the chessboards used for image mosaic depends on the installation height of cameras.To prepare for image mosaic, we lay all chessboards on the ground, around the car.The front and rear ones are in the middle of the car.The left and right ones are l1 and l2 meters off the front of the car.What should be noted is that, the chessboards should be placed in a position where their images fill in the screen as much as possible.Obviously, based on the above metric information and the size of the car, we can calculate the world coordinates of all chessboard corners conveniently.Assuming Pw to be the set of corner points (world coordinates) of the front chessboard, our method of computing the homography matrix proceeds as follows: 1) Projecting Pw to the set of image points Pv from the perspective of the virtual camera.This task can be accomplished with the OpenCV function cvProjectPoints2(), which takes Pw and the parameters of the virtual camera as input.2) Finding the set of corner points P1 from image I1 captured by the front camera.According to the introduction of camera calibration, this task can be carried out by using OpenCV functions cvFindChessboardCorners() and cvFindCornerSubPix(). 3) Calculating the homography matrix H1 based on Pv and P1 achieved in step 1) and 2) respectively.We perform this task leveraging the OpenCV function cvFindHomography().4) Computing the inverse matrix ି of H1 by using the OpenCV function cvInvert().
where P is a point (homogeneous coordinates) in image Iv.Given an image I and a point p in I, clr(I, p) represents the color of pixel p in I. Icar denotes a car image, which is in the format of RGB565.S is a 3D vector (sx, sy, 1), where sx and sy are the horizontal and vertical scale factors between Icar and Iv respectively.R0~R4, R13, R14, R23 and R24 are different regions of image Iv, which are shown in Fig. 5 (b).According to the composition of pixels, we can divide these regions into three categories: 1) Region R0 is the projection of the car, which will be filled with a car image.Its size can be determined according to the projection of car corners.2) Regions R1, R2, R3 and R4 comprise pixels coming from I1, I2, I3 and I4, respectively.3) Regions R13, R14, R23 and R24 are overlapped regions, that is, a pixel of the regions may be the fusion of two pixels originating from different cameras.For example, a pixel of R13 may be a mixture of two pixels coming from images I1 and I3 respectively.For an overlapped pixel, we compute its color based on the mrg function, which takes two source pixels as input and outputs the mixed pixel.The mrg function is crucial for our system.Taking a pixel ‫,ݔ(‬ ‫)ݕ‬ ∈ ܴ ଵଷ for example, we utilize the following mrg function in this work: (2) where ‫ݓ‬ ‫ݔ(‬ , ‫ݕ‬ ), ݅ ∈ {1,3} is the weight function.Its form is given as follows: Additionally, we have ‫ݔ(‬ ‫ݕ‬ 1) ் = H ିଵ ‫ݔ(‬ ‫ݕ‬ 1) ் According to [15], the image mosaic technique based on the above weight function should work well.
In order to improve the real-time performance of our system, for each pixel (x, y) in image Iv, we save the following information into the Flash in the first phase: 1) pixel (x1, y1) in the first source image; 2) pixel (x2, y2) in the second source image; 3) the weight of the first source pixel; and 4) the weight of the second source pixel.If a pixel has only one source pixel (e.g.pixels in regions R0~R4), we invalidate the other source pixel by setting its weight to zero.Based on the results of the first phase, we can generate the panorama image Iv from (2) directly in each frame.Our method is efficient, because the first phase is performed only once.In each frame, we carry out only two multiplications and one addition for each pixel.

Experiments
In order to investigate the effect of our system, we performed experiments based on a mimical car, which is a cardboard box of the size of 1.4 × 1.2 meters.The chessboards used for image mosaic have the same size of an A4 paper.We mounted the cameras on the four sides of the cardboard box and launched our system.The effect of our system is shown in Fig. 6.As we can see, the panorama video is displayed on the left side, from which we can observe everywhere around a car from a top view.The side video is displayed on the right side, which can be switched automatically among the four cameras.
We also investigated the real-time performance of our system and found that our system achieved 25 frames per second on average, which can satisfy the requirements for practical use.

Conclusions
In this paper, we gave the design of a vehicle panorama system.Our system leverages the techniques of camera calibration, image distortion correction and image mosaic to provide drivers with a panorama view of a car.The panorama view is able to extend the vision of drivers to 360 degrees, which provides a strong assurance for driving cars safely.In this work, we introduced design and implementation details of our system.We believe it is a good reference for the development of similar systems.

Fig. 4 .
Fig. 4.An example image captured by our system.