Experiments on mobile robot stereo vision system calibration under hardware imperfection

Calibration is essential for any robot vision system for achieving high accuracy in deriving objects metric information. One of typical requirements for a stereo vison system in order to obtain better calibration results is to guarantee that both cameras keep the same vertical level. However, cameras may be displaced due to severe conditions of a robot operating or some other circumstances. This paper presents our experimental approach to the problem of a mobile robot stereo vision system calibration under a hardware imperfection. In our experiments, we used crawler-type mobile robot «Servosila Engineer». Stereo system cameras of the robot were displaced relative to each other, causing loss of surrounding environment information. We implemented and verified checkerboard and circle grid based calibration methods. The two methods comparison demonstrated that a circle grid based calibration should be preferred over a classical checkerboard calibration approach.


Introduction
Stereo vision is widely used in robotics for navigation, obstacle avoidance and path planning [1], object tracking, 3D scene reconstruction, simultaneous localization and mapping [2].Calibration of a stereoscopic vision system is a necessary step in 3-D computer vision in order to achieve high accuracy in deriving metric information from 2D images such as distance between objects and their dimensions [3].Calibration is used in order to estimate specific parameters of cameras, which help establishing proper correlations between 2D images and 3D world, and to determine lens distortion models.
Ideally, stereo vison system sensors should be perfectly aligned along the same horizontal line passing through the principal points (image plane center) of both images of the stereo cameras.Otherwise, misalignment could lead to the error growth in measurements [4].However, in practice, we have to deal with initially presented or appeared later due to multiple reasons hardware imperfection issues, for example, due to severe operation conditions.As in general case it is not possible to solve this issue operatively, the calibration process becomes more complicated and error-prone.
This paper presents our experimental approach to the problem of a mobile robot stereo vision system calibration under a hardware imperfection.In our experiments, we used crawler-type mobile robot «Servosila Engineer» (Fig. 1).The left camera of the stereo pair was displaced after the robot had felt from a significant height during an experiment.Our goal was to verify whether it is possible to achieve acceptable stereo system calibration results in a presence of such hardware imperfection.We implemented and verified checkerboard and circle grid based calibration methods.The two methods comparison demonstrated that a circle grid based calibration should be preferred over a classical checkerboard calibration approach.The rest of the paper is organized as follows.Section 2 presents mobile robot stereo vision system.Section 3 is dedicated to camera calibration approaches.Experiments on calibration with a checkerboard and a circle grid based methods are presented in Section 4. Finally, we conclude in Section 5.

Robot's vision-related features
Russian crawler-type mobile robot «Servosila Engineer» [5] is equipped with four cameras: -a forward-view optical zoom camera; -a wide angle of view rear-view camera; -a pair of cameras for stereo vision.
Stereoscopic vision system consists of two monocular cameras which are located on the front of the robot's head (Fig. 2).Supported pixel formats and frame resolutions of the stereo system are described in Table 1.  3 Camera calibration

Problem statement
Camera calibration is used in order to estimate extrinsic and intrinsic parameters and distortion coefficients of the camera systems (i.e., monocular or stereo) [6,7].Intrinsic parameters, which specify a camera itself, include internal optical characteristics of the camera: -a principal point (location of an image center) -a focal length (a distance between the camera lens and an image plane) -a skew coefficient -a radial and a tangential distortion coefficients of the camera lens Extrinsic, or external, parameters relate the world coordinate system to the camera centered coordinate system.It is defined by a translation vector and a rotation matrix.Both intrinsic and extrinsic parameters enable us to transform objects of 3D space into a 2D image and vice versa.This process is referred as a perspective projection and it maps each 3D point to a 2D image plane [6].
For stereoscopic vision systems, intrinsic parameters are calculated separately for both cameras.However, external parameters are estimated for an entire stereo vision system, and thus spatial relations between the cameras are determined [8].Obtained parameters are used to align images from both cameras along a same plane.This process is called stereo rectification and it is a crucial step in a case of misalignment of the cameras.

Calibration using 2D apparatus
To calculate camera parameters, a proper mapping between 3D world points and their corresponding 2D image points is required.This mapping could be obtained applying several images of a calibration pattern (e.g., a checkerboard).With regard to calibration object dimensions, approaches could be classified into four groups [3]: -self-calibration, or zero-dimensional calibration -one-dimensional object-based calibration -two-dimensional plane-based calibration -three-dimensional object-based calibration The process of calibration using a 2D plane includes tracking of a planar pattern, which is located at different orientations relatively to the camera [9], and various patterns could be used (Fig. 3).Comparing to other calibration approaches, calibration with 2D apparatus is easier to implement as 2D calibration pattern is easy to create.For example, selfcalibration techniques are generally less accurate and require solving more complicated mathematical problem resulting in a more difficult calibration process.In contrast with 3D object-based calibration (Fig. 4), which uses up to three orthogonal to each other planes, 2D object-based calibration does not require an elaborate setup and has a relatively good calibration quality.

Pinhole camera model
Most calibration methods are based on a pinhole camera model (Fig. 5) and deal with radial distortion model.The pinhole camera parameters are represented in a matrix that is referred as camera matrix.In Figure 5 an optical center of the camera � denotes the origin of the camera's coordinate system and the axes of this coordinate system define an image plane.Position of a 3D point M and the coordinates of its corresponding projection to the image plane m, representing a 2D point, are denoted as � � ��, �� � . ( The ray, which connects point � and the camera's optical center O, intersects the image plane at point �.
The corresponding augmented vectors of 2D and 3D points are denoted respectively as Intrinsic, or internal, parameters of the camera are defined as the following 3-by-3 matrix (intrinsic matrix): where � � and � � denote focal length in � and � axes directions respectively, �� � , � � � are the coordinates of the center of the image plane, and � is a skew coefficient.
If the image plane axes are orthogonal to each other, then the skew coefficient is equal to zero.Extrinsic, or external, parameters of the camera represent a transformation from a 3D world coordinate system to a 3D coordinate system of the camera.It includes of rotation and translation: where � is a rotation matrix, � is a translation vector.A 3-by-4 matrix � , consisting of both extrinsic and intrinsic parameters (Eq.5, 6), is called camera projection matrix, or camera matrix, and it could be defined as follows: Using Eq. 3, 4, 7 it is possible to define the relationship between 3D point � and its image projection �: where � is a arbitrary scale factor.
This way, 3D world points are transformed into 3D camera model coordinates using the external parameters.Given 3D points camera coordinates are projected into the image plane using the internal camera parameters (Eq.5).To estimate the camera parameters, 3D world points and their corresponding 2D image points are used.In order to get those correspondences (samples), multiple images of a calibration pattern could be used, such as a checkerboard or circle grid (Fig. 3).Thus, the camera parameters can be determined.

Calibration experiments 4.1 Hardware imperfection issues
In our experimental work the goal was to calibrate the stereo vision system of Russian crawler-type mobile robot «Servosila Engineer» (Fig. 2).Even though initially the stereo pair was mounted correctly by the manufacturer, the left camera was displaced from its original position as a result of the robot turning upside down and falling from a significant height during an experiment.Figures 6-7 demonstrate that the images from the left and from the right cameras are displaced relative to each other (the left camera's frame is higher), so their principal points are not on the same horizontal line; no cropping or any other image processing were applied to the original images.Thus, approximately 20% of visual information will be discarded during the rectification procedure.In addition, barrel lens distortion affects the images.

Calibration with a checkerboard
The first calibration method that we used in our experiments was based on checkerboard calibration pattern approach [13].This method requires the dimensions of the pattern to be known and to be constant during the calibration process.The algorithm detects chessboard corners (as an intersection of two black squares) from left and right images being captured by the left and the right cameras of the stereo vision system respectively.In order to estimate camera system parameters, images are obtained at different orientations of the calibration pattern.
Calibration was conducted using robot operating system (ROS).To capture video stream from the stereo system of the robot we employed our own ROS package for simultaneous video streaming from several different cameras [14].Stereo camera calibration ROS package was used for calibration purposes [15].It required a checkerboard template with known dimensions (number of squares and their size in meters).It should be noted that the pattern must be hold horizontally and the number of its columns should exceed the number of rows.When the calibration process successfully terminates, the camera system parameters are displayed and applied.
Four different checkerboard patterns were tested in order to obtain an optimal solution (Table 2).Calibration pattern with dimensions of 8 by 6 and a square side of 26 mm showed acceptable results (Fig. 8, 9).The process of calibration took approximately 5 minutes on our machine.In an attempt to achieve more accurate calibration results, we tried increasing the number of checkerboard columns by one column.However, the pattern of 9 by 6 with cells demonstrated almost the same results (Fig. 10).In contrast to the calibration pattern with a cell size of 26 mm, a checkerboard with a cell size of 46 mm and dimensions of 8 columns by 6 rows demonstrated poor calibration quality results, producing an epipolar error that exceeded one pixel (Fig. 11).Some of the experimental trials did not converge at all and the obtained distortion coefficients were zero.In the next experiment, we used a checkerboard with 46 mm cells and 11 columns by 7 rows and successfully estimated distortion (Fig. 12).The obtained epipolar error was greater than the one for the 26 mm cell size pattern (0.48 vs 0.33) and camera parameters estimation time also increased with the size of the pattern.When the pattern size was increased (number of rows and columns), it become harder to accomplish the calibration process.Nevertheless, after successful calibration, images had less common pixels due to cameras displacement (Fig. 13) and some information about the surrounding environment was lost.

Calibration with a circle grid
Circle grid based calibration is another option of camera calibration [16], which could be more accurate than classical checkerboard based approaches.We experimented with robot stereo vision system calibration applying a 6 by 6 circle grid pattern with a circle diameter of 23 mm (Fig. 14).ROS stereo camera calibration package was used.Calibration resulted in estimated distortion coefficients and rectified images (Fig. 15).The obtained epipolar error was 0.18 pixels, which is almost two times better than the result of the best checkerboard based calibration.

Monocular calibration experiments
In addition to main set of stereo vision system calibration experiments, we attempted to calibrate only one of the robot cameras and to apply the estimated distortion directly on the other camera.For calibration we selected the right camera as it had not been displaced from its original location.Based on our previous conclusion on a better calibration approach, the circle grid was used.As a result, the error remained, approximately, the same, as if stereo calibration with circle grid pattern was used (Fig. 16).

Conclusions
In this paper, we reviewed and experimentally compared two popular stereo vision system calibration approaches, the checkerboard and the circle grid based approaches.Moreover, the experiments were performed for a mobile robot stereo vision system calibration under a hardware imperfection.We used a crawler-type mobile robot «Servosila Engineer».Stereo system cameras of the robot were displaced relative to each other, causing loss of surrounding environment information.We implemented and verified checkerboard and circle grid based calibration methods.The two methods comparison demonstrated that a circle grid based calibration should be preferred over a classical checkerboard calibration approach.Although fairly good calibration results were achieved, the rectification process discarded almost 20% of the surrounding environment information, which decreases robot's stereo vision perception information.
As a part of our future work, we plan to implement calibration methods, which are based on fiducial markers.These approaches allow camera calibration in the presence of marker occlusions and rotations [17] and such features are especially important for autonomous self-calibration of a robot in real world operation conditions.

Fig. 6 .
Fig. 6.An image from the right camera of «Servosila Engineer» robot stereo vision system.

Fig. 7 .
Fig. 7.An image from the left camera of «Servosila Engineer» robot stereo vision system.

Fig. 13 .
Fig. 13.Loss of information due to left camera displacement.

Fig. 16 .
Fig. 16.Calibration of the right camera of the stereo vision system (with a circle grid).