Effects of rotation and systematic occlusion on fiducial marker recognition

. Fiducial marker systems consist of patterns that are placed in environment for miscellaneous applications and are further automatically detected with cameras. A variety of applications determines the criteria, which characterize qualitative properties of a marker and include such evaluation benchmarks as resilience to occlusion, distance to a marker, false positive and false negative rates, sensitivity to illumination, and others. The paper compares existing ARTag, AprilTag, and CALTag systems utilizing a high fidelity camera, which is a main vision sensor of a full-size Russian humanoid robot AR-601M. In experiments the comparison of the three marker systems reliability and detection rate in occlusions of various types and intensities was verified. Finally, a preferable for AR-601M robot visual applications marker system was selected.


Introduction
Coordinate marks are important elements of camera tracking systems that are used in robotics and augmented reality applications to stabilize camera field of view.In robotics, it allows to calibrate a camera and mechanical parts of a robotic system, which are required for industrial applications [1], human-robot interaction and humanoids [2], SLAM [3], rescue robotics [4] and other fields.Many marker design variations exist, e.g., a marker could have a simple form, such as a pattern of dots, or a compound form, such as a barcode of a rectangular shape.
ARToolKit marker system is one of the first popular fiducial marker systems that uses a square shape with a black outline boundary and a unique image inside.During operation, outer black square edges enable localization of a potential fiducial marker and then the potential marker interior allows identification utilizing a set of expected markers.While using square markers, four corners of a located fiducial marker enable unambiguous evaluation of fiducial position and orientation relatively to a camera that is being calibrated.
This paper focuses on comparing ARTag, AprilTag, and CALTag marker systems in a presence of occlusion.We test their reliability and detection rate in the presence of occlusions of various types and intensities using humanoid robot AR-601M mono camera acA640-90gc by Basler AG.In particular, these preliminary experiments are important prior to verification of real occlusions within AR-601M robot field of vision, which often occur due to self-collisions and self-occlusions by its manipulators.The experimental analysis helped us to select a best-fit option among the three marker systems.

Fiducial marker systems
Most markers, also referred as tags, have a common design feature -an outlining square shape frame with a pattern image inside, which encodes information.The square shape is popular due to four special points (that correspond to the square corners) detection that allow camera calibration and marker position calculation.To reduce the sensitivity to illumination configuration, planar marker systems use monochrome markers, i.e., a bitonal image [5].Many markers use some of bits to encode redundant information that allows identifying and correcting errors during detection and decoding process.
This section introduces ARTag, AprilTag and CALTag marker systems.These markers have two stages of detection: unique features detection and identification (or recognition) stage [6].First, the environment image is searched for the unique feature (quadrilateral shapes).Then, at identification stage, the selected image interior is verified in order to determine if it is marker or some other general object of the environment.
Different tag design, detection and recognition algorithms determine strengths and weaknesses of tags in various situations.There are several criteria for evaluating tag performance: resilience to occlusion (full or partial overlapping of tag by another object), resistance to illumination changes, minimal size of marker (distance to marker), false positive effect, false negative effect and others [7]. Figure 1 demonstrates examples of ARTag, AprilTag and CALTag.A large number of other fiducial marker systems exist, and some of them were briefly introduced in [8][9].

ARTag marker system
ARToolKit fiducial marker system, which had a first release in 1999, is a classical open-source visual tracking library for calculating orientation and position of a camera relatively to markers in real time [10]; this system is popular for projecting virtual objects onto a real world environment.ARTag is an inspired by ARToolKit alternative [11], which uses more complicated data processing in order to increase reliability and resistance to illumination changes.ARTag is supposed to replace ARToolKit due to improved 2Dbased markers as it succeeds avoiding a known false positive effect and inter-marker confusion.The ARTag system uses the same concept of squares with an internal image inside, but unlike ARToolKit, the system uses a digital approach for an internal pattern reading (i.e., a binary code).
ARTag system contains 2002 individual tags of square shape: 1001 tag has a white frame and 1001 tag has a black frame with an image (barcode) inside.Marker system detects tags with edge points based approach: an edge point detector finds edge pixels that form segments, which are then grouped into quadrilaterals.A tag internal image forms a 6x6 cell grid, which is composed of black and white cells, each representing 36 bit-values of "1" and "0".The first 10 of 36 bits form marker ID, the remaining 26 bits are redundant and are used to detect and correct errors, and to insure the uniqueness of the four possible marker orientations [7].ARTag system ensures fast marker identification as it does not require matching its internal tag image with a library of stored templates, as it was previously implemented in ARToolKit.

AprilTag marker system
AprilTag is applicable for a wide range of tasks, including camera calibration, robotics and augmented reality.AprilTag detection system calculates exact position, orientation and identity of a marker relatively to a camera [12].Representing a two-dimensional barcode, these tags look similar to QR code.However, AprilTag is designed to encode less data (4 to 36 bits), which allows quick and accurate detection of the tag.
The detection process consists of several stages: searching for linear segments, detecting squares, calculating the position and orientation of the tag, decoding the barcode.Directed linear segments search uses a similar to ARTag approach, and then sequences of segments are processed to form a square.Square detection applies a recursive 4-level depth search and at each level, a search tree adds one side of the square.At the identification stage, the validity of the barcode inside the discovered tag is verified.To encode an internal picture, AprilTag uses a lexicode system characterized by two parameters: a number of codeword (internal pattern) bits n and minimal Hamming distance between any two codes d.Lexicode generates codes for tags, which allow detecting and fixing bits' errors [13].AprilTag has several tag families that differ in these two parameters, n and d.For example, "Tag36h11" refers to a 36-bit tag (6x6 array) with n = 36 and d = 11, and "Tag16h5" refers to a 16-bit tag (4x4 array) with n = 16 and d = 5.

CALTag marker system
After analysis of classical chessboard-based camera calibration and fiducial markers approach, CALTag marker was proposed as a specially designed for camera calibration solution [14].This system consists of two components: a marker and a detection algorithm.A calibration marker is used in a proposed calibration grid, which is externally identical to a chessboard.The tags layout within this grid has two variations on tags density.A grid with a highest markers density provides a larger number of calibration points and is thus more reliable and efficient for recognition.
Each marker consists of M x N matrix of black and white squares, which are encapsulated within a boundary that consists of strictly black or white pixels.After initial detection of potential markers, they are filtered and verified by accessing their binary codes.Any missed calibration points of the template are restored as the chessboard layout is known by CALTag system.The binary tag code is validated by calculating the first P bits checksum and comparing it with a checksum that is obtained from four possible marker positions.

Experimental design
We define occlusion as a partial overlapping of a marker with other objects.Experimental work was performed in order to compare ARTag, AprilTag and CALTag markers with regard to occlusion resilience.Image capturing during experiments was performed with AR-601M humanoid robot (Fig. 2), that uses Basler acA640-90gc camera as front facing camera (Fig. 3).
For each marker we provided identical conditions of room illumination and camera posture with respect to the tag.To analyze the effect of occlusion on tag recognition, we selected four samples of ARTag tags, four AprilTag tags and two CALTag tags.ARTag and AprilTag marker IDs were selected randomly (Fig. 4) with a unique ID being encoded into an internal pattern of a tag.For CALTag markers we selected 4x4 and 9x6 grid size.
ARTag ID was encoded in 10 bits of 36 bits and tags with ID 2, 3, 6, 34 were selected.All AprilTags were selected from "36h11" tags family, i.e., each of tag ID is encoded in a n = 36 bit codeword with d = 11 bits; we have arbitrarily selected tags with IDs 4, 6, 8, and 9 (Fig. 4).Four types of experiments were conducted: systematic type occlusion and random type occlusion (arbitrary overlap with an object), rotation with regard to 3 axes and a combination of rotations with systematic occlusion.

Type 1 -Marker rotation
For the experiments, we set up rotation axes with regard to the marker plane (Fig. 5).X-axis is orthogonal to the marker plane and points into the camera direction; it was responsible for roll rotations of the marker in Type 1C experiments.Y-axis (horizontal) and Z-axis (vertical) are in the marker plane.Y-axis was responsible for pitch rotations of a marker in Type 1B experiments, Z-axis was responsible for yaw rotations of a marker in Type 1A experiments.
In Type 1A experiments, each marker was fixed at two points that formed a vertical rotation axis Z being drawn from a bottom to a top of a marker and passed through the marker center.The rotations were performed clockwise and counter-clockwise for 10, 20, 30, 45, 55 and 65 degrees in both directions around Z-axis (Fig. 6 demonstrates Type 1A experiment with ARTag marker).In Type 1B experiments, for each marker the rotations were performed clockwise and counter-clockwise around Y-axis for 10, 20, 30, 45, 55 and 65 degrees in both directions (Figure 7 demonstrates an experiment with AprilTag marker).In Type 1C, each marker was fixed at its central bottom point and the rotation was performed clockwise and counter-clockwise around X-axis for 0, 22.5, 45, 67.5, and 90 degrees in both directions.

Type 2 -Systematic occlusion
Each tag was covered with a white paper rectangular template starting from the bottom to the top of tag so that the template was occluding K% of the marker's area, where K are 0%, 10%, 20%, 50%, 70% of the marker's area.

Type 3 -Marker rotation with systematic occlusion
In marker rotation with systematic occlusion experiments Type 1C and Type 2 experiments were combined together.The marker was rotated clockwise and counter-clockwise around X-axis in its plane for 0, 22.5, 45, 67.5, and 90 degrees and simultaneously occluded with the white paper rectangular template for 0%, 10%, 20%, 50%, and 70% of its area.Figure 8 demonstrates an example of rotated CALTag 4x4 marker by 22.5, 67.5, and 90 degrees in both directions and occluded by 20%.

Type 4 -Arbitrary overlap with an object
Each tag was randomly overlapped with one of three different objects so that an object was entirely located within tag's area and thus the overlap percentage was always kept constant.The first object was a white thick paper strip of 13 cm width and 2.5 cm length with 32.5 cm area.The second object was a metal scissors with 7.99 cm area.The third object was a black plastic strip of 15,7 cm width and 2.6 cm length with 40.82 cm area.This way, for each experiment the constant overlap percentage was always known in advance.In the black strip case, it covers the interior and crosses the boundaries of each tag (Fig. 9).In the white strip case and scissors, they cover only the internal pattern of each tags (Fig. 10).With each of the three object for each tag, 25 experimental trials were conducted.We emphasize a special case for the 4x4 size CALTag: if the strip is placed strictly along the marker side, the occupied area percentage decreases as strip's width exceeds marker size.In this case, the overlap percentage varies between 25.5% and 33.84% in the white strip case and 26.53% and 40.20% in the black strip case.

Experimental results
For the experiments, we used publically available official software of AprilTag and CALTag.For ARTag we used ArUco library, which allows detecting and recognizing various kinds of tag families [15].These (tags were printed on white paper with the following sizes: ARTag: 15.2 x 15.2 cm, total area 231.04 cm 2 ; AprilTag: 13.5 x 13.5 cm, total area 182.25 cm 2 ; CALTag 4x4: 9.8 x 9.8 cm, total area 96.04 cm 2 ; CALTag 9x6: 21.7 x 14.7 cm, total area 318.99 cm 2 .It is important to emphasize that different types of occlusion had slightly different experimental implementation.Systematic occlusions (Types 2 and 3) reflect a very typical real world occlusion situation when an object overlaps both an interior and marker's boundaries, and also becomes partially invisible due to its rotations and inclination in 3D space.Moreover, since tag boundaries serve as a unique tag feature that is necessary for its detection, these types of occlusion affect both tag detection and recognition stages.Type 4 occlusion has an overlapping object completely within internal pattern of the tag, which has effect only on the recognition stage of tag pattern detection.The experimental results are summarized in Tables 1-7.Table 1-4 present the results of systematic occlusion experiments.All tags were completely resistant to rotations about vertical Z-axis in Type 1A experiments (Fig. 6) and rotations about horizontal Y-axis in Type 1B experiments (Fig. 7).The markers were detected and recognized at any given angle for both rotation axes, and thus because of the paper space limitations we excluded the tables, which showed these results.Tables 1 and 3 demonstrate results of marker rotation around X-axis with a simultaneous systematic occlusion, that correspond to Type 3 experiments (Figure 8 shows experiments of rotations together with 20% of marker area occlusion).ARTag and AprilTag demonstrated identical results, and their success rates are summarized in Table 1.In Tables 1 and 3, ''2/2'' denotes a successful detection of the marker under rotation about an axis in both clockwise and counter-clockwise directions; ''1/2'' corresponds to successfully detection only for one of rotation directions, while ''0/2'' marks unsuccessfully detection for both clockwise and counter-clockwise directions.4 shows success rate (percentage) with regard to the marker rotation degree around X-axis for all markers.
Table 5 demonstrates the results of arbitrary overlap with an object Type 4 experiments for the black strip object.As the tags have different sizes while the strip size is constant, the percentage of occluded tag area differs between the tags.The strip was arbitrarily placed within an internal part of the tag.For each tag, twentyfive experiments were performed so that the position of the strip on the tag was different in each experiment.The black colour of the strip makes it difficult to read binary code of the tag, because all of tags are monochrome, while the strip crosses the tags boundaries.A special case was CALTag 4x4 tag, because due to its small size the overlapped by the strip area varied from 25.5% to 33.84%.ARTag and AprilTag demonstrated particular sensitivity to tag edges overlapping with a black strip.Edge overlapping disables tag unique feature (edge) detection, which in turn results into a failure of tag discovery stage.Yet, CALTag 4x4 and 9x6 markers were successfully detected in all 50 experiments with black strip.
Table 6 shows the result of arbitrary overlap with an object Type 4 experiments for the scissors object.These experiments demonstrated the dependence of tag recognition algorithm on overlap of a marker interior only with a complex object.Again, CALTag 9x6 and 4x4 were successfully detected and recognized in all 50 experiments.At the same time, ARTag and AprilTag turned to be sensitive to the interior overlapping within a complex object.While scissors took only 4.38 (for AprilTag) and 3.45 (for ARTag) percent of the tag area, its complex shape significantly affected the recognition.

Conclusions and Future Work
This paper is focused on ARTag, AprilTag and CALTag marker systems evaluation for systematic and random types of occlusion sensitivity as well as verification of their resistance to rotations.Four types of experiments were performed: systematic occlusion, random occlusion, rotation about all three principal axes of the reference frame and a combination of systematic occlusion with a marker rotation around its centre in marker's plane (i.e., orientation in 2D plane).Systematic occlusion considered partial occlusion with a white paper template for 0, 10, 20, 50, 70 percent of marker area.While CALTag was successfully detected up to 70% of occlusion, ARTag and AprilTag markers demonstrated high sensitivity to edge occlusions, which limits their effective use.In random occlusion experiments, two occluding objects were used: a black plastic strip, a metal scissors and a white paper strip.The first object crossed the boundaries of markers and overlapped interior, while the second and the third objects overlapped only marker's interior and thus affected only the recognition stage of marker pattern detection.For random occlusion, ARTag and AprilTag showed good results for the second object, poor for the third and failed all experiments with the first object.CALTag showed high resilience to all objects in random occlusion type experiments.All of tags show high results of normal and lateral rotation.Yet, only CALTag markers succeeded to resist a longitudinal axis rotation, both without occlusion and with systematic occlusion of up to 70% of marker area.To summarize, we have selected CALTag marker system as a preferred marker system for our applications.
For our future work, we plan to conduct more occlusion resilience experiments, including rotations around all three principal axes, and to increase number of experiments.We plan to verify marker resistance with regard to other criteria such as resistance for illumination conditions changes, inter-marker confusion.Special attention will be paid to the behaviour of CALTag marker, which have demonstrated the best performance in our current empirical research.

Fig. 5
Fig. 5 Types of marker rotation with regard to the camera and the marker plane.

Fig. 10 .
Fig. 10.Arbitrary overlapping of two AprilTag markers with a white strip (top set of images) and metal scissors (bottom set of images).

Table 1 .
Systematic approach: occlusion and rotation around X-axis of AprilTag and ARTag.

Table 2 .
Successful detection rate with regard to the marker occlusion percentage.

Table 4 .
Successful detection rate with regard to the marker rotation degree around X-axis.

Table 2
summarizes results of successful detection rate with regard to the marker systematic occlusion percentage in experiments of Type 2 for all markers.Table

Table 5 .
Results of arbitrary overlap experiments with a black strip object.

Table 6 .
Results of arbitrary overlap experiments with a scissors object.

Table 7
shows the results of arbitrary overlap in Type 4 experiments for the white strip object.CALTag 9x6 and 4x4 showed impressive results: within 50 Type 4 experiments, the detection algorithm had failed only once.

Table 7 .
Results of arbitrary overlap experiments with a white strip object.