Applications of hand feature points detection and localization algorithms

. Growing interest in hand gesture recognition system, bring new possibilities in developing new ways of intuitive control. Controlling the everyday surrounding can be much easier with use of natural body language. This article presents the review of hand gestures systems basing on color images, focusing on hand feature points detection. Additionally, the ways of hand feature points control possibilities are discussed, along with possible examples of hand gestures control. Hand gestures are introduced with discussion about their usage in everyday equipment control.


Introduction
It can be observed that the development of new technologies tends to focus on the intuitiveness of control [1][2][3][4][5][6].Not so long ago, touch screens were the cutting edge technology, now they are commonly used in nearly all devices.In the future, the touchless technology can become the new standard of intuitive control.With growing hardware and software possibilities focusing on gesture recognition, it will not be long until gesture based control would be as common as touch panels [7].
Gestures was always the most intuitive ways of communication between humans.We use gestures every day, sometimes unconditionally.Moreover, for deaf people, gestures and sign language is the only way of communication.
This article proposes the ways of controlling with use of hand gestures, along with a review of color image based, hand feature recognizing algorithms [8].The aim of this paper is to show the present research in hand feature points detection and propose the interesting ways of usage of their results.

Hand feature points
Many literature solutions, concerning hand gesture recognition, focus on hand feature points detection.Feature points are characteristic regions of hand, that are easy to detect.The proposed hand model consists of 25 characteristic points, that guarantee the possibility of full hand reconstruction with use of those points coordinates.Human hand, as well as whole human skeleton, consist of constant length rigid bones, that are connected in joints, that in case of fingers are called knuckles [9][10][11].Moreover, human hand consists of 27 bones, with 14 phalanges of fingers, 5 metacarpal bones and 8 wrist bones.
Each finger, except thumb, consist of 3 phalanges, that can be visually observed.Those bones positions can be approximated without the use of x-ray image.Having in mind that their length is constant, it is enough to represent their position with use of two points line segment starting and ending in knuckles or finger tips.
The palm region and wrist region consists of multiple bones, that cannot be visually separated.Thus, the wrist region is represented as a line connecting two contour points, while the palm is represented as a circle or parabola, connecting wrist points and binger base points.
With addition of 4 concave points between fingers, the total of 25 feature points is a whole set of feature points that describes the hand arrangement.The visualization of hand feature points positions are presented in Fig. 1.However, some points cannot be seen in particular hand orientation, other are limited by algorithm capabilities.The missing points should be concluded Detecting and tracking hand feature points can become the easy and intuitive way of control.Human hand complexity allows to make various movements, while algorithms tracking mutual distances between feature points can bring huge possibilities in intuitive control.

Related literature
With growing interest of intuitive control, there are multiple solutions on gesture recognition, each aiming in different aspect of hand gesture.The main approaches uses the hardware sensors called data gloves, special markers that allow easy tracking, 3D depth cameras or traditional color images.A detailed review on types of gesture recognition technologies, can be found in [12].All approaches differ.It is important to choose one approach and compare the results with similar solutions.In the related work, a solution based on color images and computer vision algorithms has been chosen.Usually, the mask of hand is found with use of skin detection algorithm [13], and then, this mask is subject to various transformations.
It is important to notice that there are two main research fields in hand gesture recognition domain.Namely, the methods can be divided as appearance based methods and feature based methods.
The capabilities of the algorithms described in the literature differ from the simple rise fingers counting [14], through rise finger position evaluation [15], to full fingertip and finger base description, with detecting the palm region [16,17], or even the concave points between fingers [18].
In the previous research, described in authors other publications, an algorithm of wrist detection algorithm has been described [19].This algorithm has become the base of the hand feature points detection and localization with use of directional image, capable of accurate fingertip and finger base detection.Moreover, the literature solutions has been discussed and tested, and the mutual comparison can be found in [20].
The detection algorithm must have a purpose.The most common application is a Sign Language translator.The part of Sign Language for deaf people covers each letter of alphabet by the different hand arrangement.The aim of some solutions is to translate the gestures shown by user [14].Unfortunately, the task is difficult, and many solutions bound the number of recognizable gestures [21].The typical gesture set of a Sign Language varies in about 25-30 gestures, dependent on language.
Other common application is to use the gestures as a augmented way of intuitive control.The commonly available smart TVs and game consoles includes the possibility of simple gesture control [22,23].However, the recognition is limited to simple body tracking.Other solutions uses the hand position and gestures to replace the computer mouse or create the advanced controller [24,25].
Finally, the most of hand recognition algorithms, does not have any application at all.The solutions focus on accuracy of rise finger counting or finger identification, without giving any practical usage.

Feature points detection
There are many hand feature points extraction algorithms described in the literature, however one can divide the approaches by used image processing method.Nearly all feature detection algorithms assumes a hand mask as an algorithm input.The most common image processing methods used in hand feature points detection are: contour analysis, distance transform and template matching.The division is presented in Fig. 2.

Figure 2
The division of hand gesture recognition methods with presentation of most common algorithms and solutions.
The mask contour based algorithms uses the hand mask silhouette, that is a set of points that lie on the border of hand region and background region.Usually, the algorithms are build basing on the mutual distances or angles of contour points, wrist point and palm center.The function of distance or angle between wrist or palm point and consecutive contour points is obtained and analyzed.Usually the local maxima of this function are considered to be the finger tips.
Distance transform of the hand mask brings useful information about palm region and fingers.The position of global maximum of a distance transform is treated as a palm center and the value is treated as palm radius.The mask points that lie outside of the palm region are considered as fingers.All points that can be used in finger representation can be found by analyzing the local maxima of distance transform.
Template matching algorithms are basing on the observation that fingers in hand mask are similar to round objects.Usually a circle image with radius similar to finger width is fitted into mask image with use of cross correlation or other template matching technique.After obtaining the resulting probability map, the global maximum is searched.The position of maximum is saved and the values in vicinity of this point are set to zero, in order to find the next global maximum.The positions of maxima are treated as fingertip point.Additionally, the There are also other approaches in hand feature points detection, however the presented above are the most common.

Possible applications
Human hand is very complex, yet it is easy to perform a various and quick movements.Tracking only fingertips and their mutual distance can lead to multiple applications.It can be observed, that touch platforms such as smartphones of computer touchpad can introduce multiple, yet simple gestures that can carry various meanings, such as rotate, zoom, navigate or perform a specified macro command.Few years ago, many types of button and switch control were replaced by the touch panels.In the future, new algorithms would focus on creating the touchless technology.
In case of a computer control support, hand gestures would be the good replacement for mouse.The single finger tracking algorithms are easy to implement, leaving place for several button actions represented as gestures.Moreover, the combination of gestures and movements can cover complicated actions as drag and drop, select all, copy or paste.
Human hand is capable of preforming complex tasks.A hand feature tracking algorithm can record those tasks and reproduce them with use of specified robot manipulators.The path of each finger can be followed with high precision.This can lead to creation of a universal robot manipulator capable of human imitation.
Moreover, gestures can be used to precisely control the complicated machines.Nowadays, difficult tasks are performed with use of an operator equipped with a joystick, many times working in different reference.It would be much easier and more intuitive to operate those machines with use of simple hand movements.We can imagine a construction worker lifting a heavy load, just by moving a hand in air.
However, there exist a problem with fast movements.An operator can have a cramp, or become tired, or a vision system can detect an impulsive error.This could lead to damaging fast position changing during the complex precision tasks.In order to overcome this issue, it should be kept in mind that a security filter should be implemented in those types of solutions.The filter should detect the drastic changes, preventing the fast movement or approximating the impulsive errors.

Exemplary gestures
Most solutions focus on simple, one finger tracking (Fig. 3a).However, there are many possibilities of hand position arrangement, leading to intuitive control of many appliances.Fig. 3 shows few examples of feature tracking possibilities.Analyzing the following images, few applications are introduced: a) simple one finger tracking can be used to control nearly everything position, from mouse pointer to robot arm position; b) detecting the mutual position of two fingers can introduce the pinch gesture, widely used in touch panels to enlarge or rotate objects.Detecting two fingers in 3D can be used to control the position and clamping of a manipulator gripper; c) introducing the third finger can create the slider control, with middle finder indicating the volume value; d) grabbing gesture can be used to grab and move objects; e) performing a "C" shaped gesture, and rotating it can be used to control all types of knobs; f) performing a "L" shaped gesture, and rotating it can be used to rotate, for example pictures on computer; ) Fig. 3 Examples of gestures that uses feature tracking.
g) tracking multiple fingers is similar to one finger tracking, but an additional feature can be introduced.Basically each everyday action can be performed, as it would be used on the traditional equipment.Those are only some examples of hand positions.There are many more, depending on the actual application and need of a user.

Conclusions
It is not the question whether the gesture control will become popular.The question is "when?".Simple tasks concerning TV or game control can already be managed with use of gestures.More advanced applications are already developing.Human hand is capable of multiple position arrangement without much effort.This article discussed many possibilities of everyday equipment control with use of gestures.
Unfortunately, the precision is still the problem.In conclusion, simple tasks, such as hand tracking are commonly used in wide applications.The more advanced solutions still needs to be improved.

Fig. 1
Fig. 1 Human hand with marked bones and proposed feature points.
) multiple actions can also be introduced while pinching gesture is performed.Position of two fingers can be correlated with position of manipulator gripper.The number of rise fingers can for example indicate the speed of manipulator.
For example, tracking one finger moves mouse pointer.When two fingers are present, a selection region is drawn.When three fingers are present, drag and drop command is performed.Rising thumb can open a context window; h