Robust Face Recognition via Occlusion Detection and Masking

. Sparse representation-based classification (SRC) method has demonstrated promising results in face recognition (FR). In this paper, we consider the problem of face recognition with occlusion. In sparse representation-based classification method, the reconstruction residual of test sample over the training set is usually heterogeneous with the training samples, highlighting the occlusion part in test sample. We detect the occlusion part by extracting a mask from the reconstruction residual through threshold operation. The mask will be applied in the representation-based classification framework to eliminate the impact of occlusion in FR. The method does not assume any prior knowledge about the occlusion, and extensive experiments on publicly available databases show the efficacy of the method.


Introduction
Face recognition (FR) is one of the most intensively investigated topics in biometrics.It has a wide range of applications, e.g., information security, access control, smart payment and law enforcement [1].A common face recognition system mainly consists of two modules, namely feature extraction and classifier design.In the past decades, many algorithms have been proposed for the two aspects.For feature extraction, classical dimensionality reduction methods include Principle Component Analysis (PCA) [2] and Linear Discriminative Analysis (LDA) [3].These methods aim to construct projections that can map the high-dimensional face image into a lower-dimensional space to get a compact representation of face image.The most used classifiers are Nearest Neighbor (NN) and Nearest Feature Subspace (NFS) [4].
Generally speaking, occlusion poses a significant obstacle in a robust, real-world face recognition system [5][6].This difficulty is mainly due to the error in query sample incurred by occlusion.In recent years, representation based methods for classification [6,11,14] has attracted much attention due to their powerful performance in FR.The most representative approach is sparse representation-based classification (SRC).Wright et al. [6] exploited the discriminative nature of sparse representation via l1-norm regularization.The method has shown great potential in handling various occlusions, such as real disguise, continuous occlusion, random pixel corruption, etc. SRC triggers many following works, including the l1-graph for image classification [7], kernel based SRC [8], Gabor feature based SRC [9], and robust sparse coding [10].
Meanwhile, Naseem et al. [11] proposed a linear regression classification (LRC) algorithm using the concept that patterns from a single-object class lie on a linear subspace.Luan et al. [12] indicated that error image derived from the LRC was a better choice than original image to identify occluded regions, and presented an appearance-based method for face recognition based on LRC and level set method to deal with contiguous occlusion.Sparse error of Robust PCA was also extracted for FR under varying illumination and occlusion [13].
Zhang et al. [14] indicated that it is the collaborative representation but not the l1-norm sparsity that makes SRC powerful for face classification, and present a collaborative representation based classification (CRC) method.CRC aims to represent the query sample using all the training samples with a weaker l2-norm regularization, and then perform classification according to the contribution of each class in representing the query sample.Nevertheless, the estimated coefficients of an occluded test sample may be corrupted and unstable.
In this paper, we present a novel method for robust face recognition in the presence of occlusion, such as block occlusion, pixel occlusion and real disguise.As introduced above, CRC actually performs a separation of query sample into the parts similar and dissimilar with the training set.The occlusion part in query samples is considered to be heterogeneous with the training samples, and thus can not be well reconstructed by the training samples, and is left in the reconstruction residual.We exploit the reconstruction residual to detect occlusion, and get a mask for the test sample.The mask will be used in the CRC scheme to isolate the side effect of occlusion in FR.The remainder of this paper is organized as follows: In section 2, we introduce the CRC and the proposed method.Section 3 presents the experimental results.The conclusion is drawn in Section 4.  (1) Considering that the dictionary D can be underdetermined, they use l 2 -norm to regularize the solution.The CRC aims to solve the following problem.Where X i is the representation coefficient vector in X associated with training class D i .

Proposed method
As mentioned earlier, the goal of the proposed method is to detect and eliminate the occlusion part in test sample.Firstly, we need to detect the occlusion part in test sample.
As discussed in [6,12,14], in representation based classification methods, such as LRC, the reconstruction residual was a better choice than original image to identify occluded regions.Similarity, the reconstruction residual in CRC also has such property.The occluded parts in query sample are always heterogeneous with the training samples, highlighting the occlusion part in query sample.Figure 1  Notably, we set η a relatively large value to relax the regularization on the reconstruction fidelity, which helps for the extraction of occlusion part.Then we get the reconstruction residual (y-DX) of y.The pixels whose value meet the condition in ( 5) is assigned as 1, otherwise 0.
V m y DX (5) σ in ( 5) is a threshold needed to be specified.This way, we can get a mask of the test sample.The zero entities in m indicate the outliers in test sample.Figure 2.shows an example of this case.Once the binary mask of test sample obtained, we solve the following regularized least square problem with mask.Setp3.Use Eq. ( 4) and Eq. ( 5) to estimate the pixels shaded in y, and get a mask for it.Setp4.Solve the regularized least square problem with binary mask in Eq. ( 6).
Setp5.Give the identity of the test sample y on the basis of Eq. ( 9).The essence of our method is to detect and eliminate the impact of occlusion in query sample, and use the clear part to perform classification.

Experiments
To verify the performance of our algorithm in dealing with occlusion, we use four state-of-the-art algorithms for comparison.They are the Nearest Neighbor (NN), LRC [11] and CRC [14] and SRC [6].All the Matlab implementations are on a PC with Intel core i5 3.3GHz CPU, 8 GB RAM, and Windows 7 operating system.

Recognition with random pixel occlusion
In this experiment, we test the robustness of the proposed method in the presence of pixel corruption.We use the Extended Yale B database for this experiment.The Extended Yale B database [15] consists of 2414 frontal face images of 38 individuals under various laboratory-controlled lighting conditions.Subset 1 consists of 266 images (seven images per subject) under normal lighting conditions.Subsets 2 includes 12 images per subject, characterize slight illumination variations (as shown in Figure 3).For Subset 2, we replace 10%~60% of its pixels by uniformly distributed random values within [0,255] (as shown in Figure 4).The corrupted pixels are randomly chosen for each query image and the location are unknown to the algorithm.All images are cropped and resized to 64×56.Table 1 reports the recognition results of each algorithm.From this table, one can see our method classifies all the test samples correctly even in the case of 60% of the pixels are occluded.

Recognition with contiguous occlusion
The Extended Yale B database is used in this experiment.We use Subset 1 for training and Subsets 2 for testing.Various levels of contiguous occlusion are simulated by replacing a square block of each query image with an unrelated image (as shown in Figure 5.).The location of occlusion is randomly chosen and is unknown to the algorithms.Compared to random pixel corruption, contiguous occlusion is certainly a worse type of errors.As reported in Table 2, our algorithm outperforms other competing methods in the presence of contiguous occlusion.

Recognition with real disguise occlusion
In this section, experiments are carried out using a subset from the AR database [16] consisting of 100 individuals, 50 male and 50 female.799 images of nonoccluded frontal views with various facial expressions are used for training, while two separate sets of 200 images are selected for testing [12].The first test set contains images of the subjects wearing sunglasses, which occlude roughly 25% of the face image.The second test set considered includes images of the subjects wearing a ICEICE 2016 01039-p.3scarf, which occludes roughly 40% of the face image.All images are cropped and resized to 84×60 (as shown in Figure 6).
We also present the recognition accuracy of SRC and LRC using partition scheme [6,11].Each test sample was partitioned into 8 regions.From the recognition results in Table 3, It can be seen that our algorithm has a higher recognition rate comparing with other contrastive methods.

Conclusions
In this paper, we present a novel method for robust face recognition in the presence of occlusion.We exploit the reconstruction residual of CRC model to detect the occlusion in test sample, and get a binary mask of the occlusion.Then the mask is used in the CRC scheme to isolate the side effect of occlusion in FR.Experimental results on Extended Yale B and AR database demonstrate that our algorithm performs well in dealing pixel occlusion , block occlusion and real disguise.The method does not assume any prior knowledge about the occluded regions, and can be analytically solved, and thus can be applied in a robust, real-time face recognition system.

DOI: 10
.1051/ C Owned by the authors, published by EDP Sciences, 201

2. 1
. CRC CRC was proposed by Zhang et al. [14].The work argued the function of collaborative representation in SRC [6], and empirically showed that the non-sparse l2-norm regularization could outperform the state-of-the-art SRC based on l1-norm regularization, both in computational complexity and classification accuracy.We first review the typical CRC algorithm.Assume there are C distinguished training classes with n i samples in each class.Let D = [D 1 , D 2 …D C ], where D i ∈R m×ni is the data matrix of the ith class.Each column of D i denotes a training sample of the ith class.For a test sample y∈R m , CRC uses all the samples in the training set to collaborative represent it as y DX solution of (2) can be analytically derived as X = (D T D+μ•I) -1 D T y.The class-specified representation residual ||y-D i X i || 2 together with the coefficient X i are used for classification.
shows an example for distribution of intensity values of reconstruction residual by CRC.We can find most of the values are near 0, in the interval [-0.005, 0.005].The large values in the reconstruction residual are known to be outliers in query sample that can not be well fitted by the training samples.Based on this, we propose to extract the occluded parts in query sample from the reconstruction residual of CRC model.Firstly, we use all the training samples to represent the test sample, and solve the following problem

Figure 2 (
b) is a mask extracted from the reconstruction residual of Figure 2(a), and Figure 2(c) shows the test sample with mask, from which we can see that the occlusion part is well eliminated.

Figure 1 .
Figure 1.An example for distribution of intensity values of reconstruction residual in CRC.
The closed form solution of above equation can be derived by taking partial derivations and setting them equal to is made in favor of the class with the minimum regularized residual.
the main steps of our method are as follows.Setp1.Input: matrix of training samples D R m×n , a test sample with occlusion y R m .Setp2.Normalize the columns of D and y to have unit l 2norm.

Figure 2 .
(a) A test sample with sunglass.(b) the binary mask extracted from (a).(c) test sample with mask.

Figure 3 .
Figure 3. Two Subsets of Extended Yale B database.First row illustrates images from Subset 1, and second row Subset 2.

Figure 4 .
Figure 4. Test samples under varying level of random pixel occlusion, from left 10% to right 60%.

Figure 5 .
Figure 5.Test samples under varying level of contiguous occlusion, from left 10% to right 60%.

Figure 6 .Table 3 .
The training and test samples with sunglasses and scarf in AR database.(a) is the training sample, and (b)-(c) for test.Recognition results on the AR database with real disguise

Table 1 .
Recognition results on the Extended Yale B database under varying level of random pixel occlusion using Subset 1 as training set.

Table 2 .
Recognition results on the Extended Yale B database under varying level of contiguous occlusion using Subset 1 as training set.