Incremental Learning Algorithm of Least Square Twin KSVC

In view of the batch implementations of standard support vector machine must be retrained from scratch every time when the training set is incremental modified, an incremental learning algorithm based on least squares twin multi-class classification support vector machine (ILST-KSVC) is proposed by solving two inverse matrix. The method will be applied on online environment to update initial data, which avoided cumbersome double counting. ILST-KSVC inherited the advantages of the basic algorithm and has some merits of Least square twin support vector machine for excellent performance on training speed and support vector classification regression for K-class’s well classification accuracy. The result will be confirmed no matter in low dimension or in high dimension in UCI datasets.


Introduction
Support vector machine is a type of machine learning algorithm, which aims at finding one optimal hyperplane to make two kinds of training data as far as possible away from the hyperplane in the high dimensional space [1].It has great advantages of solving small sample, nonlinear and high dimensional pattern recognition problem, therefore, its been successfully applied to many aspects, such as, medical image detection [2], network information classification [3,4], pattern recognition [5], etc. Classical SVM needs to solving a Quadratic Programming Problem (QPP), which computational complexity is O(l3)(l refers to the total number of training data), so when dealing with large-scale data is restricted.Twin support vector machines (TSVM) been proposed by Jayadeva, etc in 2007.It divides one QPP into two smaller QPPs from SVM.By solving two smaller QPPs to construct two nonparallel hyperplanes, and makes each hyperplane as far as possible close to the kind of sample data and away from the other.TSVM simplifies the computational complexity of SVM and theory has been shown that its computational complexity is O(l3/4), about a quarter of the SVM [6].Least squares twin support vector machine (LS-TSVM) [7] greatly simplify the computational complexity of TSVM.It constructs a squared loss function modifying with convex linear system to replace the TSVM convex QPPs and substitutes equality constraints for inequality constraints.Thus, solving two equations improves the solving efficiency and accuracy of classification.
Classical SVM and TSVM are designed for binary classification problems; however, we often encounter many classification problems in real life.We usually use '1-versus-1' [8] and '1-versus-1' [9] to dealing with muticlass classification problems in these methods.For K-class, the first method is to construct K two classifiers, each class corresponding to one of them been separated from other classes, the method is faster but easy to cause the imbalance of classifier.The second method construct all the possible binary classifiers in K class of training samples , so need to construct K(K-1)/2 binary classifiers.Each class training data only taken from the corresponding two classes, without considering other sample information, which leading to classification's result is not ideal.2003,Angulo proposed the '1-versus-1versus-rest 'multi-class classification algorithm---support vector classification regression for K-class classification(K-SVCR) [10].It constructs K(K-1)/2 K-SVCR triple classifier to deal with K class classification problem, this structure improve accuracy of muti-class problems, but the computational complexity is high.Twin multi-class classification support vector machine (Twin-KSVC) has been come up with, based on TSVM and K-SVCR [11].It takes advantages of TSVM and SVCR that running time is close to TSVM but far less than K-SVCR.In order to reduce the computational complexity and simplified the structure of classifier further, Jalal a. etc. combined with the LS-TSVM and K-SVCR put forward Least squares twin multi-class classification support vector machine (LST-KSVC) [12].LST-KSVCR inherited the faster training speed of LS-TSVM and high accuracy of KSVC.
In many online scenario, the new data will be continuously added to the data set, while the standard SVM, TSVM will no longer be applicable to the classification of the real-time data.In order to solve the classification problem of the real-time data, many incremental learning algorithm based on SVM and TSVM [13]- [16] has been proposed.In this paper, we propose an incremental learning algorithm suitable for an online environment called Incremental Learning Algorithm for LST-KSVC; we also call its ILST-KSVC, briefly.The Algorithm introduces SMW (Sherman -Morrison -Woodbury) formula to solve the inverse of the matrix, and every time a new sample data increases, the only need to update inverse matrix on the basis of original weight matrix, without calculating weight matrix inverse matrix, the process will not have an impact on the result of the classification.Decremental is the inverse process of incremental, therefore, the algorithm also applies to the decremental process.
2 Least square twin K-SVCR is the other class training data labeled '0'.l is the total training data and l=l 1 +l 2 +l 3 .
For K-class classification, the method chooses each i, j( j i z ) as left class and right class, the rest K-2 classes as class 0, called 'one-versus-one-versus-rest'.Training each i, j class to get K (K -1) / 2 ternary classifier, corresponding to get K (K -1) / 2 the output of the decision function.In order to determine the category of the test sample, using the voting as follows: when the decision function to output 1,cast class i one vote; When the decision function output of -1, cast class j one vote; When the decision function output is 0, do not vote.Inspection after all the K (K -1) / 2 the output of the decision function, put the test sample into the category with the most votes

linear LST-KSVC
Twin-KSVC needs to address two QPPs, which costs high computational.And LST-KSVCR reduced computation cost by solving two reversible matrix, make it can be used for processing data of high dimension matrices.LST-KSVC change Twin-KSVC's inequality constraints to the linear equality constraint conditions, thus, the original optimization problem as follows.Where, c 1 , c 2 , c 3 , c 4 are penalty parameter y>0, z >0 is slack variable, H is between 0 and 1. w, b is weights and bias for solving.Substituting the equality constraint into the objective function in formula (1), we obtain Setting the gradient of (3) with respect to w 1 , b 1 to zeros gives In the same way, we get )) By formula (5) and formula (6), we can obtain two linear separable non parallel hyperplanes.0

°°®
For K class classification problems, choose section 2.1 of the classified voting method.

non-linear LST-KSVC
By introducing a gaussian kernel function, linear LST -K SVC can be extended to non-linear.The original optimiza tion problem of non-linear LST-KSVC as follows.is weights and bias for solving.Similarly in the solution process of linear LST-KSVC, the solution of formula (9) and formula (10) are as follows.
))  (14) For K class classification problems, also choose section 2.1 of the classified voting method.

Incremental learning algorithm of LST-KSVC (ILST-KSVC)
In this section, we extend the LS-TSVCR to incremental learning and decremental learning version from linear to non-linear as well as LS-TSVCR.First, we need to simplify some symbols for convenience.Let 1 w replace 6), so 1 ŵ is the update of w 1 , and 2 ŵ is the update of w 2 .

linear ILST-KSVC
In this paper, we adopt a kind of split matrix inverse, and reorganization of the inverse of the matrix.Finally obtain incremental learning algorithm which has a simple structure and small amount of calculation.Adding new positive training data S i =[x i T , 1] T , for example.Then the matrix E can be extended to According to formula (5) and formula (6), we get )) 1 ( ( For solving formula (16) and formula (17), we need to solve two inverse matrices by introducing the SMW theorem. Let

D S
. According to formula (11) and formula (12), in nonlinear LST-KSVC, we can easily get the new weight and bias after updating, like formula ( 24) -(25). )) Now we found two ) 1 ( ) 1 ( u l l dimension invertible matrices in the above formula.In order to reduce the computation cost, we use SMW method theorem to solve the inverse matrix. Let

S I I [
.Then the formula (25) and formula (26 Thus, we just need to solve the 1 L and 1 R to get these inverse matrixes updating matrixes.3

ICCAE 2016
The solving process of 1 L is as follows.Let Similarly to 1 L , 1 R can be expressed as

Decremental learning of LST-KSVC
When constantly add new data to a certain degree, storage space will be occupied larger.Besides the original data is no longer useful, so need to delete the data to reduce the storage space.Decremental learning algorithm delete data without affecting the original classification results and is the inverse process of incremental learning.In this section, we imitate the incremental learning algorithm to conduct decremental learning algorithm.In the case of nonlinear, removing a class , for example.Similarly to formula (28)-( 29), deremenrtal expression is Thus, we only need to recalculate 1 I and 2 I , and make use of L -1 and R -1 , which has been required in the incremental learning process to gain the updating result.

Experiment and result
The validity of this algorithm has been tested in two experiments.All experiments are based on a 2.5 GHz pentium dual-core processor, 2GB memory hardware platform and using MATLAB 2013b to write programs.Experiment 1: selecting 10 group on UCI datasets, which is widely used for classification as showed in Table 1.Noting the dimensions of the car dataset is very high.We will leave it in the second experiments.Article in reference [12] compared LST -KSVC, K -SVCR, TSVM, Twin -KSVC and one-to-rest TSVM algorithms, the results show that the LST -KSVC has a great advantage in classification accuracy and the algorithm running time.Therefore, the first experiment mainly compares for ILST -KSVC and LST -KSVC training time and recognition accuracy.
We  Experiment 2: the increment algorithm of binary classification time-consuming is very big when dealing with high-dimensional data, this paper puts forward the incremental algorithm based on ternary classification problem to do with thousands of dimensional data, which only need a short period of time.This Experiment will increase the dimension from low to high gradually on a data named car which dimension is 1728 * 6.Figure 1 and Figure 2 compare LST KSVC, which apply to offline environment for ILST KSVC that suitable for the online environment in classification accuracy and running time, respectively.From figure 1 we can see: with increasing of data dimension, the accuracy of both methods is reduced, but always stay above 90%, and has little difference.Figure 2 is the contrast of these two algorithms, it can be seen that with the continuous improvement of data dimension, training time also increases, but still at a low level.Thus, the incremental algorithm proposed in this paper can be employed for a number of high-dimensional data processing without affecting the classification results.

Ending
This paper proposed an incremental learning algorithm (ILST -KSVC) suitable for the online environment by solving inverse matrixs based on least squares twin multiclass classification support vector machine (LST -KSVC).ILST -KSVC use historical data to update training data avoiding the repeated calculation, and reducing computational complexity.Selecting 10 groups of UCI data sets, based on MATLAB simulation platform for the two groups of experiments, the experimental results show that the ILST -KSCV inherits the advantages of LST -KSVC that has faster training speed and a high classification recognition rate when dealing with multiple data not only suitable for low dimensional data processing, also can be applied to the processing of high-dimensional data.

Figure 2 .
Figure 2. Contrast on training time.

Table 2
shows the LST -KSVC and ILST -KSVC classification results.Analysis of Table2, on the left side of the ternary classification data, ILST KSVC can always get with LST -KSVC compare to the classification accuracy and didn't take too much running time.Table2on the right is the training data, which category number is more than 3.In the same way, it's easy to observe and obtain the same conclusion as the left side.Experiment show that ILST KSVC recognition accuracy can match with LST -KSVC, and training time is not that long for multiple classification.