Using AdaBoost-based Multiple Functional Neural Fuzzy Classifiers Fusion for Classification Applications

In this study, two intelligent classifiers, the AdaBoost-based incremental functional neural fuzzy classifier (AIFNFC) and the AdaBoost-based fixed functional neural fuzzy classifier (AFFNFC), are proposed for solving the classification problems. The AIFNFC approach will increase the amount of functional neural fuzzy classifiers based on the corresponding error during the training phase; while the AFNFC approach is equipped with a fixed amount of functional neural fuzzy classifiers. Then, the weights of AdaBoost procedure are assigned for classifiers. The proposed methods are applied to different classification benchmarks. Results of this study demonstrate the effectiveness of the proposed AIFNFC and AFFNFC methods.


Introduction
The object classification methods, as stated in [1]- [2], are to divide the initially undefined entities into classes. In each class the individuals are close to one another in some sense. These methods can be used to determine the particular symptoms of the diseases in medical science, to classify the species in biology, and to solve common classification problems in engineering and other fields. Most traditional statistical classification approaches, such as discrimination analysis, are built on the Bayesian decision theory [3], and each method generally has its own explicit probability model, but it works well and performs effectively only when the underlying assumptions are correct. For measurement of the similarity, twelve similarity structures were listed in the work of [4]. In supervised machine learning for classification, about 126 relative works are summarized in recent literature [5], which works are based on logical/symbolic techniques, perceptron-based techniques, and Statistics.
The boosting approaches are meta-algorithms for supervised learning, which are originally proposed to improve the accuracy of classification and prediction problem. Derived from machine learning, Valiant [6] presented the probably approximately correct (PAC) architecture and defined the notation of learnability. Furthermore, Kearns and Valiant [7] introduced the idea of weak learnability, and claimed the minimum performance of an algorithm should be better than a random guess. The above guess was proved by Schapire [8]. Schapire stated that the original classification results of dealing with the 2-class classification problem can be improved if a weak classifier could be divided into two new classifiers via a specific process of data filtering and grouping. Such approach was named as Boosting by Schapire. Later, Freund [9] developed a more efficient approach, but it suffered from some practical drawback. Afterwards, Freund and Schapire [10] proposed the AdaBoost algorithm and claimed the method can reduce the learning error and continuously decrease the testing error without over-fitting phenomenon. The traditional AdaBoost approach is hard to deal with the multi-class problems, and its performance highly depends on the design of weak classifiers. Consequently, the general solution for multi-class problem is to deploy hundreds or thousands weak classifiers for the traditional AdaBoost method.
Vast researches in neural classification have shown that neural networks [11][12] are promising alternatives to conventional classification methods. However, the meaning of each neuron and the decision of each weight are difficult to understand in the neural networks. Fuzzy entropy measure [13] is employed to partition the input feature space into decision regions and to select relevant features with good separability for the classification task. Neural fuzzy networks [14] integrate the advantages of both neural networks and fuzzy systems, which bring the low level learning ability, optimization capability and computational power of neural networks into fuzzy systems and give the high-level human-like thinking and reasoning of fuzzy systems to neural networks. Besides, the fuzzy inference systems can be adapted into with the ability of self-tuning and getting closer to neural networks.
The rest of this paper is organized as follows. In section 2, we introduce the traditional Adaboost method, functional neural fuzzy classifier proposed early, and our proposed AdaBoost-based approaches, and section III to their performance applying on three classification benchmarks. Finally, conclusions are drawn in Section 4.

The Proposed AdaBoost-based Approaches
This section describes the proposed AdaBoost-based incremental functional neural fuzzy classifier (AIFNFC) and AdaBoost-based fixed functional neural fuzzy classifier (AFFNFC). Besides, the AFFNFC can be divided into two different models, named the AFFNFC-1 and AFFNFC-2, based on the influence factor c. The general architecture of the proposed classifiers is shown in Figure 1.

The Original AdaBoost Method
The AdaBoost method constructs a complex nonlinear strong classifier as a linear combination of simpler weak classifier. In every epoch, the method assigns the weights to the weak classifiers for committee voting, and a weak classifier with lower error rate will be assigned a larger weight. In the following, we show the details of the AdaBoost algorithm in the form of pseudo codes [15].

For
(1) Fit a classifier to the training data using weights ; (2) Calculate the training error Output: In step 3, represents an identical function, when it returns 1, otherwise is 0; is the error rate, is the logodds rate of and the weighted value of weak classifier at the epoch. It can be easily observed that a smaller will increase the value of ; while is closed to 1/2, will approach to zero.

The Functional Neural Fuzzy Classifier
Before introducing the proposed AdaBoost-based classifiers, the weak classifiers -functional neural fuzzy classifier (FNFC) [16], is briefly reviewed in this subsection. The FNFC was a traditional TSK-type functional neural fuzzy network integrated with the functional-link neural network (FLNN) [17] generating complex nonlinear combinations of input variables to the consequent part of the fuzzy rules. As shown in Figure 2, the network architecture can be divided into five layers.
The FNFC implements a fuzzy IF-THEN rule as following,

Rule-j:
where and are the input and local output variables, respectively; N is the number of input variables; Rule-j is the fuzzy rule; is the linguistic term of precondition part with Gaussian membership function; is the link weight of the local output; is the basis trigonometric function of input variables and M is the number of basis function.
Functional Expansion (1) Layer 2: The calculated membership value represents the degree which an input value belongs to a fuzzy set in layer 2. We use the Gaussian membership function here, which is defined as (2) where and are the mean and variance of Gaussian membership function, respectively.

Layer 3:
The product operator is adopted to perform the precondition part of the fuzzy rules. The corresponding equation of each inference node is presented as, Layer 4: Node in layer 4 received the output from layer 3 and the FLNN output. As a result, the equation can be summarized as following, where is the corresponding link weight of the FLNN and is the functional expansion of input variables. The functional expansion uses a trigonometric polynomial basis function, given by for twodimensional input variables. Therefore, M is the number of basis functions, , where N is the number of input variables.

Layer 5:
The out node in layer 5 acts as a defuzzifier with (5) where R is number of fuzzy rules and y is the output of FNFC model.
The learning algorithm of the FNFC includes the structure learning phase and parameter learning phase. The division of input space and the establishment of fuzzy rules are performed automatically in structure learning phase; meanwhile, the link weights in consequent part and the parameters of the membership function are adjusted in the parameter learning phase. The details of the FNFC can be found in [16].

The Proposed AIFNFC Approach
When handling the multi-class classification problem, most boost-based methods restricted the problem into multiple two-class problems. Without this restriction, we proposed a new AdaBoost-based incremental functional neural fuzzy classifier (AIFNFC), which uses the FNFCs as the weak classifiers. By using the FNFC as the weak classifier, AIFNFC approach improves the disadvantage of boost-based methods for multi-class classification. In each epoch, the weights will be updated by AdaBoost approach; the better classifier is assigned a higher weight to improve the final classification result. The pseudo code of AIFNFC is listed in the following.

The Proposed AFFNFC Approach
We observe that the poor classifier in the AIFNFC approach is generally followed by a relatively better classifier. Hence, one modification of AIFNFC is needed to force all classifiers on contributing themselves with greater effectiveness. The proposed AdaBoost-based fixed functional neural fuzzy classifier (AFFNFC) is characterized by a customized amount of weak classifiers, and the AdaBoost method is used to update the weight of these weak classifiers that lastly contribute the final outcome via their weights. In addition, the amount of weak classifiers should be defined before executing the epoch process. The pseudo code of AIFNFC is listed in the following.  (11) (2) Calculate the weight of this weak classifier by (12) where the influence factor (3) Update the i th data weight by (13)   There are two design schemes in this paper for different influence factor c. When there are some classifiers of the same error rate, the ranking factor r would be set according to their processing order, and this scheme is named AFFNFC-1. The other design scheme, called the AFFNFC-2, will assign the same ranking factor r to all the classifiers of the identical error rate.

Experimental Results
In this study, three benchmark dataset, Iris, Wine, and Breast Cancer are measured to evaluate the performance of the proposed methods. These three datasets come from the Donald Bren School of Information and Computer Science University of California at Irvine (UCI), the well-known Machine Learning Repository website [18]; Iris classification is a low-dimensional problem, and Breast Cancer and Wine classifications are multi-dimensional problems. Table 1 summarizes the amounts of samples, attributes and classified classes. Since this website recoded many benchmark datasets for machine learning, researchers can easily apply these benchmarks to verify their classification approaches, and conduct the necessary comparison of the corresponding training and testing results. In the rest of this section, we compare the performances of AIFNFC, AFFNFC-1, AFFNFC-2, and other methods combined with FNFC model, such as voting method with multiple FNFCs and single FNFC. To compare with other methods, all the commonly used parameters are set identical initially, the learning rate η is set to 0.01, the entropy is set to 0.3, the amount of training epochs is set to 100 times, and the numbers of weak classifiers FNFCs are respectively set to 5, 10 and 15 for different experiment. In these experiments, half patterns of dataset are selected randomly as the training set, and the other patterns are used as testing data. For example, 75 instances of original 150 examples in Iris dataset will be used as training data and the rest 75 instances are applied for the testing. In addition, the randomized selecting process will be carried out five times for each benchmark dataset.

Example 1: Classification of the Iris Data
The Iris dataset is by Fisher [19], which is frequently chosen to verify newly designed algorithms in machine learning. The classification of the iris dataset is to differentiate the species based on the iris sepal (length and width) and petal (length and width). The Iris data comprises 150 actual instances that are distributed into three species: Iris Setosa, Iris Versicolout and Iris Virginica. Each iris dataset has four input features and one predicted attribute, , shown in Table 2. We repeated the experiment on 5 different trainingtesting datasets that were obtained via a random selecting process from the original Iris data. Table 3 summarizes the five-run averaged experimental accuracy rates of the AIFNFC, AFFNFC-1, AFFNFC-2, single FNFC, and voting method with multiple FNFCs. Compared to the other approaches, the proposed AFFNFCs are generally with better performances.

Example 2: Classification of the Wine Data
The Wine dataset [20] is the results of a chemical analysis of 178 wines which are brewed in the same area in Italy but derived from three different cultivars. The classification of wine type is based on 13 constituents found in the wines. The information of constituents and predicted attribute are presented in Table 4.  Table 5 lists the five-run averaged accuracy rates experimented on the Wine dataset. In this experiment, the AIFNFC method with fewer classifiers is a bit weak on performance since the applied number of AIFNFC classifiers is not fixed. Therefore, there are fluctuations of accuracy rate in classification results of the AIFNFC method. But, the proposed AFFNFCs still have better performance than other methods.  Table 6. Better testing classification results are with the AIFNFC method, since the AIFNFC is great at improving the best classifier which finally influences the final classification results. The AFFNFC always has good results in training phase, but it did not conduct a superior performance in this case. The experimental results are listed in Table 7. When massive voting decisions are applied, the experimental results show that the wrong decisions of most of the classifiers will increase the misclassification rate for the voting method with multiple FNFCs. On the other hand, the proposed AdaBoost-based approaches have the benefit of arranging larger associated weights for the classifiers of higher accuracies, which release the influence of low-accuracy classifiers. Besides, if a training instance is misclassified by a classifier of greater weight, the final classification result can be improved with the other classifiers when the summation of their weights is bigger than the one of misguided classifier. For example, for the 34th instance in Iris dataset, both the AFFNFC and voting method are with 7 misclassified classifiers, then the voting method makes a wrong decision, but the final classification of the AFFNFC remained correct.

Conclusions
In this paper, we propose two intelligent AdaBoostbased approaches, AIFNFC and AFFNFC, for the classification applications. The AIFNFC approach will increase the amount of functional neural fuzzy classifiers based on the corresponding error during the training phase; while the AFNFC approach is equipped with a fixed amount of functional neural fuzzy classifiers. Then, the weights of AdaBoost procedure are assigned for classifiers. Experimental results show that the proposed AIFNFC and AFFNFC approaches can effectively improve the classification performances.