Adaptive Control of Neural Network Synthesis

Neural Network Synthesis is a method based on based on Analytic Programming and asynchronous implementation of Self-Organising Migration Algorithm. This synthesis woks as an algorithm capable of creating and learning and artificial neural networks as well as optimizing their structures and connections. This paper introduce an idea of adaptive control of Self-Organising Migration Algorithm based on complexity of processed neural network structure. Such approach already recorded several successful application in modelling and simulation.


Introduction
Neural Network Synthesis (ANN synthesis) is an algorithm capable of creating and learning and artificial neural networks as well as optimizing their structures and connections.To describe this method in all details would significantly exceed a possible extend of this paper.Nevertheless ANN synthesis mechanisms are very well elaborated in papers [1] and [2].An interested reader is respectfully asked to study this previous publications as are referred at the end of the paper.
ANN synthesis is based on Analytic Programming (AP) and asynchronous implementation of Self-Organizing Migration Algorithm (SOMA).Important facts about AP which are necessary for basic understanding of an idea proposed in the paper as an adaptive individual handling are described in chapters 2. And 3. For better understanding of SOMA, please, refer paper [3] or [4].
This paper explores a possibility to make it more effective by adaptive control of SOMA parameters.The main idea is an intelligent control of the ANN synthesis based on complexity of processed neural network structure.

Analytic Programming
The main principle (core) of AP is based on a discrete set handling (DSH) (Fig. 1).DSH shows itself as a universal interface between the EA and the symbolically solved problem.This is why AP can be used almost by any evolutionary algorithm.In the case of ANN synthesis the algorithm is SOMA.=All simple functions and operators are in the so called General Function Set (GFS) divided into groups according to the number of arguments which can be inserted during the evolutionary process to create subsets GFS3, GFS2...GFS0.

Figure 2. GFS subsets hierarchy
The functionality of AP can be seen in the specific example in Fig. 3: The individual consists of 6 arguments (indices, pointers to GFS).The first index is 3, meaning that it is taken from the set of functions GFSall.The function minus has two arguments; therefore indexes 7 and 9 are arguments of minus.6 + 7 (1) Index 7 is then replaced by Abs and index 9 by Sin.
Abs + Sin (2) Abs and Sin are one-argument functions.Then, index 9 follows index 11, which is replaced by t.

Abs(t) + Sin
(3) Sin is also a one-argument function.Then, after index 11, the individual takes index 9, which is replaced by Sin and this Sin becomes an argument of the previous Sin.
Sin(Tan) + Sin(Sin( (4) The last index is 2, but in this case there is the function Plus.Plus needs two arguments to work properly.AP will not allow this, as there is not any other free pointer to be used as the argument.Instead of Plus, AP will jump into the subspace, in this case directly to the GFS0arg.In the GFS0arg it finds the second element, which is K.And by doing so, we get (5).

Abs(t) + Sin(Sin(K))
(5) The number of pointers actually used from an individual before the synthesized expression is closed is called depth.This example is based on the relevant and previously published work in.

Neural Network Synthesis
The previous chapter described basic concepts of AP.Such concepts can be easily employed to synthetize different ANN structures as can be seen from Fig. 4.This process of structural synthesis as well as ANN learning is well described for example in [3] and [4].

Experiment Designed for Adaptive Control
Processes described in chapters 2. and 3. have to be optimized by a specific evolutionary algorithms.In case of ANN synthesis it basically means to try variously complex ANN structures (individuals in AP) to improve them in evolutionary way.In place of such evolutionary algorithm ANN synthesis commonly employ SOMA.An algorithm which is well described in [3] and [4].However SOMA is not anyhow adaptive to coop with various lengths of individuals in AP.SOMA simply treat all such individuals as an individual of constant length (typically 100).For this paper one control parameter of SOMA was chosen to improve this behavior.
An adaptive control proposed in this paper aim to influence control parameter of SOMA named PRT to improve its ability to deal with different lengths of individuals.For each individual SOMA will used different setting of PRT bases on length of a solved individual itself.
Commonly, SOMA is set on PRT = 0.1.In contradiction this paper proposes adaptive strategy so PRT will differ from individual to individual PRT = 1 / depth.
In order to statistically evaluate this new adaptive handling approach the function approximation problem was chosen as an aim of the experiment.The function (6) proposed by [5] as an appropriate approximation benchmark was chosen to be approximated by the ANN.y = xi 5 -2 xi 3 + xi where xi is in <-1,by the step 0.04 ,1> (6) Fig. 5 (automatically generated by ANN synthesis software) shows an example of synthetized ANN approximating (6).The difference between the ANN and ( 7) is depicted as a red area which could be minimized by the process of synthesis.An goal of ANN synthesis in this case can be simply described as founding smallest possible ANN able to approximate given function.Simplicity of such function is purely purposeful here as the main goal is not test ability of ANN synthesis to solve complicated tasks but to test how the method improve while applying proposed adaptive strategy.7) by synthetized ANN AP was executed 100 times (physically on 8 cores of the Super Micro Server) to produce an ANN with the RMSD < 0.005.The main intention was to find such an ANN which met this condition and which simultaneously used as few AN as possible.All SOMA's control parameters required for repetition of this experiment are described in following tables.The setting of Asynchronous SOMA used as the EA for AP can be seen in Table 1. and SOMA setting used for ANN learning in Table 2. Based on experiment setting PRT which SOMA used to optimize Kn is set either conservatively to PRT = 0.1 or adaptively PRT = 1 / depth.

Results
The control of SOMA parameters consists in replacement of the static PRT value by the value which inversely depends on depth of a currently operated individual.A total of 1,189,870 evaluations of AP individual fitness were completed during 100 AP executions while the PRT was set to 0.1 and the separate SOMA run was performed for all of them to set their Kn value.Without the adaptive PRT, AP was able to find an optimal ANN in only 1 case in comparison with 4 successful cases in the original experiment.

Conclusion
ANN synthesis already recorded several successful application considering practical casers of modelling and simulation [6] - [9].It was also applied on large set of widely recognized benchmark functions [10], [11] with respect to the function approximation, prediction and problems.This results vindicate efforts for its further development.
Obtained results of experiment considered in this paper proves an ability of proposed adaptive control to further improve ANN synthesis.It can significantly improve ANN synthesis ability to overshadow concurrent method of ANN optimization as are [12] - [17].
The method of adaptive individual handling proved in this paper are going to be applied on a practical industrial example and comparison of the improved ANN synthesis is to be subjected by a future study.

Figure 4 .
Figure 4.An Example of ANN synthesis

Table 1 .
Setting of SOMA used as EA for AP

Table 2 .
Setting of SOMA used to optimize Kn