A Novel Single Neuron Perceptron with Universal Approximation and XOR Computation Properties

Hindawi Publishing Corporation Computational Intelligence and Neuroscience Volume 2014, Article ID 746376, 6 pages http://dx.doi.org/10.1155/2014/746376 Research Article A Novel Single Neuron Perceptron with Universal Approximation and XOR Computation Properties Ehsan Lotfi1 and M.-R. Akbarzadeh-T2 1 2 Department of Computer Engineering, Torbat-e-Jam Branch, Islamic Azad University, Torbat-e-Jam, Iran Electrical and Computer Engineering Departments, Center of Excellence on Soft Computing and Intelligent Information Processing, Ferdowsi University of Mashhad, Iran Correspondence should be addressed to Ehsan Lotfi; Received 9 February 2014; Accepted 7 April 2014; Published 28 April 2014 Academic Editor: Cheng-Jian Lin Copyright © 2014 E. Lotfi and M.-R. Akbarzadeh-T. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. We propose a biologically motivated brain-inspired single neuron perceptron (SNP) with universal approximation and XOR computation properties. This computational model extends the input pattern and is based on the excitatory and inhibitory learning rules inspired from neural connections in the human brain’s nervous system. The resulting architecture of SNP can be trained by supervised excitatory and inhibitory online learning rules. The main features of proposed single layer perceptron are universal approximation property and low computational complexity. The method is tested on 6 UCI (University of California, Irvine) pattern recognition and classification datasets. Various comparisons with multilayer perceptron (MLP) with gradient decent backpropagation (GDBP) learning algorithm indicate the superiority of the approach in terms of higher accuracy, lower time, and spatial complexity, as well as faster training. Hence, we believe the proposed approach can be generally applicable to various problems such as in pattern recognition and classification. 1. Introduction In various computer applications such as pattern recognition, classification, and prediction, a learning module can be implemented by various approaches including statistical, structural, and neural approaches. Among these methods, artificial neural networks (ANNs) are inspired by physiological workings of the brain. They are based on mathematical model of single neural cell (neuron) named single neuron perceptron (SNP) and try to resemble the actual networks of neurons in the brain. As computational models, SNP has particular characteristics such as the ability to learn and generalize. Although the multilayer perceptron (MLP) can approximate any functions [1, 2], traditional SNP is not universal approximator. MLP can learn through the error backpropagation algorithm (EBP), whereby the error of output units is propagated back to adjust the connecting weights within the network. In MLP architecture, by increasing the number of neurons in input layer or (and) the number of neurons in output layer or (and) the number of neurons in hidden layer(s), the number of learning parameters and the algorithm computational complexity are significantly increased. This problem is usually referred to as the curse of dimensionality [3, 4]. So many researchers have tried to propose more powerful single layer architectures and faster algorithms such as functional link networks (FLNs) and Levenberg-Marquardt (LM) and its modified and extended versions [5–20]. In contrast to the MLP, SNP and FLNs do not impose high computational complexity and are far from the curse of dimensionality. But because of disregarding the universal approximation property, SNP and FLNs are not very popular in the applications. In contrast to the previse knowledge about SNP, this paper aims to propose a novel SNP model that can solve the XOR problem and we show that it can be universal approximator. Proposed SNP can solve XOR problem only if additional nonlinear operator is used. As illustrated in the next section, the SNP universal approximation property can simply be archived by extending the input patterns and using the nonlinear operator max. Like functional link networks 2 Computational Intelligence and Neuroscience Input: Initial random weights; w1 , w2 , . . . , wn , wn+1 and input bias b (1) Take 𝑘th learning sample (𝑘th 𝑝 and 𝑇) (2) 𝑝𝑛+1 = max𝑗=1,...,𝑛 (𝑝𝑗 ) (3) Calculate the final output 𝐸𝑜 and error 𝑛+1 𝐸𝑜 = tan sig ( ∑ 𝑤𝑗 × 𝑝𝑗 + 𝑏) 𝑗=1 𝑒 = 𝑇 − 𝐸𝑜 (4) Update the weights by using excitatory rule 𝑤𝑗 = 𝑤𝑗 + 𝛼 max(𝑒, 0)𝑠𝑗 ; for 𝑗 = 1, . . . , 𝑛 + 1 𝑏 = 𝑏 + 𝛼 max(𝑒, 0) (5) Update the weights by using inhibitory rule 𝑤𝑗 = 𝑤𝑗 − 𝛼 max(−𝑒, 0)𝑠𝑗 ; for 𝑗 = 1, . . . , 𝑛 + 1 𝑏 = 𝑏 − 𝛼 max(−𝑒, 0) (6) If 𝑘 < number of training patterns then 𝑘 = 𝑘 + 1 and proceed to the first (7) Let epoch = epoch + 1 and 𝑘 = 1 (8) If the stop criterion has not satisfied proceed to the first Algorithm 1: Proposed SNP algorithm. Actually, max operation increases the input dimension to 𝑛 + 1. So, the new input pattern has 𝑛 + 1 elements. In Figure 1, the input pattern is illustrated by vector 𝑝1 ≤𝑗≤𝑛+1 and the 𝐸𝑜 calculated by the following formula is the final output: 𝑛+1 𝐸𝑜 (𝑝) = 𝑓 ( ∑ 𝑤𝑗 × 𝑝𝑗 + 𝑏) , (2) 𝑗=1 where 𝑓 is activation function and 𝑤1 , 𝑤2 , . . . , 𝑤𝑛+1 , and b are adjustable weights. So, error can be achieved as follows: Figure 1: Proposed SNP. 𝑒 = 𝑇 − 𝐸𝑜 (FLNs) [21], the proposed SNP does not include hidden units or expand the input vector, but guarantees universal approximation. FLNs are single-layer neural networks that can be considered as an alternative approach in the data mining to overcome the complexities associated with MLP [22] but they do not guarantee universal approximation. The paper is organized as follows. Proposed SNP and universal approximation theorem are proposed in Section 2. Section 3 presents the numerical results, where the proposed SNP is compared with backpropagation MLP. There are various versions of backpropagation algorithms. In classification problems, we compare with gradient descent backpropagation (GDBP) [23], that is, the standard basic algorithm. Finally, conclusions are made in Section 4. 2. Proposed Single Neuron Perceptron Figure 1 shows the proposed SNP. In the figure, the model is presented as 𝑛 + 1-inputs single-output architecture. The variable 𝑝 is the input pattern and the variable 𝑇 is related target applied in the learning process (3). Let us extend the input pattern as follows: 𝑝𝑛+1 = max (𝑝𝑗 ) . 𝑗=1,...,𝑛 (1) (3) and the learning weights can be adjusted by the following excitatory learning rule: 𝑤𝑗 = 𝑤𝑗 + 𝛼 max (𝑒, 0) 𝑝𝑗 ; for 𝑗 = 1, . . . , 𝑛 + 1 (4) and then by the following inhibitory rule: 𝑤𝑗 = 𝑤𝑗 − 𝛼 max (−𝑒, 0) 𝑝𝑗 ; for 𝑗 = 1, . . . , 𝑛 + 1, (5) where 𝑇 is target, 𝐸𝑜 is output of network, 𝑒 is related error, and 𝛼 is the learning rate. Also 𝑏 can be trained by 𝑏 = 𝑏 + 𝛼 max (𝑒, 0) , 𝑏 = 𝑏 − 𝛼 max (−𝑒, 0) . ( (...truncated)