Spiking neural network vs multilayer perceptron: who is the winner in the racing car computer game

Urszula Markowska-Kaczmar 0 1 Mateusz Koldowski 0 1 0 U. Markowska-Kaczmar ( 1 Communicated by E. Lughofer The paper presents two neural based controllers for the computer car racing game. The controllers represent two generations of neural networks-a multilayer perceptron and a spiking neural network. They are trained by an evolutionary algorithm. Efficiency of both approaches is experimentally tested and statistically analyzed. - Neural networks (NN) are widely applied in many areas because of their ability to learn. Controllers for various devices are examples of their successful application. In the context of this paper the project ALVINN was one of the first fruitful controller based on a neural network approach. It learns to control a vehicle by observing live sensor data as a human drives the vehicle. The classic multilayer perceptron (MLP) neural network trained using backpropagation method was applied. The project, started in early 90-ties. The input to ALVINN were 30 32 images taken by a camera connected to a digitizer. This technique was further developed by other researchers, eg. the paper (Oh et al. 1998) presents application of reinforcement learning. The work (Cao et al. 2007) describes the controller which comprises a MLP and a radial basis function (RBF). The MLP network is used to adjust the parameters of the controller on-line. The RBF network is used to establish a nonlinear prediction model. In the recent paper (Zheng et al. 2013) the impact of heavy vehicles on lane-changing decisions of following car drivers has been investigated, and the lane-changing model based on two neural network models is proposed. Computer games can be perceived as a virtual representation of real world problems, and they provide rich test beds for many artificial intelligence methods. An example which can be mentioned here is successful application of multilayer perceptron in racing car computer games, described for instance in Ebner and Tiede (2009) and Togelius and Lucas (2005). Inspiration for our research were the results shown in Yee and Teo (2013) presenting spiking neural network based controller for racing car in computer game. This study has shown that cars controlled by evolved spiking neuron models could perform well and they demonstrated sophisticated driving behaviors. The aim of our research was to check whether spiking neural network based car controller outperforms those based on the classic MLP in comparable conditions. The paper differs from the approach described in Yee and Teo (2013) in that, apart from NN weights, parameters of spiking neural model are evolved also. Our racing game is built on purpose of the research referring to the neural based controllers. We performed also a statistical comparison of the results of both types of controllers. The paper consists of six sections. Section 2 presents related works. In Sect. 3 a short description of MLP and spiking networks are presented. In Sect. 4 the basis of evolutionary algorithms used to train both types of neural networks is briefly described. Section 5 presents the details of the developed Neural Racing game and methods applied in the system. The experimental results are shown in Sect. 6. Summary concludes the paper. 2 Related works In the literature we can find many papers devoted to the controllers in games which are based on neural networks. The paper (Charles and McGlinchey 2004) presents a general survey of neural networks applied to computer games. A short survey of neural-based agent approaches in computer games is included in Qualls and Russomanno (2009). These agents have the ability to overcome some of the shortcomings associated with implementing classical AI techniques in computer game design. Neural networks can be used in many diverse ways in computer games ranging from agent control, environmental evolution, to content generation. The paper (Togelius and Lucas 2005) describes the evolution of controllers based on neural networks for racing a simulated radio-controlled car around a track, modelled on a real physical track. Cardamone et al. (2009) claim that online evolutionary learning of a fast controller from scratch can effectively improve the performance achieved during the learning process. Most projects mentioned so far used typical multilayer perceptron networks that are example of second generation of neural networks. Spiking neural networks fall into the third generation of neural network models, which increase the level of realism in a neural simulation. They are more and more popular in developing controllers for robots (Lee and Hallam 1999; Floreano and Mattiussi 2001; Wang et al. 2008; Huemer et al. 2009; Mitic and Miljkovic 2014). The paper (Batllori et al. 2011) describes a sequence of experiments in which a neural network based controller was evolved. The task was light-seeking while avoiding obstacles. The paper (Hagras et al. 2004) proposes adaptive genetic algorithm to train spiking neural network. Bouganis and Shanahan (2010) present a spiking neural network architecture that autonomously learns to control a 4 degree-of-freedom robotic arm. As it was mentioned, in our research we were inspired by the paper of Yee and Teo (2013), but we were focused on the comparison of both types of controllers in the same environment. 3 Neural networks Information processing performed in artificial neural network imitates processes in human brain. When we speak about neural network the following elements are essential: its architecture (the way the neurons are connected), the model of a single neuron and a learning rule. The first model of a nerve cell was proposed by Pitts and McCulloch. It was very simple. A step function was implemented as its activation function, so on its outputs only 0 or 1 was produced. It is the first generation of neural networks. The main problem with these networks was a lack of training rule that would Fig. 1 Feedforward, two layered neural network architecture allow to use more complex networks. The second generation of neural networks was started with backpropagation learning rule elaboration. It needs a differentiable activation function therefore a continuous activation function is necessary in such a model. In order to simulate processes existing in a human brain in more adequate way spiking neural networks were proposed. They can be also trained using the backpropagation method. The evolutionary approach to train NN is very useful when there is no possibility to acquire desired output for a given input pattern, as it is needed in backpropagation training method. 3.1 MLP neural networks MLP neural network with the sigmoidal or hiperbolic tangent activation function is the most popular network in various applications. It consists of layers of neurons. There are no connections between neurons in the same layer. Neurons between neighboring layers are fully interconnected. Information is processed from the input layer to the outpu (...truncated)