第1页 / 共212页
第2页 / 共212页
第3页 / 共212页
第4页 / 共212页
第5页 / 共212页
第6页 / 共212页
第7页 / 共212页
第8页 / 共212页
深度脉冲神经网络.pdf
Abstract
Declaration
Copyright Statement
Acknowledgements
List of Acronyms
List of Symbols
Introduction
Motivation and Aims
Thesis Statement and Hypotheses
Contributions
Papers and Workshops
Papers
Workshops
Thesis Structure
Summary
Spiking Neural Networks (SNNs)
Biological Neural Components
Neuron
Neuronal Signals
Signal Transmission
Modelling Spiking Neurons
Neural Dynamics
Neuron Models
Synapse Model
Synaptic Plasticity
Simulating Networks of Spiking Neurons
Software Simulators
Neuromorphic Hardware
Neuromorphic Sensory and Processing Systems
Summary
Deep Learning
Brief Overview
Classical Models
Combined Approaches
Convolutional Networks
Network Architecture
Backpropagation
Activation Function and Vanishing Gradient
Autoencoders (AEs)
Structure
Training
Restricted Boltzmann Machines (RBMs)
Energy-based Model
Objective Function
Contrastive Divergence
Summary
Off-line SNN Training
Introduction
Related Work
Siegert: Modelling the Response Function
Biological Background
Mismatch of The Siegert Function to Practice
Noisy Softplus (NSP)
Generalised Off-line SNN Training
Mapping NSP to Concrete Physical Units
Parametric Activation Functions (PAFs)
Training Method
Fine Tuning
Results
Experiment Description
Single Neuronal Activity
Learning Performance
Recognition Performance
Power Consumption
Summary
On-line SNN Training with SRM
Introduction
Related Work
Spike-based Rate Multiplication (SRM)
Training Deep SNNs
Experimental Setup
AEs
Noisy RBMs
Spiking AEs
Spiking RBMs
Problem of Spike Correlations
Solution 1 (S1): Longer STDP Window
Solution 2 (S2): Noisy Threshold
Solution 3 (S3): Teaching Signal
Combined Solutions (S4)
Case Study
Experimental Setup
Trained Weights
Classification Accuracy
Reconstruction
Summary
Benchmarking Neuromorphic Vision
Introduction
Related Work
NE Dataset
Guiding Principles
The Dataset: NE15-MNIST
Data Description
Performance Evaluation
Model-level Evaluation
Hardware-level Evaluation
Results
Training
Testing
Evaluation
Summary
Conclusion and Future Work
Confirming Research Hypotheses
Future Work
Off-line SNN Training
On-line Biologically-plausible Learning
Evaluation on Neuromorphic Vision
Closing Remarks
Detailed Derivation Process of Equations
Detailed Experimental Results
DEEP SPIKING NEURAL NETWORKS
A thesis submitted to the University of Manchester
for the degree of Doctor of Philosophy
in the Faculty of Science and Engineering
2018
Qian Liu
School of Computer Science
Contents
Abstract
Declaration
Copyright Statement
Acknowledgements
List of Acronyms
List of Symbols
1 Introduction
1.1 Motivation and Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Thesis Statement and Hypotheses . . . . . . . . . . . . . . . . . . . . .
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Papers and Workshops . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.1 Papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.2 Workshops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 Spiking Neural Networks (SNNs)
2.1 Biological Neural Components . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 Neuronal Signals
. . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.3
Signal Transmission . . . . . . . . . . . . . . . . . . . . . . . . .
2
10
11
12
13
14
17
21
24
25
26
28
28
30
31
31
33
33
34
35
36
2.2 Modelling Spiking Neurons . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 Neural Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2 Neuron Models
. . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.3
Synapse Model
. . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.4
Synaptic Plasticity . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Simulating Networks of Spiking Neurons
. . . . . . . . . . . . . . . . .
2.3.1
Software Simulators . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2 Neuromorphic Hardware . . . . . . . . . . . . . . . . . . . . . .
2.3.3 Neuromorphic Sensory and Processing Systems
. . . . . . . . .
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Deep Learning
3.1 Brief Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1 Classical Models
. . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.2 Combined Approaches
. . . . . . . . . . . . . . . . . . . . . . .
3.2 Convolutional Networks
. . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . .
3.2.2 Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.3 Activation Function and Vanishing Gradient . . . . . . . . . . .
3.3 Autoencoders (AEs)
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1
Structure
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.2 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Restricted Boltzmann Machines (RBMs) . . . . . . . . . . . . . . . . .
3.4.1 Energy-based Model
. . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 Objective Function . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.3 Contrastive Divergence . . . . . . . . . . . . . . . . . . . . . . .
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 Off-line SNN Training
4.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Siegert: Modelling the Response Function . . . . . . . . . . . . . . . .
4.3.1 Biological Background . . . . . . . . . . . . . . . . . . . . . . .
3
37
37
39
43
44
45
46
47
48
50
51
51
52
53
54
54
56
58
59
59
60
61
61
62
63
64
65
65
66
69
69
4.3.2 Mismatch of The Siegert Function to Practice . . . . . . . . . .
4.3.3 Noisy Softplus (NSP) . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Generalised Off-line SNN Training . . . . . . . . . . . . . . . . . . . . .
4.4.1 Mapping NSP to Concrete Physical Units
. . . . . . . . . . . .
4.4.2 Parametric Activation Functions (PAFs) . . . . . . . . . . . . .
4.4.3 Training Method . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.4 Fine Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.1 Experiment Description . . . . . . . . . . . . . . . . . . . . . .
4.5.2
Single Neuronal Activity . . . . . . . . . . . . . . . . . . . . . .
4.5.3 Learning Performance
. . . . . . . . . . . . . . . . . . . . . . .
4.5.4 Recognition Performance . . . . . . . . . . . . . . . . . . . . . .
4.5.5 Power Consumption . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 On-line SNN Training with SRM
5.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
71
78
80
81
81
85
86
86
87
87
90
91
95
95
97
97
99
5.3 Spike-based Rate Multiplication (SRM) . . . . . . . . . . . . . . . . . . 101
5.4 Training Deep SNNs
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.4.2 AEs
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.4.3 Noisy RBMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4.4
Spiking AEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.4.5
Spiking RBMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.5 Problem of Spike Correlations . . . . . . . . . . . . . . . . . . . . . . . 119
5.5.1
Solution 1 (S1): Longer STDP Window . . . . . . . . . . . . . . 120
5.5.2
Solution 2 (S2): Noisy Threshold . . . . . . . . . . . . . . . . . 120
5.5.3
Solution 3 (S3): Teaching Signal . . . . . . . . . . . . . . . . . . 123
5.5.4 Combined Solutions (S4) . . . . . . . . . . . . . . . . . . . . . . 124
5.6 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.6.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 124
4
5.6.2 Trained Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.6.3 Classification Accuracy . . . . . . . . . . . . . . . . . . . . . . . 129
5.6.4 Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6 Benchmarking Neuromorphic Vision
139
6.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.3 NE Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.3.1 Guiding Principles
. . . . . . . . . . . . . . . . . . . . . . . . . 144
6.3.2 The Dataset: NE15-MNIST . . . . . . . . . . . . . . . . . . . . 145
6.3.3 Data Description . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 148
6.4.1 Model-level Evaluation . . . . . . . . . . . . . . . . . . . . . . . 149
6.4.2 Hardware-level Evaluation . . . . . . . . . . . . . . . . . . . . . 154
6.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.5.1 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.5.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.5.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7 Conclusion and Future Work
165
7.1 Confirming Research Hypotheses
. . . . . . . . . . . . . . . . . . . . . 165
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7.2.1 Off-line SNN Training . . . . . . . . . . . . . . . . . . . . . . . 170
7.2.2 On-line Biologically-plausible Learning . . . . . . . . . . . . . . 172
7.2.3 Evaluation on Neuromorphic Vision . . . . . . . . . . . . . . . . 173
7.3 Closing Remarks
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
A Detailed Derivation Process of Equations
B Detailed Experimental Results
177
179
Word count 41086
5
List of Tables
4.1 LIF parameter settings. . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 SNN training methods comparison.
. . . . . . . . . . . . . . . . . . . .
72
93
5.1 Parameter setting of SRM. . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.2 Mean synaptic event rate.
. . . . . . . . . . . . . . . . . . . . . . . . . 134
6.1 SNN descriptions at the model level . . . . . . . . . . . . . . . . . . . . 150
6.2 Model-level comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.3 Hardware-level comparison . . . . . . . . . . . . . . . . . . . . . . . . . 153
6.4 LIF parameter setting using PyNN. . . . . . . . . . . . . . . . . . . . . 156
6.5 Comparisons of NEST and SpiNNaker performance. . . . . . . . . . . . 162
6
List of Figures
1.1 The outline of the thesis.
. . . . . . . . . . . . . . . . . . . . . . . . .
32
2.1 Two neurons connected by synapses [Reece et al., 2011; Hodgkin and
Huxley, 1939]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Example of rate coding [Hubel and Wiesel, 1962].
. . . . . . . . . . . .
2.3 Example of temporal coding [Liu et al., 2013].
. . . . . . . . . . . . . .
2.4 Post-synaptic potential.
. . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Summation of post-synaptic potentials [Reece et al., 2011].
. . . . . . .
2.6 Comparisons of an artificial and a spiking neuron.
. . . . . . . . . . . .
2.7 The cell membrane acts like a RC circuit [Gerstner et al., 2014].
. . . .
2.8 Spike-Timing-Dependent Plasticity (STDP) [Bi and Poo, 2001].
. . . .
2.9 Neuromorphic hardware systems using SpiNNaker. . . . . . . . . . . . .
3.1 Typical ConvNet architecture. . . . . . . . . . . . . . . . . . . . . . . .
3.2 An artificial neuron.
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Activation functions
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 A typical Autoencoder structure.
. . . . . . . . . . . . . . . . . . . . .
3.5 A typical RBM structure.
. . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Gibbs sampling on a RBM.
. . . . . . . . . . . . . . . . . . . . . . . .
4.1 A spiking neuron. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Response function of the LIF neuron. . . . . . . . . . . . . . . . . . . .
4.3 Recorded response firing rate driven by NoisyCurrentSource.
. . . . . .
4.4 NoisyCurrentSource samples from a Gaussian distribution.
. . . . . . .
4.5 Recorded response firing rate driven by a noisy synaptic current. . . . .
4.6 Noisy currents generated by Poisson spike trains.
. . . . . . . . . . . .
34
36
37
38
38
40
40
44
49
54
56
58
60
61
64
67
71
73
74
76
77
7
4.7 NSP models the LIF response function. . . . . . . . . . . . . . . . . . .
4.8 NSP in 3D.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.9 NSP fits to the response firing rates.
. . . . . . . . . . . . . . . . . . .
4.10 A conventional artificial neuron. . . . . . . . . . . . . . . . . . . . . . .
4.11 An artificial spiking neuron modelled by NSP.
. . . . . . . . . . . . . .
4.12 An artificial spiking neuron modelled by PAF-NSP.
. . . . . . . . . . .
4.13 Images presented in spike trains convolve with a weight kernel. . . . . .
4.14 The recorded firing rate of the convolution outcomes.
. . . . . . . . . .
4.15 Comparisons of loss during training. . . . . . . . . . . . . . . . . . . . .
4.16 Classification accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.17 The classification accuracy after fine tuning.
. . . . . . . . . . . . . . .
79
80
82
83
84
84
88
89
90
91
94
5.1 ReSuMe algorithm.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.2 The architecture of an ADALINE network. . . . . . . . . . . . . . . . . 101
5.3 Rectangular STDP curve.
. . . . . . . . . . . . . . . . . . . . . . . . . 103
5.4 Reconstruction using AEs and RBMs. . . . . . . . . . . . . . . . . . . . 106
5.5 Noisy input gathered from Poisson spike trains.
. . . . . . . . . . . . . 107
5.6 AE training of the reconstruction tests. . . . . . . . . . . . . . . . . . . 109
5.7 AE-NI training of the reconstruction tests.
. . . . . . . . . . . . . . . . 110
5.8 nRBM training of the reconstruction tests.
. . . . . . . . . . . . . . . . 112
5.9 nRBM-NI training of the reconstruction tests.
. . . . . . . . . . . . . . 113
5.10 Network architecture and the learning algorithm of a spiking AE.
. . . 115
5.11 SAE training of the reconstruction tests.
. . . . . . . . . . . . . . . . . 116
5.12 Network architecture and the learning algorithm of a spiking RBM.
. . 117
5.13 SRBM training of the reconstruction tests.
. . . . . . . . . . . . . . . . 118
5.14 Comparisons of solutions in training SAE.
. . . . . . . . . . . . . . . . 121
5.15 Comparisons of solutions in training SRBM.
. . . . . . . . . . . . . . . 122
5.16 AE and RBM structure for MNIST tasks.
. . . . . . . . . . . . . . . . 125
5.17 Comparisons of trained weights using (spiking) AEs.
. . . . . . . . . . 127
5.18 Comparisons of trained weights using (spiking) RBMs.
. . . . . . . . . 128
5.19 Comparisons of classification accuracy between conventional and spik-
ing models.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.20 Classification accuracy per time step.
. . . . . . . . . . . . . . . . . . . 132
8
相关推荐
2023年江西萍乡中考道德与法治真题及答案.doc
2012年重庆南川中考生物真题及答案.doc
2013年江西师范大学地理学综合及文艺理论基础考研真题.doc
2020年四川甘孜小升初语文真题及答案I卷.doc
2020年注册岩土工程师专业基础考试真题及答案.doc
2023-2024学年福建省厦门市九年级上学期数学月考试题及答案.doc
2021-2022学年辽宁省沈阳市大东区九年级上学期语文期末试题及答案.doc
2022-2023学年北京东城区初三第一学期物理期末试卷及答案.doc
2018上半年江西教师资格初中地理学科知识与教学能力真题及答案.doc
2012年河北国家公务员申论考试真题及答案-省级.doc
2020-2021学年江苏省扬州市江都区邵樊片九年级上学期数学第一次质量检测试题及答案.doc
2022下半年黑龙江教师资格证中学综合素质真题及答案.doc
