Ego autem et domus
mea serviemus Domino.
DEEP TIME SERIES
FORECASTING
With PYTHON
An Intuitive Introduction to Deep Learn-
ing for Applied Time Series Modeling
Dr. N.D Lewis
Copyright © 2016 by N.D. Lewis
All rights reserved. No part of this publication may be reproduced, dis-
tributed, or transmitted in any form or by any means, including photo-
copying, recording, or other electronic or mechanical methods, without
the prior written permission of the author, except in the case of brief quo-
tations embodied in critical reviews and certain other noncommercial uses
permitted by copyright law. For permission requests, contact the author
at: www.AusCov.com.
Disclaimer: Although the author and publisher have made every effort to
ensure that the information in this book was correct at press time, the
author and publisher do not assume and hereby disclaim any liability to
any party for any loss, damage, or disruption caused by errors or omissions,
whether such errors or omissions result from negligence, accident, or any
other cause.
Ordering Information: Quantity sales. Special discounts are available on
quantity purchases by corporations, associations, and others. For details,
email: info@NigelDLewis.com
Image photography by Deanna Lewis with helpful assistance
from Naomi Lewis.
ISBN-13: 978-1540809087
ISBN-10: 1540809080
Contents
Acknowledgements
Preface
How to Get the Absolute Most Possible Benefit from this Book
Getting Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Learning Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . .
1 The Characteristics of Time Series Data Simplified
Understanding the Data Generating Mechanism . . . . . . . . . . . . . . . . .
Generating a Simple Time Series using Python . . . . . . . . . . . . . . . . .
Randomness and Reproducibility
. . . . . . . . . . . . . . . . . . . . . . . .
The Importance of Temporal Order . . . . . . . . . . . . . . . . . . . . . . . .
The Ultimate Goal
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
For Additional Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 Deep Neural Networks Explained
What is a Neural Network? . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Role of Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Deep Learning in a Nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Generating Data for use with a Deep Neural Network . . . . . . . . . . . . .
Exploring the Sample Data . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Translating Sample Data into a Suitable Format
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
A Super Easy Deep Neural Network Tool
Assessing Model Performance . . . . . . . . . . . . . . . . . . . . . . . . . . .
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . .
3 Deep Neural Networks for Time Series Forecasting the Easy Way
Getting the Data from the Internet
. . . . . . . . . . . . . . . . . . . . . . .
Cleaning up Downloaded Spreadsheet Files
. . . . . . . . . . . . . . . . . . .
Understanding Activation Functions . . . . . . . . . . . . . . . . . . . . . . .
How to Scale the Input attributes
. . . . . . . . . . . . . . . . . . . . . . . .
Assessing Partial Autocorrelation . . . . . . . . . . . . . . . . . . . . . . . . .
A Neural Network Architecture for Time Series Forecasting . . . . . . . . . .
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . .
i
iii
viii
1
3
3
4
5
7
7
9
12
13
14
15
17
17
18
19
21
22
25
26
28
30
31
31
33
36
39
42
45
49
4 A Simple Way to Incorporate Additional Attributes in Your Model 51
51
54
56
58
59
60
62
66
Working with Additional Attributes
. . . . . . . . . . . . . . . . . . . . . . .
The Working of the Neuron Simplified . . . . . . . . . . . . . . . . . . . . . .
How a Neural Network Learns . . . . . . . . . . . . . . . . . . . . . . . . . . .
Gradient Descent Clarified . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How to Easily Specify a Model
. . . . . . . . . . . . . . . . . . . . . . . . . .
Choosing a Learning Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Efficient Way to Run Your Model
. . . . . . . . . . . . . . . . . . . . . .
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . .
5 The Simple Recurrent Neural Network
Why Use Keras? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
What is a Recurrent Neural Network? . . . . . . . . . . . . . . . . . . . . . .
Gain Clarity on the Role of the Delay Units . . . . . . . . . . . . . . . . . . .
Follow this Approach to Create Your Train and Test Sets
. . . . . . . . . . .
Parameter Sharing Clarified . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Understand Backpropagation Through Time
. . . . . . . . . . . . . . . . . .
A Complete Intuitive Guide to Momentum . . . . . . . . . . . . . . . . . . .
How to Benefit from Mini Batching . . . . . . . . . . . . . . . . . . . . . . . .
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . .
6 Elman Neural Networks
Prepare You Data for Easy Use . . . . . . . . . . . . . . . . . . . . . . . . . .
How to Model a Complex Mathematical Relationship with No Knowledge . .
Use this Python Library for Rapid Results . . . . . . . . . . . . . . . . . . . .
Exploring the Error Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Super Simple Way to Fit the Model
. . . . . . . . . . . . . . . . . . . . . .
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . .
67
67
68
71
72
73
73
76
78
81
83
84
85
88
89
91
93
7 Jordan Neural Networks
95
96
The Fastest Path to Data Preparation . . . . . . . . . . . . . . . . . . . . . .
97
A Straightforward Module for Jordan Neural Networks . . . . . . . . . . . . .
Assessing Model Fit and Performance . . . . . . . . . . . . . . . . . . . . . .
98
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . . 100
8 Nonlinear Auto-regressive Network with Exogenous Inputs
103
What is a NARX Network? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Spreadsheet Files Made Easy with Pandas . . . . . . . . . . . . . . . . . . . . 105
Working with Macroeconomic Variables . . . . . . . . . . . . . . . . . . . . . 107
Python and Pandas Data Types
. . . . . . . . . . . . . . . . . . . . . . . . . 111
A Tool for Rapid NARX Model Construction . . . . . . . . . . . . . . . . . . 113
How to Run the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . . 117
9 Long Short-Term Memory Recurrent Neural Network
119
Cyclical Patterns in Time Series Data . . . . . . . . . . . . . . . . . . . . . . 119
What is an LSTM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Efficiently Explore and Quickly Understand Data . . . . . . . . . . . . . . . . 123
The LSTM Memory Block in a Nutshell
. . . . . . . . . . . . . . . . . . . . . 127
Straightforward Data Transformation for the Train and Test Sets . . . . . . . 128
Clarify the Role of Gates
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Understand the Constant Error Carousel . . . . . . . . . . . . . . . . . . . . . 131
Specifying a LSTM Model the Easy Way . . . . . . . . . . . . . . . . . . . . 132
Shuffling Examples to Improve Generalization . . . . . . . . . . . . . . . . . . 136
A Note on Vanishing Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Follow these Steps to Build a Stateful LSTM . . . . . . . . . . . . . . . . . . 139
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . . 144
10 Gated Recurrent Unit
145
The Gated Recurrent Unit in a Nutshell . . . . . . . . . . . . . . . . . . . . . 145
A Simple Approach to Gated Recurrent Unit Construction . . . . . . . . . . 148
A Quick Recap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
How to Use Multiple Time Steps . . . . . . . . . . . . . . . . . . . . . . . . . 151
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . . 154
11 Forecasting Multiple Outputs
155
Working with Zipped Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
How to Work with Multiple Targets
. . . . . . . . . . . . . . . . . . . . . . . 159
Creation of Hand Crafted Features . . . . . . . . . . . . . . . . . . . . . . . . 161
Model Specification and Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . . 166
12 Strategies to Build Superior Models
169
Revisiting the UK Unemployment Rate Economic Data . . . . . . . . . . . . 169
Limitations of the Sigmoid Activation Function . . . . . . . . . . . . . . . . . 171
One Activation Function You Need to Add to Your Deep Learning Toolkit . . 172
Try This Simple Idea to Enhance Success
. . . . . . . . . . . . . . . . . . . . 176
A Simple Plan for Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . 180
Additional Resources to Check Out . . . . . . . . . . . . . . . . . . . . . . . . 184
Index
189