Demystifying Deep Learning: An Introduction to the Mathematics of Neural Networks

Table of Contents:
About the Author Acronyms 1 Introduction 1 1.1 AI/ML - Deep Learning? 7 1.2 A Brief History 9 1.3 The Genesis of Models 14 1.3.1 Rise of the Empirical Functions 15 1.3.2 The Biological Phenomenon and the Analogue 20 1.4 Numerical Computation - Computer Numbers Are Not Real 21 1.4.1 The IEEE 754 Floating Point System 24 1.4.2 Numerical Coding Tip: Think in Floating Point 28 1.5 Summary 31 1.6 projects 32 2 Deep Learning and Neural Networks 33 2.1 Feed-Forward and Fully-Connected Artificial Neural Networks 34 2.2 Computing Neuron State 41 2.2.1 Activation Functions 42 2.3 The Feed-Forward ANN Expressed with Matrices 45 2.3.1 Neural Matrices: A Convenient Notation 47 2.4 Classification 48 2.4.1 Binary Classification 50 2.4.2 One-Hot Encoding 52 2.4.3 The Softmax Layer 54 2.5 Summary 57 2.6 Projects 58 3 Training Neural Networks 59 3.1 Preparing the Training Set: Data Preprocessing 60 3.2 Weight Initialization 66 3.3 Training Outline 68 3.4 Least Squares: A Trivial Example 71 3.5 Backpropagation of Error for Regression 75 3.5.1 The Terminal Layer (Output) 80 3.5.2 Backpropagation: The Shallower Layers 84 3.5.3 The Complete BackPropagation Algorithm 90 3.5.4 A Word on the Rectified Linear Unit (ReLU) 92 3.6 Stochastic Sine 95 3.7 Verification of a Software Implementation 98 3.8 Summary 105 3.9 Projects 106 4 Training Classifiers 107 4.1 Backpropagation for Classifiers 107 4.1.1 Likelihood 108 4.1.2 Categorical Loss Functions 111 4.2 Computing the Derivative of the Loss 114 4.2.1 Initiate Backpropagation 118 4.3 Multilabel Classification 120 4.3.1 Binary Classification 121 4.3.2 Training A Multilabel Classifier ANN 122 4.4 Summary 124 4.5 Projects 124 5 Weight Update Strategies 127 5.1 Stochastic Gradient Descent 128 5.2 Weight Updates as Iteration and Convex Optimization 134 5.2.1 Newton’s Method for Optimization 137 5.3 RPROP+ 141 5.4 Momentum Methods 144 5.4.1 AdaGrad and RMSProp 145 5.4.2 ADAM 147 5.5 Levenberg-Marquard Optimization for Neural Networks 151 5.6 Summary 159 5.7 Projects 159 6 Convolutional Neural Networks 161 6.1 Motivation 162 6.2 Convolutions and Features 164 6.3 Filters 170 6.4 Pooling 174 6.5 Feature Layers 175 6.6 Training a CNN 179 6.6.1 Flatten and the Gradient 180 6.6.2 Pooling and the Gradient 181 6.6.3 Filters and the Gradient 183 6.7 Applications 189 6.8 Summary 190 6.9 Projects 190 7 Fixing the Fit 193 7.1 Quality of the Solution 193 7.2 Generalization Error 194 7.2.1 Bias 195 7.2.2 Variance 197 7.2.3 The Bias-Variance Tradeoff 198 7.2.4 The Bias-Variance Tradeoff in Context 199 7.2.5 The Test Set 201 7.3 Classification Performance 204 7.4 Regularization 209 7.4.1 Forward Pass During Training 210 7.4.2 Forward Pass During Normal Inference 211 7.4.3 Backpropagation of Error 212 7.5 Advanced Normalization 216 7.5.1 Batch Normalization 217 7.5.2 Layer Normalization 223 7.6 Summary 228 7.7 Projects 228 8 Design Principles for a Deep Learning Training Library 231 8.1 Computer Languages 232 8.2 The Matrix: Crux of a Library Implementation 239 8.2.1 Memory Access and Modern CPU Architectures 241 8.2.2 Designing Matrix Computations 246 8.3 The Framework 251 8.4 Summary 255 8.5 Projects 255 9 Vistas 257 9.1 The Limits of ANN Learning Capacity 257 9.2 Generative Adversarial Networks 260 9.2.1 GAN Architecture 262 9.2.2 The GAN Loss Function 265 9.3 Reinforcement Learning 269 9.3.1 The Elements of Reinforcement Learning 272 9.3.2 A Trivial RL Training Algorithm 275 9.4 Natural Language Processing Transformed 284 9.4.1 The Challenges of Natural Language 285 9.4.2 Word Embeddings 287 9.4.3 Attention 291 9.4.4 Transformer Blocks 294 9.4.5 Multi-Head Attention 300 9.4.6 Transformer Applications 303 9.5 Neural Turing Machines 305 9.6 Summary 309 9.7 Projects 309 A Mathematical Review 311 A.1 Linear Algebra 311 A.1.1 Vectors 311 A.1.2 Matrices 313 A.1.3 Matrix Properties 316 A.1.4 Linear Independence 317 A.1.5 The QR Decomposition 317 A.1.6 Least Squares 318 A.1.7 Eigenvalues and Eigenvectors 319 A.1.8 Hadamard Operations 319 A.2 Basic Calculus 320 A.2.1 The Product Rule 321 A.2.2 The Chain Rule 322 A.2.3 Multi-Variable Functions 322 A.2.4 Taylor Series 323 A.3 Advanced Matrices 324 A.4 Probability 324 B Glossary 327 Bibliography 336 Index 363

About the Author :
Douglas Santry, PhD, is a lecturer in Computer Science at the University of Kent, UK. Dr. Santry obtained his PhD from the University of Cambridge. Prior to his current position, he worked extenstively as an important figure in industry with Apple Computer Corp, NetApp and Goldman Sachs.

Review :
"I recently read DEMYSTIFYING DEEP LEARNING and it really exceeded my expectations! It is incredibly comprehensive and well organized with plenty of references to examples available on the book's website. The amount of knowledge in the book makes it a must-have for anyone interested in deep learning. As someone who had only basic knowledge of the subject, the numerous examples available were invaluable in understanding complex concepts.? Alessandro Migliaccio, President, AiShed