Caterini / Chang Deep Neural Networks in a Mathematical Framework

1. Auflage 2018
ISBN: 978-3-319-75304-1
Verlag: Springer International Publishing
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 95 Seiten, eBook

Reihe: SpringerBriefs in Computer Science

This SpringerBrief describes how to build a rigorous end-to-end mathematical framework for deep neural networks. The authors provide tools to represent and describe neural networks, casting previous results in the field in a more natural light. In particular, the authors derive gradient descent algorithms in a unified way for several neural network structures, including multilayer perceptrons, convolutional neural networks, deep autoencoders and recurrent neural networks. Furthermore, the authors developed framework is both more concise and mathematically intuitive than previous representations of neural networks.This SpringerBrief is one step towards unlocking the black box of Deep Learning. The authors believe that this framework will help catalyze further discoveries regarding the mathematical properties of neural networks.This SpringerBrief is accessible not only to researchers, professionals and students working and studying in the field of deep learning, but alsoto those outside of the neutral network community.

Zielgruppe

Research

Autoren/Hrsg.

Caterini, Anthony L.

Chang, Dong Eui

Inhaltsverzeichnis

1;Preface;7
2;Contents;10
3;Acronyms;12
4;1 Introduction and Motivation;13
4.1;1.1 Introduction to Neural Networks;14
4.1.1;1.1.1 Brief History;14
4.1.2;1.1.2 Tasks Where Neural Networks Succeed;15
4.2;1.2 Theoretical Contributions to Neural Networks;16
4.2.1;1.2.1 Universal Approximation Properties;16
4.2.2;1.2.2 Vanishing and Exploding Gradients;17
4.2.3;1.2.3 Wasserstein GAN;18
4.3;1.3 Mathematical Representations;19
4.4;1.4 Book Layout;19
4.5;References;20
5;2 Mathematical Preliminaries;23
5.1;2.1 Linear Maps, Bilinear Maps, and Adjoints;24
5.2;2.2 Derivatives;25
5.2.1;2.2.1 First Derivatives;25
5.2.2;2.2.2 Second Derivatives;26
5.3;2.3 Parameter-Dependent Maps;27
5.3.1;2.3.1 First Derivatives;28
5.3.2;2.3.2 Higher-Order Derivatives;28
5.4;2.4 Elementwise Functions;29
5.4.1;2.4.1 Hadamard Product;30
5.4.2;2.4.2 Derivatives of Elementwise Functions;31
5.4.3;2.4.3 The Softmax and Elementwise Log Functions;32
5.5;2.5 Conclusion;34
5.6;References;34
6;3 Generic Representation of Neural Networks;35
6.1;3.1 Neural Network Formulation;36
6.2;3.2 Loss Functions and Gradient Descent;37
6.2.1;3.2.1 Regression;37
6.2.2;3.2.2 Classification;38
6.2.3;3.2.3 Backpropagation;39
6.2.4;3.2.4 Gradient Descent Step Algorithm;40
6.3;3.3 Higher-Order Loss Function;41
6.3.1;3.3.1 Gradient Descent Step Algorithm;44
6.4;3.4 Conclusion;45
6.5;References;46
7;4 Specific Network Descriptions;47
7.1;4.1 Multilayer Perceptron;48
7.1.1;4.1.1 Formulation;48
7.1.2;4.1.2 Single-Layer Derivatives;49
7.1.3;4.1.3 Loss Functions and Gradient Descent;50
7.2;4.2 Convolutional Neural Networks;52
7.2.1;4.2.1 Single Layer Formulation;52
7.2.1.1;Cropping and Embedding Operators;53
7.2.1.2;Convolution Operator;55
7.2.1.3;Max-Pooling Operator;58
7.2.1.4;The Layerwise Function;61
7.2.2;4.2.2 Multiple Layers;62
7.2.3;4.2.3 Single-Layer Derivatives;62
7.2.4;4.2.4 Gradient Descent Step Algorithm;63
7.3;4.3 Deep Auto-Encoder;64
7.3.1;4.3.1 Weight Sharing;64
7.3.2;4.3.2 Single-Layer Formulation;65
7.3.3;4.3.3 Single-Layer Derivatives;66
7.3.4;4.3.4 Loss Functions and Gradient Descent;67
7.4;4.4 Conclusion;69
7.5;References;70
8;5 Recurrent Neural Networks;71
8.1;5.1 Generic RNN Formulation;71
8.1.1;5.1.1 Sequence Data;72
8.1.2;5.1.2 Hidden States, Parameters, and Forward Propagation;72
8.1.3;5.1.3 Prediction and Loss Functions;74
8.1.4;5.1.4 Loss Function Gradients;74
8.1.4.1;Prediction Parameters;75
8.1.4.2;Real-Time Recurrent Learning;76
8.1.4.3;Backpropagation Through Time;77
8.2;5.2 Vanilla RNNs;82
8.2.1;5.2.1 Formulation;82
8.2.2;5.2.2 Single-Layer Derivatives;83
8.2.3;5.2.3 Backpropagation Through Time;84
8.2.4;5.2.4 Real-Time Recurrent Learning;86
8.2.4.1;Evolution Equation;86
8.2.4.2;Loss Function Derivatives;87
8.2.4.3;Gradient Descent Step Algorithm;88
8.3;5.3 RNN Variants;88
8.3.1;5.3.1 Gated RNNs;89
8.3.2;5.3.2 Bidirectional RNNs;90
8.3.3;5.3.3 Deep RNNs;90
8.4;5.4 Conclusion;90
8.5;References;91
9;6 Conclusion and Future Work;92
9.1;References;93
10;Glossary;94