Weights and biases form an integral part of any deep neural network optimization and here we define a couple of functions to automate these initializations. It is a good practice to initialize weights with small noise to break symmetry and prevent zero gradients. Additionally, a small positive initial bias would avoid inactivated neurons, suitable for ReLU activation neurons.
Using functions to initialize weights and biases
by Achyutuni Sri Krishna Rao, Dr. PKS Prakash
R Deep Learning Cookbook
- Preface
- Getting Started
- Deep Learning with R
- Starting with logistic regression
- Introducing the dataset
- Performing logistic regression using H2O
- Performing logistic regression using TensorFlow
- Visualizing TensorFlow graphs
- Starting with multilayer perceptrons
- Setting up a neural network using H2O
- Tuning hyper-parameters using grid searches in H2O
- Setting up a neural network using MXNet
- Setting up a neural network using TensorFlow
- Convolution Neural Network
- Introduction
- Downloading and configuring an image dataset
- Learning the architecture of a CNN classifier
- Using functions to initialize weights and biases
- Using functions to create a new convolution layer
- Using functions to create a new convolution layer
- Using functions to flatten the densely connected layer
- Defining placeholder variables
- Creating the first convolution layer
- Creating the second convolution layer
- Flattening the second convolution layer
- Creating the first fully connected layer
- Applying dropout to the first fully connected layer
- Creating the second fully connected layer with dropout
- Applying softmax activation to obtain a predicted class
- Defining the cost function used for optimization
- Performing gradient descent cost optimization
- Executing the graph in a TensorFlow session
- Evaluating the performance on test data
- Data Representation Using Autoencoders
- Introduction
- Setting up autoencoders
- Data normalization
- Setting up a regularized autoencoder
- Fine-tuning the parameters of the autoencoder
- Setting up stacked autoencoders
- Setting up denoising autoencoders
- Building and comparing stochastic encoders and decoders
- Learning manifolds from autoencoders
- Evaluating the sparse decomposition
- Generative Models in Deep Learning
- Comparing principal component analysis with the Restricted Boltzmann machine
- Setting up a Restricted Boltzmann machine for Bernoulli distribution input
- Training a Restricted Boltzmann machine
- Backward or reconstruction phase of RBM
- Understanding the contrastive divergence of the reconstruction
- Initializing and starting a new TensorFlow session
- Evaluating the output from an RBM
- Setting up a Restricted Boltzmann machine for Collaborative Filtering
- Performing a full run of training an RBM
- Setting up a Deep Belief Network
- Implementing a feed-forward backpropagation Neural Network
- Setting up a Deep Restricted Boltzmann Machine
- Recurrent Neural Networks
- Reinforcement Learning
- Application of Deep Learning in Text Mining
- Application of Deep Learning to Signal processing
- Transfer Learning
Using functions to initialize weights and biases
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.