Title Page Copyright and Credits Hands-On Deep Learning Algorithms with Python Dedication About Packt Why subscribe? Contributors About the author About the reviewers Packt is searching for authors like you Preface Who this book is for What this book covers To get the most out of this book Download the example code files Download the color images Conventions used Get in touch Reviews Section 1: Getting Started with Deep Learning Introduction to Deep Learning What is deep learning? Biological and artificial neurons ANN and its layers Input layer Hidden layer Output layer Exploring activation functions The sigmoid function The tanh function The Rectified Linear Unit function The leaky ReLU function The Exponential linear unit function The Swish function The softmax function Forward propagation in ANN How does ANN learn? Debugging gradient descent with gradient checking Putting it all together Building a neural network from scratch Summary Questions Further reading Getting to Know TensorFlow What is TensorFlow? Understanding computational graphs and sessions Sessions Variables, constants, and placeholders Variables Constants Placeholders and feed dictionaries Introducing TensorBoard Creating a name scope Handwritten digit classification using TensorFlow Importing the required libraries Loading the dataset Defining the number of neurons in each layer Defining placeholders Forward propagation Computing loss and backpropagation Computing accuracy Creating summary Training the model Visualizing graphs in TensorBoard Introducing eager execution Math operations in TensorFlow TensorFlow 2.0 and Keras Bonjour Keras Defining the model Defining a sequential model Defining a functional model Compiling the model Training the model Evaluating the model MNIST digit classification using TensorFlow 2.0 Should we use Keras or TensorFlow? Summary Questions Further reading Section 2: Fundamental Deep Learning Algorithms Gradient Descent and Its Variants Demystifying gradient descent Performing gradient descent in regression Importing the libraries Preparing the dataset Defining the loss function Computing the gradients of the loss function Updating the model parameters Gradient descent versus stochastic gradient descent Momentum-based gradient descent Gradient descent with momentum Nesterov accelerated gradient Adaptive methods of gradient descent Setting a learning rate adaptively using Adagrad Doing away with the learning rate using Adadelta Overcoming the limitations of Adagrad using RMSProp Adaptive moment estimation Adamax – Adam based on infinity-norm Adaptive moment estimation with AMSGrad Nadam – adding NAG to ADAM Summary Questions Further reading Generating Song Lyrics Using RNN Introducing RNNs The difference between feedforward networks and RNNs Forward propagation in RNNs Backpropagating through time Gradients with respect to the hidden to output weight, V Gradients with respect to hidden to hidden layer weights, W Gradients with respect to input to the hidden layer weight, U Vanishing and exploding gradients problem Gradient clipping Generating song lyrics using RNNs Implementing in TensorFlow Data preparation Defining the network parameters Defining placeholders Defining forward propagation Defining BPTT Start generating songs Different types of RNN architectures One-to-one architecture One-to-many architecture Many-to-one architecture Many-to-many architecture Summary Questions Further reading Improvements to the RNN LSTM to the rescue Understanding the LSTM cell Forget gate Input gate Output gate Updating the cell state Updating hidden state Forward propagation in LSTM Backpropagation in LSTM Gradients with respect to gates Gradients with respect to weights Gradients with respect to V Gradients with respect to W Gradients with respect to U Predicting Bitcoin prices using LSTM model Data preparation Defining the parameters Define the LSTM cell Defining forward propagation Defining backpropagation Training the LSTM model Making predictions using the LSTM model Gated recurrent units Understanding the GRU cell Update gate Reset gate Updating hidden state Forward propagation in a GRU cell Backpropagation in a GRU cell Gradient with respect to gates Gradients with respect to weights Gradients with respect to V Gradients with respect to W Gradients with respect to U Implementing a GRU cell in TensorFlow Defining the weights Defining forward propagation Bidirectional RNN Going deep with deep RNN Language translation using the seq2seq model Encoder Decoder Attention is all we need Summary Questions Further reading Demystifying Convolutional Networks What are CNNs? Convolutional layers Strides Padding Pooling layers Fully connected layers The architecture of CNNs The math behind CNNs Forward propagation Backward propagation Implementing a CNN in TensorFlow Defining helper functions Defining the convolutional network Computing loss Starting the training Visualizing extracted features CNN architectures LeNet architecture Understanding AlexNet Architecture of VGGNet GoogleNet Inception v1 Inception v2 and v3 Capsule networks Understanding Capsule networks Computing prediction vectors Coupling coefficients Squashing function Dynamic routing algorithm Architecture of the Capsule network The loss function Margin loss Reconstruction loss Building Capsule networks in TensorFlow Defining the squash function Defining a dynamic routing algorithm Computing primary and digit capsules Masking the digit capsule Defining the decoder Computing the accuracy of the model Calculating loss Margin loss Reconstruction loss Total loss Training the Capsule network Summary Questions Further reading Learning Text Representations Understanding the word2vec model Understanding the CBOW model CBOW with a single context word Forward propagation Backward propagation CBOW with multiple context words Understanding skip-gram model Forward propagation in skip-gram Backward propagation Various training strategies Hierarchical softmax Negative sampling Subsampling frequent words Building the word2vec model using gensim Loading the dataset Preprocessing and preparing the dataset Building the model Evaluating the embeddings Visualizing word embeddings in TensorBoard Doc2vec Paragraph Vector – Distributed Memory model Paragraph Vector – Distributed Bag of Words model Finding similar documents using doc2vec Understanding skip-thoughts algorithm Quick-thoughts for sentence embeddings Summary Questions Further reading Section 3: Advanced Deep Learning Algorithms Generating Images Using GANs Differences between discriminative and generative models Say hello to GANs! Breaking down the generator Breaking down the discriminator How do they learn though? Architecture of a GAN Demystifying the loss function Discriminator loss First term Second term Final term Generator loss Total loss Heuristic loss Generating images using GANs in TensorFlow Reading the dataset Defining the generator Defining the discriminator Defining the input placeholders Starting the GAN! Computing the loss function Discriminator loss Generator loss Optimizing the loss Starting the training Generating handwritten digits DCGAN – Adding convolution to a GAN Deconvolutional generator Convolutional discriminator Implementing DCGAN to generate CIFAR images Exploring the dataset Defining the discriminator Defining the generator Defining the inputs Starting the DCGAN Computing the loss function Discriminator loss Generator loss Optimizing the loss Train the DCGAN Least squares GAN Loss function LSGAN in TensorFlow Discriminator loss Generator loss GANs with Wasserstein distance Are we minimizing JS divergence in GANs? What is the Wasserstein distance? Demystifying the k-Lipschitz function The loss function of WGAN WGAN in TensorFlow Summary Questions Further reading Learning More about GANs Conditional GANs Loss function of CGAN Generating specific handwritten digits using CGAN Defining the generator Defining discriminator Start the GAN! Computing the loss function Discriminator loss Generator loss Optimizing the loss Start training the CGAN Generate the handwritten digit, 7 Understanding InfoGAN Mutual information Architecture of the InfoGAN Constructing an InfoGAN in TensorFlow Defining generator Defining the discriminator Define the input placeholders Start the GAN Computing loss function Discriminator loss Generator loss Mutual information Optimizing the loss Beginning training Generating handwritten digits Translating images using a CycleGAN Role of generators Role of discriminators Loss function Cycle consistency loss Converting photos to paintings using a CycleGAN StackGAN The architecture of StackGANs Conditioning augmentation Stage I Generator Discriminator Stage II Generator Discriminator Summary Questions Further reading Reconstructing Inputs Using Autoencoders What is an autoencoder? Understanding the architecture of autoencoders Reconstructing the MNIST images using an autoencoder Preparing the dataset Defining the encoder Defining the decoder Building the model Reconstructing images Plotting reconstructed images Autoencoders with convolutions Building a convolutional autoencoder Defining the encoder Defining the decoder Building the model Reconstructing the images Exploring denoising autoencoders Denoising images using DAE Understanding sparse autoencoders Building the sparse autoencoder Defining the sparse regularizer Learning to use contractive autoencoders Implementing the contractive autoencoder Defining the contractive loss Dissecting variational autoencoders Variational inference The loss function Reparameterization trick Generating images using VAE Preparing the dataset Defining the encoder Defining the sampling operation Defining the decoder Building the model Defining the generator Plotting generated images Summary Questions Further reading Exploring Few-Shot Learning Algorithms What is few-shot learning? Siamese networks Architecture of siamese networks Prototypical networks Relation networks Matching networks Support set embedding function Query set embedding function The architecture of matching networks Summary Questions Further reading Assessments Chapter 1 - Introduction to Deep Learning Chapter 2 - Getting to Know TensorFlow Chapter 3 - Gradient Descent and Its Variants Chapter 4 - Generating Song Lyrics Using an RNN  Chapter 5 - Improvements to the RNN Chapter 6 - Demystifying Convolutional Networks Chapter 7 - Learning Text Representations Chapter 8 - Generating Images Using GANs Chapter 9 - Learning More about GANs Chapter 10 - Reconstructing Inputs Using Autoencoders Chapter 11 - Exploring Few-Shot Learning Algorithms Other Books You May Enjoy Leave a review - let other readers know what you think