Like sparse autoencoders, contractive autoencoders add a new regularization term to the loss function of the autoencoders. They try to make our encodings less sensitive to the small variations in the training data. So, with contractive autoencoders, our encodings become more robust to small perturbations such as noise present in our training dataset. We now introduce a new term called the regularizer or penalty term to our loss function. It helps to penalize the representations that are too sensitive to the input.

Our loss function can be mathematically represented as follows:

The first term represents the reconstruction error and the second term represents the penalty term or the regularizer and it is basically the Frobenius norm of the Jacobian matrix. Wait! What does that mean?

The Frobenius norm, also called the Hilbert-Schmidt norm, of a matrix is defined as the square root of the sum of the absolute square of its elements. A matrix comprising a partial derivative of the vector-valued function is called the Jacobian matrix.

Thus, calculating the Frobenius norm of the Jacobian matrix implies our penalty term is the sum of squares of all partial derivatives of the hidden layer with respect to the input. It is given as follows:

Calculating the partial derivative of the hidden layer with respect to the input is similar to calculating gradients of loss. Assuming we are using the sigmoid activation function, then the partial derivative of the hidden layer with respect to the input is given as follows:

Adding the penalty term to our loss function helps in reducing the sensitivity of the model to the variations in the input and makes our model more robust to the outliers. Thus, contractive autoencoders reduce the sensitivity of the model to the small variations in the training data.