The procedure used to carry out the learning process in a neural network is called the training algorithm. The learning algorithm is what the machine learning algorithm chooses as model with the best optimization. The aim is to minimize the loss function and provide more accuracy. Here we illustrate some of the optimization techniques, other than gradient descent.

The Particle Swarm Optimization (PSO) method is inspired by observations of social and collective behavior on the movements of bird flocks in search of food or survival. It is similar to a fish school trying to move together. We know the position and velocity of the particles, and PSO aims at searching a solution set in a large space controlled by mathematical equations on position and velocity. It is bio-inspired from biological organism behavior for collective intelligence.

Simulated annealing is a method that works on a probabilistic approach to approximate the global optimum for the cost function. The method searches for a solution in large space with simulation.

Evolutionary methods are derived from the evolutionary process in biology, and
evolution can be in terms of reproduction, mutation, selection, and recombination.
A fitness function is used to determine the performance of a model, and based on this
function, we select our final model.

The Expectation Maximization (EM) method is a statistical learning method that uses an iterative method to find maximum likelihood or maximum posterior estimate, thus minimizing the error.