Exponential linear unit (ELU), like Leaky ReLU, has a small slope for negative values. But instead of having a straight line, it has a log curve, as shown in the following diagram:
It can be expressed as follows:
The ELU function is implemented in Python as follows:
def ELU(x,alpha=0.01):
if x<0:
return ((alpha*(np.exp(x)-1))
else:
return x