"GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performance"

"Our findings reinforce the exceptional performance of the GELU activation function, which attains the highest test accuracy and lowest test loss among the activation functions investigated. Other activation functions, such as Hardswish and ReLU6, exhibit commendable performance as well..."

#GELU #ReLU #HardShrink #leakyReLU #ReLU6

🔗https://arxiv.org/pdf/2305.12073v1.pdf