ACTIVATION FUNCTION USED IN ANN

🧠 Activation Functions Used in ANN

Click the individual button to learn about each activation function

1️⃣ Binary Step

f(x) = 1, x ≥ 0
f(x) = 0, x < 0

Range: {0,1}

📌 Use Simple binary decisions and classical perceptrons.

✅ Advantages • Very simple.
• Fast computation.
• Produces a clear binary output.

❌ Disadvantages • Not differentiable at zero.
• Gradient is zero almost everywhere.
• Not suitable for modern backpropagation.

🎯 Best Used For Basic binary decision demonstrations and historical perceptron models.

2️⃣ Linear

f(x) = x

Range: −∞ to +∞

📌 Use Commonly used in regression output layers.

✅ Advantages • Extremely simple.
• Fast computation.
• Suitable for continuous unrestricted output.

❌ Disadvantages • Does not introduce non-linearity.
• Cannot model complex nonlinear relationships by itself.

🎯 Best Used For Regression output layers.

3️⃣ Sigmoid

f(x) = 1 / (1 + e^−x)

Range: 0 to 1

📌 Use Commonly used in the output layer for binary classification.

✅ Advantages • Output is between 0 and 1.
• Smooth and differentiable.
• Easy to interpret as a probability-like output.

❌ Disadvantages • Vanishing-gradient problem at extreme inputs.
• Not zero-centered.
• Can slow training in deep networks.

🎯 Best Used For Binary classification output.

4️⃣ Tanh

f(x) = (e^x − e^−x) / (e^x + e^−x)

Range: −1 to +1

📌 Use Historically common in hidden layers and still useful when zero-centered bounded activations are desired.

✅ Advantages • Zero-centered.
• Smooth function.
• Stronger gradients near zero than sigmoid.

❌ Disadvantages • Can suffer from vanishing gradients.
• Saturates at large positive and negative inputs.

🎯 Best Used For Some hidden-layer architectures and recurrent networks.

5️⃣ ReLU

f(x) = max(0,x)

Range: 0 to +∞

📌 Use Widely used in hidden layers of deep neural networks.

✅ Advantages • Very fast.
• Simple calculation.
• Helps reduce vanishing-gradient problems for positive inputs.
• Produces sparse activations.

❌ Disadvantages • Dying ReLU problem.
• Zero gradient for negative inputs.
• Unbounded positive output.

🎯 Best Used For General-purpose hidden layers.

6️⃣ Leaky ReLU

f(x) = x, x ≥ 0
f(x) = αx, x < 0

Range: −∞ to +∞

📌 Use Used as an alternative to ReLU to maintain a small gradient for negative inputs.

✅ Advantages • Reduces dying-ReLU risk.
• Keeps a small negative gradient.
• Computationally inexpensive.

❌ Disadvantages • Requires choosing α.
• Not guaranteed to outperform ReLU.

🎯 Best Used For Hidden layers when dead neurons are a concern.

7️⃣ PReLU

f(x) = max(αx,x)

Range: −∞ to +∞

📌 Use Used when the negative slope should be learned during training.

✅ Advantages • Learnable negative slope.
• Reduces dying neurons.
• More flexible than fixed Leaky ReLU.

❌ Disadvantages • Adds trainable parameters.
• Increases model complexity.

🎯 Best Used For Deep networks where a learnable negative slope is beneficial.

8️⃣ ELU

f(x) = x, x ≥ 0
f(x) = α(e^x−1), x < 0

Range: −α to +∞

📌 Use Used as a smooth alternative to ReLU.

✅ Advantages • Smooth for negative values.
• Allows negative outputs.
• Can improve optimization in some networks.

❌ Disadvantages • Exponential calculation is more expensive.
• Requires α.

🎯 Best Used For Hidden layers where smooth negative activation is desired.

9️⃣ SELU

f(x) = λx, x > 0
f(x) = λα(e^x−1), x ≤ 0

Self-Normalizing

📌 Use Designed for self-normalizing feed-forward neural networks under appropriate architectural conditions.

✅ Advantages • Can help maintain activation statistics.
• Supports self-normalizing behavior in suitable networks.

❌ Disadvantages • Requires specific conditions to obtain its intended behavior.
• Not ideal for every architecture.

🎯 Best Used For Self-normalizing feed-forward networks.

🔟 Softmax

f(zᵢ) = e^zᵢ / Σ e^zⱼ

Outputs sum to 1

📌 Use Commonly used in the output layer for multi-class classification.

✅ Advantages • Produces a distribution across classes.
• Outputs sum to 1.
• Easy to interpret for mutually exclusive classes.

❌ Disadvantages • Generally not used in hidden layers.
• Can produce overconfident probabilities.
• Requires numerically stable implementation.

🎯 Best Used For Multi-class classification output.

1️⃣1️⃣ Softplus

f(x) = ln(1 + e^x)

Range: 0 to +∞

📌 Use Smooth alternative to ReLU.

✅ Advantages • Smooth everywhere.
• Differentiable everywhere.
• No sharp corner at zero.

❌ Disadvantages • More computationally expensive than ReLU.
• Small gradients for strongly negative inputs.

🎯 Best Used For Networks requiring a smooth ReLU-like activation.

1️⃣2️⃣ Swish

f(x) = x · sigmoid(x)

Smooth / Non-monotonic

📌 Use Used in some modern deep neural networks as an alternative to ReLU.

✅ Advantages • Smooth.
• Non-monotonic.
• Can perform well in some deep architectures.

❌ Disadvantages • More computationally expensive than ReLU.
• Performance depends on the architecture and task.

🎯 Best Used For Modern deep-learning architectures.

1️⃣3️⃣ Mish

f(x) = x · tanh(ln(1+e^x))

Smooth / Non-monotonic

📌 Use Used as a smooth non-monotonic activation in some deep-learning models.

✅ Advantages • Smooth.
• Differentiable.
• Non-monotonic behavior.

❌ Disadvantages • Computationally more expensive.
• More complex than ReLU.

🎯 Best Used For Some modern deep neural networks.

1️⃣4️⃣ GELU

f(x) = xΦ(x)

Smooth Activation

📌 Use Widely used in transformer-based and other modern neural networks.

✅ Advantages • Smooth.
• Useful gating behavior.
• Effective in many modern architectures.

❌ Disadvantages • More complex than ReLU.
• Usually unnecessary for simple ANN problems.

🎯 Best Used For Transformers and modern deep-learning architectures.

1️⃣5️⃣ Hard Sigmoid

f(x) = clip((x+1)/2,0,1)

Range: 0 to 1

📌 Use Used as a computationally cheaper approximation of sigmoid.

✅ Advantages • Simple computation.
• Faster than true sigmoid.
• Useful in lightweight models.

❌ Disadvantages • Piecewise linear approximation.
• Zero gradient outside its active region.

🎯 Best Used For Resource-constrained neural networks.

1️⃣6️⃣ Hard Swish

f(x) = x · ReLU6(x+3)/6

Efficient Activation

📌 Use Designed as an efficient approximation to Swish for lightweight deep-learning models.

✅ Advantages • Computationally efficient.
• Useful in mobile and embedded networks.
• Approximates Swish behavior.

❌ Disadvantages • Less smooth than Swish.
• May not improve performance on simple ANN tasks.

🎯 Best Used For Mobile and resource-constrained neural networks.