Google

Vanilla transformers use ReLU activation. Other functions used as activation functions include:

- Swish function -> f(x) = *x*sigmoid($\beta$ x)
- Gaussian Error Linear Unit (GELU)
- Gated Linear Units (GLU)

Activation function of the FFN in transformers

The Gaussian Error Linear Unit (GeLU) is a prominent alternative to the ReLU activation function in Large Language Models (LLMs), effectively acting as a smoothed version of it. Instead of gating outputs strictly by the sign of the input, the GeLU function operates by weighting its input using the percentile $$\Pr(h \le \mathbf{h})$$. In this formulation, $$h$$ represents a $$d$$-dimensional vector where each entry is sampled from the standard normal distribution, denoted as $$\mathrm{Gaussian}(0,1)$$, producing a vector of percentiles corresponding to the elements of the input $$\mathbf{h}$$.

Gaussian Error Linear Unit (GELU)

The Gated Linear Unit (GLU) is a family of activation functions that has gained popularity for its use in Large Language Models (LLMs). The specific variant of a GLU is defined by its internal non-linear activation function, denoted as `σ(·)`. For example, using the GELU function for `σ(·)` results in GeGLU, and using the Swish function results in SwiGLU.

Gated Linear Unit (GLU)

Match each activation function, which can be used in the feed-forward network of a transformer model, with its corresponding description.

Based on the provided scenario, evaluate the engineer's proposed solution. Justify your evaluation by comparing the properties of the original activation function with the proposed one and explaining how the change could address the observed problem.

Evaluating an Activation Function Change in a Transformer FFN

A machine learning engineer is analyzing the feed-forward network (FFN) component of a transformer model. They want to replace the standard Rectified Linear Unit (ReLU) activation function with a more modern alternative to potentially improve model performance. Which of the following statements best analyzes the rationale for using a function like the Gaussian Error Linear Unit (GELU) or Swish over ReLU in this context?

Learn Before

Related