Google

The SwiGLU function is mathematically defined by utilizing the Swish function, $$\sigma_{\mathrm{swish}}$$, as its internal non-linear activation. The formula is expressed as:

$$\sigma_{\mathrm{swiglu}}(\mathbf{h}) = \sigma_{\mathrm{swish}}(\mathbf{h}\mathbf{W}_1 + \mathbf{b}_1) \odot (\mathbf{W}_2 + \mathbf{b}_2)$$

where $$\mathbf{h}$$ is the input vector and $$\odot$$ indicates the element-wise product.

SwiGLU (Swish-based Gated Linear Unit) Formula

Consider a neural network component that processes an input vector `h` using two separate linear transformations, `(hW₁ + b₁)` and `(hW₂ + b₂)`. The final output is computed by taking the element-wise product of the results of these two transformations. This structure is missing a key non-linear operation that is characteristic of a Swish-based Gated Linear Unit. Identify this missing operation and explain its role and placement within the computation.

Analysis of a Gated Activation Function

The SwiGLU activation function is defined by the formula: `σ_swish(hW₁ + b₁) ⊙ (hW₂ + b₂)`. Match each component of this formula to its primary role in the computation.

Consider a simplified SwiGLU activation function where the input vector `h` is `[2, 1]`. The learnable parameters are defined as follows:

*   `W1 = [[3], [1]]`, `b1 = [0]`
*   `W2 = [[2], [-1]]`, `b2 = [1]`
*   The Swish activation function is defined as `swish(x) = x * sigmoid(x)`.
*   Assume `sigmoid(7) ≈ 0.999`.

Given the formula `output = swish(hW1 + b1) ⊙ (hW2 + b2)`, where `⊙` is the element-wise product, calculate the output. Which of the following is the correct result?

Learn Before

Related