Google

In the T5 model, the attention score calculation deviates from the standard scaled dot-product attention by omitting the rescaling operation. By substituting the relative position encoding $$\mathrm{PE}(i,j) = u_{b(i-j)}$$ into the base attention formula, a shared, learnable scalar bias $$u_{b(i-j)}$$ is added directly to the unscaled query-key dot product for vectors $$\mathbf{q}_i$$ and $$\mathbf{k}_j$$. The resulting formula for the attention weight is: $$\alpha(i, j) = \text{Softmax}\left(\mathbf{q}_i \mathbf{k}_j^T + u_{b(i-j)} + \text{Mask}(i, j)\right)$$ This modification, as specified by Raffel et al. (2020), removes the division by the square root of the key dimension, $$\sqrt{d}$$.

Formula for Attention with T5 Bias (Unscaled)

A standard attention mechanism calculates the unnormalized score between a query `q_i` and a key `k_j` using the expression `(q_i ⋅ k_j) / sqrt(d_k)`, where `d_k` is the dimension of the key vectors. In contrast, the T5 model's approach uses the expression `q_i ⋅ k_j + u_{b(i-j)}`, where `u` is a learnable scalar bias dependent on the relative positions of `i` and `j`. Identify the two primary modifications in the T5 expression compared to the standard one, and explain the functional role of each change within the attention mechanism.

Analysis of T5 Attention Formula Modifications

The standard scaled dot-product attention formula includes a scaling factor of `1/sqrt(d_k)` applied to the query-key dot product to counteract the effect of large dot product magnitudes. The T5 model's attention mechanism, however, omits this scaling factor. Evaluate the potential motivations and consequences of this design choice. In your answer, discuss how the absence of this scaling factor might impact model training stability and performance, and how other components of the T5 architecture, such as the learnable relative position bias, might interact with or compensate for this change.

Evaluating the Design of T5's Unscaled Attention Mechanism

A developer is implementing an attention layer for a model that incorporates positional information by adding a learnable scalar bias based on the relative distance between tokens. Given a query vector `q_i` for a token at position `i`, a key vector `k_j` for a token at position `j`, a key dimension `d_k`, and the specific learnable bias `u_{b(i-j)}` for their relative position, which of the following expressions correctly computes the unnormalized attention score (the value passed into the soft

Learn Before

Related