Google

The Sandwich method, introduced by Chi et al. in 2023, is a technique related to positional bias mechanisms in transformer models.

Sandwich Method (Chi et al., 2023)

An engineer is designing a component for a sequence-processing model. They are considering two approaches for incorporating information about the relative positions of elements in a sequence. 

**Approach A:** A positional bias is added to the attention scores *before* the main attention calculation is completed. 

**Approach B:** A positional bias is added *before* the self-attention operation, and a second, distinct positional bias is added *after* the self-attention operation has concluded, effectively 'sandwiching' the core mechanism.

Analyze the potential rationale behind Approach B. Why might adding a positional bias both before and after the self-attention operation be more effective for preserving positional information across multiple layers of the model compared to adding it only before?

Analyzing a Positional Bias Mechanism

The positional bias mechanism introduced by Chi et al. in 2023 applies its bias within every self-attention layer of a transformer model to ensure consistent spatial awareness is maintained throughout the entire network stack.

The Sandwich method calculates query-key positional bias as a sum of cosine functions based on the relative distance between a query at position $$i$$ and a key at position $$j$$. The formula is defined as: $$\mathrm{PE}(i,j) = \sum_{k=1}^{\bar{d}/2} \cos\big((i - j)/10000^{2k/\bar{d}} \big)$$, where $$\bar{d}$$ is a hyperparameter.

Sandwich Positional Bias Formula

A researcher is implementing a positional bias mechanism for a multi-layer transformer model, as introduced by Chi et al. in 2023. The goal is to influence the attention scores based on the relative positions of tokens. Given the specific design of this method, which of the following implementation strategies is correct?

Learn Before

Related