Google

The T5 relative positional bias model is capable of generalizing to sequences longer than those encountered during training. This ability stems from its strategy of sharing the same learnable parameter across similar query-key offsets. Such parameter sharing is particularly effective because large offsets are rare in training data, allowing the model to apply learned biases to novel distances by grouping them with familiar ones.

Generalization Advantage of T5 Bias through Parameter Sharing

A language model that groups various query-key offsets and assigns a single shared parameter to each group often performs well on input sequences longer than any it saw during training. Explain the core reason why this parameter-sharing strategy enables the model to effectively handle these previously unseen, large offsets.

Generalization Through Parameter Sharing

Based on the provided case study, analyze the likely cause of the model's performance degradation on longer documents and explain how implementing a parameter-sharing strategy for relative positions could address the issue.

Diagnosing Generalization Failure in a Transformer Model

A language model was trained exclusively on text segments with a maximum length of 512 tokens. During inference, it must process a 1000-token document, encountering a query-key offset of 700 for the first time. Why is a model architecture that groups offsets into 'buckets' and shares a single learnable parameter per bucket better equipped to handle this novel offset than a hypothetical model that learns a unique, separate parameter for every individual offset?

Learn Before

Related