University of Michigan - Ann Arbor

The Transformer can use either fixed sinusoidal positional encodings or learned positional embeddings. On the newstest2013 English-to-German development set, the learned variant nearly matched the base sinusoidal model: both had a perplexity of 4.92, while BLEU was 25.7 for learned embeddings and 25.8 for the base model. The sinusoidal variant was retained because its fixed functions may extrapolate to sequence lengths beyond those encountered during training.

Transformer Positional Encoding: Sinusoidal versus Learned Embeddings

Explain the sequence length extrapolation capability of fixed sinusoidal positional encodings in the Transformer architecture. In your response, define what this capability entails and explain why it served as the deciding factor in selecting sinusoidal encodings over learned positional embeddings.

When evaluating learned positional embeddings against the base sinusoidal model on the newstest2013 English-to-German development set, what was the empirical outcome?

Learn Before

Related