University of Michigan - Ann Arbor

The Transformer offers several computational and performance advantages over recurrent architectures:

- **Constant sequential operations**: A self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent neural network (RNN) requires $$O(n)$$ sequential operations. In terms of computational complexity, self-attention layers are faster than recurrent layers whenever the sequence length is smaller than the representation dimensionality.
- **Faster training**: For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers.
- **State-of-the-art translation results**: On both the WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, the Transformer achieved a new state-of-the-art. On the English-to-German task, the best model outperformed even all previously reported ensembles.

Advantages and Performance of the Transformer Model

Based on the performance evaluation of the Transformer, explain how its training speed compares to recurrent or convolutional architectures for translation tasks, and describe the achievement of the best Transformer model relative to prior ensemble models on the WMT 2014 English-to-German task.

Learn Before

Related