University of Michigan - Ann Arbor

In a Transformer architecture, the feed-forward network is called positionwise because it applies the identical Multi-Layer Perceptron (MLP) to transform the representation at every sequence position independently. For an input tensor $$X$$ with the shape (batch size, number of time steps, number of hidden units), this two-layer MLP processes each time step's vector in isolation. Consequently, only the innermost dimension is transformed, resulting in an output tensor of shape (batch size, number of time steps, $$	ext{ffn\_num\_outputs}$$). Because the exact same MLP transforms all positions, identical inputs at different positions will produce identical outputs.

Positionwise Nature of Transformer Feed-Forward Networks

Match each architectural concept of the Transformer feed-forward layer with its correct functional description.

Determine the shape of the output tensor produced by this feed-forward network, identify which specific dimension of tensor X is transformed, and explain why the remaining dimensions remain unchanged.

Learn Before

Related