Google

In a prefix-tuned model, the complete hidden state for layer $$l+1$$, denoted as $$\mathbf{H}^{l+1}$$, is formed by concatenating the layer-specific trainable prefix vectors with the processed hidden states of the original input sequence. This composition is represented by the formula: $$ \mathbf{H}^{l+1} = \mathbf{p}_0^{l+1} \mathbf{p}_1^{l+1} \dots \mathbf{p}_n^{l+1} \overline{\mathbf{H}}^{l+1} $$ where $$\overline{\mathbf{H}}^{l+1}$$ is the sequence of output hidden states corresponding to the original input, which can be further expanded as: $$ \mathbf{H}^{l+1} = \mathbf{p}_0^{l+1} \mathbf{p}_1^{l+1} \dots \mathbf{p}_n^{l+1} \mathbf{h}_0^{l+1} \mathbf{h}_1^{l+1} \dots \mathbf{h}_m^{l+1} $$

Next-Layer Input Composition Formula in Prefix-Tuning

Based on the standard procedure for this tuning method, what is the fundamental error in the engineer's observed composition of the hidden state sequence? Explain the correct composition and the reasoning behind it.

Analyzing an Incorrect Hidden State Composition

In a transformer model adapted with prefix vectors, consider the input to layer `l+1`. The prefix for this layer consists of 10 trainable vectors. The hidden states corresponding to the original text input, which were processed by the previous layer, form a sequence of 512 vectors. Each vector in the model has a dimension of 768. Describe the structure of the complete hidden state sequence that is fed into the self-attention mechanism of layer `l+1`, and state its final dimensions.

Constructing the Input Hidden State for a Prefix-Tuned Layer

In a specific parameter-efficient tuning method, each layer of a transformer is adapted by prepending a sequence of new, trainable vectors to the sequence of hidden states from the previous layer. Suppose for a given layer, the sequence of these new trainable vectors has a length of 20, and the sequence of hidden states corresponding to the original text input has a length of 128. After this layer processes the combined sequence, a new set of hidden states is generated. How is the complete hidde

Learn Before

Related