Google

For sequence-level classification tasks, a standard pipeline is often employed. An input sequence, prepared with a special $$[\mathrm{CLS}]$$ token at the beginning, is first processed by a Transformer encoder. This yields a sequence of hidden state vectors, $$\{\mathbf{h}_0, \dots, \mathbf{h}_m\}$$. The hidden state corresponding to the $$[\mathrm{CLS}]$$ token, $$\mathbf{h}_0$$, is then isolated, as it serves as an aggregate representation of the entire sequence's meaning. Finally, this single vector $$\mathbf{h}_0$$ is passed through a classification layer, such as Softmax, to produce the final output, for instance, in a binary classification system.

Sequence Classification Pipeline using the [CLS] Token Output

A data scientist is using a pre-trained Transformer model for a sentiment analysis task (classifying text as positive or negative). Arrange the following steps in the correct order to form the classification pipeline.

Analyze the following description of a text classification pipeline that is performing poorly. Identify the conceptual error in the described process and explain why the standard approach is preferred for this type of task.

Diagnosing a Flawed Classification Pipeline

An engineer is building a model to classify customer reviews as 'positive' or 'negative'. The input text is first prepared by adding a special classification token to the very beginning. This entire sequence is then processed by an encoder, which generates a final output vector for each token. To make the final classification decision for the entire review, which specific vector should be passed to the final classification layer?

Learn Before

Related