Google

- Encoder-Decoder: sequence to sequence (language modeling)
- Encoder Only: outputs of the encoder are utilized as a representation for the input sequence. This is usually used for classification or sequence labeling problems (i.e. BERT)
- Decoder Only: cross-attention module is removed; this is typically used for sequence generation, such as language modeling (i.e. GPT)

Model Usage of Transformers

The decoder-only Transformer architecture is a prevalent design for Large Language Models (LLMs). It is typically created by modifying a standard Transformer decoder, specifically by eliminating the cross-attention sub-layers. The central components of this architecture are $$L$$ stacked Transformer blocks, each comprising a self-attention sub-layer and a feed-forward network (FFN) sub-layer. To prevent the model from accessing the right-context, a masking variable is incorporated into the self-attention mechanism. Finally, the output layer uses a Softmax function to generate a probability distribution for the next token, given the sequence of previous tokens, enabling auto-regressive generation.

Decoder-Only Transformer as a Language Model

An engineering team is tasked with building a system to perform sentiment analysis on customer reviews. The goal is to classify each review as 'positive', 'negative', or 'neutral'. For an accurate classification, the model must be able to understand the full context of the entire review, including how words at the end of a sentence can influence the meaning of words at the beginning. Which of the following architectural approaches is best suited for this specific task?

Based on the following scenario, which architectural pattern is most suitable for the described task, and why is the other main component of a full sequence-to-sequence model unnecessary for the core generative function?

Architectural Design for a Creative Writing Assistant

A development team is building a system to generate concise summaries of long articles. They are considering two architectural approaches: an encoder-only model that would be trained to classify the importance of each sentence in the original text, versus a full encoder-decoder model that would be trained to generate a new, abstractive summary. Evaluate the strengths and weaknesses of each approach for this specific task. In your answer, justify which architecture you would recommend and why, considering how each model processes the input and produces the final summary.

Architectural Choice for Document Summarization

You are a machine learning engineer evaluating different model architectures for three distinct natural language processing projects. Match each project description with the most suitable architectural approach based on its core requirements.

Learn Before

Related