Google


The main task of these models is to predict the next word based on previous words, and they are represented by the GPT model family. The AR infrastructures are composed of the Transformer’s decoder part, so a masking mechanism is used in the training phase, causing attention calculations to only see the content before a word. Applications are suitable for NLG tasks. 


Auto-Regressive (AR) Models 

Auto-regressive language models calculate the probability of a text sequence, $\mathbf{x}$, by decomposing it into a product of conditional probabilities using the chain rule. The probability of each token $x_i$ is conditioned on all preceding tokens in the sequence. The general formula for a sequence $\mathbf{x} = (x_0, ..., x_{m-1})$ is: $$ \text{Pr}(\mathbf{x}) = \prod_{i=0}^{m-1} \text{Pr}(x_i | x_0, ..., x_{i-1}) $$ For example, for a sequence of five tokens, this expands to: $$ \text{Pr}(x) = \text{Pr}(x_0) \cdot \text{Pr}(x_1|x_0) \cdot \text{Pr}(x_2|x_0, x_1) \cdot \text{Pr}(x_3|x_0, x_1, x_2) \cdot \text{Pr}(x_4|x_0, x_1, x_2, x_3) $$

Chain Rule of Probability for Auto-regressive Language Models

Permuted Language Modeling (PLM) is a pre-training approach designed to resolve specific issues found in Masked Language Modeling, such as the mismatch between training and inference and the independence assumption among masked tokens. While it is a sequential prediction task, the actual order of tokens in the original text remains completely unchanged. Instead, the model is trained to predict the tokens sequentially according to an arbitrarily determined, permuted order.

Permuted Language Modeling (PLM)

Permuted Language Modeling is a training objective that builds upon the principles of Masked Language Modeling by incorporating the order of token prediction. The method involves shuffling the input sequence into a new order and then training the model to predict the tokens sequentially according to this permuted arrangement. For each step in the prediction process, the model uses a randomly chosen subset of other tokens from the sequence as its context.

Permuted Language Modeling

Based on the scenario provided, evaluate the suitability of using a purely auto-regressive model architecture for the described task. Justify your reasoning.

Model Architecture Suitability for Sentiment Analysis

Explain why the core operational principle of an auto-regressive model, which involves predicting the next token based only on preceding tokens, is fundamentally well-suited for tasks involving text generation.

Rationale for Auto-Regressive Model Design in Text Generation

A language model is being trained on the sentence: 'The quick brown fox jumps over the lazy dog.' The model's primary purpose is to generate new text by predicting the next word in a sequence based only on the words that came before it. When the model is calculating the representation for the word 'jumps' during this process, which part of the sentence is it allowed to consider?

Learn Before

Related