Google

An objective function, denoted as $$U(\mathbf{x}, \mathbf{y}; \theta)$$, can be formulated to guide the training of a sequence generation model, often in the context of policy optimization. It is calculated by summing the weighted log-probabilities of the policy over each step of the generated sequence. The formula is $$U(\mathbf{x}, \mathbf{y}; \theta) = \sum_{t=1}^{T} A(\mathbf{x}, y_t, \mathbf{y}_{<t}) \log \pi_\theta(y_t|\mathbf{x}, \mathbf{y}_{<t})$$, where $$\pi_\theta$$ is the model's policy (a probability distribution over outputs) and $$A(\cdot)$$ is a function that assigns a weight or advantage to each step.

Objective Function for Sequence Generation Policy Optimization

Consider the objective function $U(\mathbf{x}, \mathbf{y}; \theta) = \sum_{t=1}^{T} A(\mathbf{x}, y_t, \mathbf{y}_{<t}) \log \pi_\theta(y_t|\mathbf{x}, \mathbf{y}_{<t})$ used for training a sequence generation model. Analyze what this objective function simplifies to if the weighting function $A(\cdot)$ is set to a constant value of 1 for all steps $t$. How does this simplified objective relate to the standard training objective of a typical autoregressive language model?

Analysis of a Simplified Objective Function

A language model is being trained to generate factual summaries of news articles. The training process aims to maximize the objective function $U(\mathbf{x}, \mathbf{y}; \theta) = \sum_{t=1}^{T} A(\mathbf{x}, y_t, \mathbf{y}_{<t}) \log \pi_\theta(y_t|\mathbf{x}, \mathbf{y}_{<t})$. The weighting function $A(\cdot)$ is designed to assign a large negative value for any generated statement that is factually incorrect relative to the source article, and a small, constant positive value for each correct statement. After training, the model consistently produces overly cautious and brief summaries, such as 'The article discusses a topic,' instead of detailed, informative ones. Analyze why the model might be exhibiting this behavior, specifically explaining how the design of the weighting function $A(\cdot)$ interacts with the overall objective function to produce this outcome.

Debugging Model Behavior via the Objective Function

A sequence generation model is being trained to maximize the objective function $U(\mathbf{x}, \mathbf{y}; \theta) = \sum_{t=1}^{T} A(\mathbf{x}, y_t, \mathbf{y}_{<t}) \log \pi_\theta(y_t|\mathbf{x}, \mathbf{y}_{<t})$. The training goal is to specifically penalize the model for using repetitive phrasing. Which of the following strategies for designing the weighting function $A(\cdot)$ would best accomplish this?

Learn Before

Related