Google

The policy optimization objective can be mathematically simplified to finding the parameters $$\tilde{\theta}$$ that minimize the expected Kullback-Leibler (KL) divergence between the learned target policy $$\pi_{\theta}$$ and the optimal target distribution $$\pi^*$$. This simplification is mathematically sound because the normalization term, $$\log Z(\mathbf{x})$$, is independent of the optimization variable $$\theta$$ and can therefore be removed from the $$\argmin_{\theta}$$ operation without altering the optimal parameters. The simplified training objective is expressed as:

$$ \tilde{\theta} = \argmin_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \Big[ \mathrm{KL} \big(\pi_{\theta}(\cdot|\mathbf{x})\ ||\ \pi^{*}(\cdot|\mathbf{x}) \big) \Big] $$

Simplified Policy Optimization Objective as KL Divergence Minimization

The optimization problem of minimizing the Kullback-Leibler (KL) divergence between a learned policy $$\pi_{\theta}$$ and an optimal policy $$\pi^*$$ is uniquely solved when the two probability distributions are identical. Thus, the optimal target policy is defined by equating it to $$\pi^*$$, which incorporates the reward-weighted reference policy and the normalization factor. This relationship is formally given by:

$$ \pi_{\theta}(\mathbf{y}|\mathbf{x}) = \pi^{*}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\theta_{\mathrm{ref}}}(\mathbf{y}|\mathbf{x}) \exp \big(\frac{1}{\beta} r(\mathbf{x},\mathbf{y}) \big)}{Z(\mathbf{x})} $$

Solution to KL Divergence Minimization for Policy Optimization

Evaluate the colleague's suggestion. If the `log Z(x)` term is removed, how will this change affect (a) the final optimal parameters `θ` found after training, and (b) the overall efficiency of the training process? Justify your reasoning.

Efficiency in Policy Optimization Implementation

A colleague is working on an optimization problem with the objective function `arg min_θ Eₓ[ f(θ, x) + C(x) ]`, where `θ` are the parameters to be optimized. They simplify the objective to `arg min_θ Eₓ[ f(θ, x) ]` and justify it by stating, 'The term `C(x)` is a constant, so it can be ignored.' Evaluate this justification. Is it entirely correct? If not, refine the statement to be more precise.

Justification for Simplification in Policy Optimization

When optimizing a policy π_θ to match an optimal policy π*, the objective function is often simplified from Objective A to Objective B:

**Objective A:** `arg min_θ Eₓ[KL(π_θ(·|x) || π*(·|x)) - log Z(x)]`
**Objective B:** `arg min_θ Eₓ[KL(π_θ(·|x) || π*(·|x))]`

What is the fundamental mathematical reason this simplification is valid?

Learn Before

Related