Google

The standard delta rule for gradient descent updates a model's parameters by taking a small step in the direction of the negative loss gradient. The new parameters $$\theta_{t+1}$$ are obtained according to the formula: $$\theta_{t+1} = \theta_t - lr \cdot \frac{\partial L_{\theta_t}(\mathcal{D}_{\mathrm{mini}})}{\partial \theta_t}$$. In this equation, $$\theta_t$$ represents the latest parameters, $$lr$$ is the small step (learning rate), and the fractional term is the gradient of the loss function $$L$$ with respect to $$\theta_t$$, computed on a minibatch of training sample $$\mathcal{D}_{\mathrm{mini}}$$.

Gradient Descent Update Rule

Based on the standard parameter update rule used in optimization, which component is the most likely cause of this unstable training behavior, and why?

Diagnosing Training Instability

A model is being trained using an optimization algorithm. At a particular step, a specific parameter has a value of 2.5. The learning rate is set to 0.1, and the computed gradient of the loss function with respect to this parameter is 4.0. Calculate the new value of this parameter after one update step.

Calculating a Parameter Update

The main training loop for a deep learning model executes a systematic, iterative process to optimize parameters $$\mathbf{w}$$ and $$b$$. During each epoch, the loop passes through the entire training dataset. For every iteration within an epoch, a minibatch $$\mathcal{B}$$ is processed. The model computes the loss for this minibatch, averages it over the examples in the batch, and calculates the gradients of the averaged loss with respect to each parameter using $$\mathbf{g} \leftarrow \partial_{(\mathbf{w},b)} \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} l(\mathbf{x}^{(i)}, y^{(i)}, \mathbf{w}, b)$$. Because the loss is averaged per minibatch, the gradient in the optimization algorithm does not need to be separately divided by the batch size. Finally, the optimization algorithm updates the parameters using the rule $$(\mathbf{w}, b) \leftarrow (\mathbf{w}, b) - \eta \mathbf{g}$$. This cycle repeats until the training is complete.

Deep Learning Minibatch Training Loop

A model is being trained using an optimization algorithm where parameters are updated by taking a step in the direction opposite to the gradient of a loss function. For a specific parameter, the calculated gradient of the loss is a large negative value (-10.0). If the learning rate is set to a small positive value (0.01), how will this parameter's value change in the next update step?

Learn Before

Related