A helpful website for understanding backpropagation in RNN:
http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/

University of Michigan - Ann Arbor

In an RNN, backpropagation needs to take into account that the model is carrying forward information from each neural layer to the next, and fine tune the weights that govern this “short-term memory”.

This is called Backpropagation Through Time (BPTT). BPTT uses the chain rule to go back from the latest time step to the previous step, and then to the next-previous, each time using gradient descent to discover the best weights for each neuron and for the hidden state function.

Backpropagation Through Time (BPTT)

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow.

Vanishing/exploding gradient

Weight-tying is where you have a language model and use the same weight matrix for the input-to-embedding layer (the input embedding) and the hidden-to-softmax layer (the output embedding). The idea is that these two matrices contain essentially the same information, each having a row per word in the vocabulary.

In addition to providing improved perplexity results, this approach significantly reduces the number of parameters required for the model.

Weight typing

A computational graph visualizes the dependencies among model variables and parameters during the forward and backward computation of a recurrent neural network. For example, computing the hidden state at a specific time step depends on the current input, the hidden state from the previous time step, and the hidden layer weight parameters. To calculate gradients for training, the graph is traversed in the opposite direction of the arrows, applying the chain rule sequentially from the final output back to the initial inputs and weights.

Computational Graph of RNN Backpropagation Through Time

For a recurrent neural network, the gradient of the objective function $$L$$ with respect to the output layer weight parameter $$\mathbf{W}_\textrm{qh}$$ is calculated by summing the gradients across all time steps $$T$$. Because the objective function depends on $$\mathbf{W}_\textrm{qh}$$ via the sequence of outputs $$\mathbf{o}_1, \ldots, \mathbf{o}_T$$, we apply the chain rule to obtain:

$$ \frac{\partial L}{\partial \mathbf{W}_\textrm{qh}} = \sum_{t=1}^T \textrm{prod}\left(\frac{\partial L}{\partial \mathbf{o}_t}, \frac{\partial \mathbf{o}_t}{\partial \mathbf{W}_\textrm{qh}}\right) = \sum_{t=1}^T \frac{\partial L}{\partial \mathbf{o}_t} \mathbf{h}_t^\top $$

where $$\mathbf{h}_t^\top$$ is the transpose of the hidden state at time step $$t$$, and $$\partial L/\partial \mathbf{o}_t$$ is the gradient of the objective with respect to the model output at that time step.

Gradient of RNN Objective with Respect to Output Weights

Teacher forcing is a common training strategy for sequence models where the ground-truth token from a prior time step is used as input, rather than the model's own generated prediction. In an encoder–decoder architecture, this involves feeding the original target sequence directly into the decoder. Specifically, a special beginning-of-sequence token (e.g., `` `<bos>` ``) is prepended to the target sequence, excluding its final token. The decoder is then trained to predict the original target sequence shifted by one time step, ending with an end-of-sequence token (e.g., `` `<eos>` ``). This shifting method for self-supervised learning closely resembles standard language model training.

Learn Before

Related