Google

A training methodology where the parameters of a pre-trained Transformer model are held constant, or 'frozen'. During this process, only a set of newly introduced, learnable parameters are updated. This approach allows for model adaptation while preserving the original, powerful representations of the base Transformer.

Training Auxiliary Parameters with a Fixed Transformer Model

A machine learning engineer is tasked with adapting a large, pre-trained language model for a new, highly specialized domain. Instead of updating all of the model's original multi-billion parameters, they choose to keep the original model's parameters fixed and introduce a small number of new, trainable parameters to handle the new task. Critically evaluate this adaptation strategy. In your answer, discuss at least one major advantage and one potential disadvantage of this approach compared to a method where all model parameters are updated.

Evaluating a Model Adaptation Strategy

When adapting a large, pre-trained model by introducing and training only a small set of new parameters, the original weights of the base model are also fine-tuned, but with a much smaller learning rate to prevent drastic changes.

A research team wants to adapt a very large, pre-trained language model (with billions of parameters) to perform a new, specialized task, such as classifying medical reports. The team's primary constraint is a very limited computational budget, which makes it infeasible to update all of the model's original parameters. Which of the following training strategies best resolves this constraint while still effectively adapting the model to the new task?

Learn Before

Related