Google

After defining the model architecture and training objective for a denoising autoencoder, a key remaining step is to specify how the input data is corrupted. The BART model, developed by Lewis et al. (2020), exemplifies this by employing several different methods for corrupting the input sequence during its pre-training phase.

BART Model's Use of Diverse Input Corruption Methods

Analyze the team's pre-training approach. What is the primary limitation of using only word deletion for their intended tasks, and what is one different method of corrupting the input that they should introduce to address the observed weaknesses? Explain your reasoning.

Diagnosing a Denoising Pre-training Strategy

Compare and contrast two common methods for corrupting text inputs during the pre-training of a denoising autoencoder: 'Token Masking' (where individual tokens are replaced with a special symbol) and 'Text Infilling' (where a span of text of variable length is replaced with a single special symbol). In your analysis, explain how each method forces the model to learn different aspects of language structure and meaning.

Analysis of Input Corruption Techniques

A research team is pre-training a text-based model with the goal of making it highly robust and flexible for a wide range of downstream applications, including generating coherent paragraphs and correcting grammatical errors. The model is trained to reconstruct original text from a corrupted version. Which of the following corruption strategies applied during pre-training would be most effective for achieving this goal?

Learn Before

Related