Google

This example demonstrates the application of token masking to a bilingual sentence pair. Starting with the aligned Chinese and English sentences packed into a single sequence, `[CLS]鲸鱼 是 哺乳 动物 。 [SEP] Whales are mammals . [SEP]`, a certain percentage of tokens are replaced with the `[MASK]` symbol. This results in a corrupted input for the model, such as `[CLS][MASK] 是 [MASK] 动物 。 [SEP] Whales [MASK][MASK] . [SEP]`, where the model's task is to predict the original tokens '鲸鱼', '哺乳', 'are', and 'mammals'.

Example of Masking a Bilingual Sentence Pair

This example illustrates the encoding process for a masked bilingual sentence pair. The input sequence, `[CLS] [MASK]是 [MASK]动物。 [SEP] Whales [MASK] [MASK] . [SEP]`, is first converted into a series of token embeddings, denoted as `e0` through `e11`. This embedding sequence is then processed by a Transformer Encoder, which outputs a corresponding sequence of contextualized hidden states, `h0` through `h11`. These hidden states serve as the basis for predicting the original masked tokens: '鲸鱼', '哺乳', 'are', and 'mammals'.

Transformer Encoding of a Masked Bilingual Sentence Pair

An engineer is preparing bilingual sentence pairs for a model. The process involves taking a packed sequence (e.g., `[CLS] sentence_one [SEP] sentence_two [SEP]`) and randomly replacing some words with a `[MASK]` symbol. The model is then trained to predict the original words. The engineer notices the model becomes very good at predicting common grammatical words (like 'is', 'a', 'the') but performs poorly on important content-carrying words (like specific nouns and verbs). Describe a modification to the word replacement strategy that would compel the model to improve its performance on these content-carrying words. Justify your reasoning.

Optimizing a Model's Training Strategy

Based on the following case study, evaluate the effectiveness of the described masking strategy for the specific task. Explain why it might be leading to the observed performance issues and suggest one specific adjustment to the masking process that could help the model better learn the specialized medical terminology.

Evaluating a Masking Strategy for Specialized Translation

A model is being prepared to understand relationships between aligned sentences in different languages. An input sequence is created by joining a Spanish sentence and its English translation. To train the model to predict missing words, some original words are replaced with a special `[MASK]` symbol. Given the original packed sequence below, which option correctly demonstrates this replacement process?

Original Sequence: `[CLS] El gato se sentó en la alfombra . [SEP] The cat sat on the mat . [S

Learn Before

Related