University of Michigan - Ann Arbor

To evaluate whether the Transformer architecture could generalize beyond machine translation to tasks with strong structural constraints and long target sequences, experiments were conducted on English constituency parsing.

A 4-layer Transformer with $d_{model} = 1024$ was trained under two regimes: a discriminative setup using only the Wall Street Journal (WSJ) portion of Penn Treebank (approximately 40K sentences with a 16K token vocabulary), and a semi-supervised setup augmented with approximately 17 million sentences from high-confidence and BerkeleyParser corpora (with a 32K token vocabulary). During inference, the beam size was increased to 21, the length penalty was set to $\alpha = 0.3$, and the maximum output length was configured to input length $+ 300$.

Despite having no task-specific tuning, the Transformer achieved an F1 score of 91.3 in the WSJ-only discriminative setting and 92.7 in the semi-supervised setting on Section 23 of the WSJ. This surpassed previously reported models such as the BerkeleyParser and prior RNN sequence-to-sequence approaches, trailing only the task-specific generative Recurrent Neural Network Grammar (93.3 F1).

Generalization to English Constituency Parsing

Match each training setup or baseline model from the English constituency parsing experiments to its corresponding description.

Order the following parsing models or configurations from lowest to highest F1 score achieved on Section 23 of the Wall Street Journal (WSJ) dataset.

Analyze the structural characteristics of constituency parsing that make these results significant for evaluating the Transformer's generalization capabilities, and explain how scaling from the discriminative to the semi-supervised regime affected performance relative to established parsing baselines.

Learn Before

Related