Learn Before
Divide-and-Conquer Strategies in transformers
An input sequence can be decomposed into finer segments that can be efficiently processed by Transformer or Transformer modules
0
1
Tags
Data Science
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Learn After
Sequence Parallelism
A team is tasked with using a transformer-based model to summarize an entire book. The standard model architecture cannot process the entire book's text at once due to its length. The team implements a strategy where the book is broken into smaller, manageable chunks, each chunk is processed by the model, and the outputs are then combined. What is the fundamental computational bottleneck in the standard architecture that this segmentation strategy is designed to circumvent?
Analyzing a Hierarchical Transformer for Genomic Data
Applying a Segmentation Strategy for Long-Form Audio