Soft Prompt Learning as Context Compression via Knowledge Distillation

Formula for Optimizing Soft Prompts via Context Compression

Alternative Methods for Soft Prompt Optimization

A developer is tasked with creating a compact, learned 'soft prompt' that can effectively replace a very long and detailed set of instructions (the 'full context') for a language model. The objective is to ensure that for any given user query, the model's final output is nearly identical whether it's conditioned on the long instructions or the new compact prompt. Which of the following optimization strategies directly targets this specific objective?

When training a soft prompt to act as a compressed version of a longer context, the primary optimization objective is to ensure the learned soft prompt's vector representation is as close as possible to the vector representation of the original context.

Interpreting the Soft Prompt Optimization Formula