Google

The instruction-following ability of Large Language Models (LLMs) is their capacity to correctly perform tasks by adhering to instructions provided in a user's prompt. In many practical applications, a prompt is composed of a direct instruction and user input, and the LLM's success is determined by how well it executes the given command.

Instruction-Following Ability in LLMs

Text summarization is a common task that demonstrates the instruction-following ability of Large Language Models. A typical prompt for this task, often denoted as `x`, is composed of an instruction and a user input. For example, a prompt might be structured with an instruction like 'Summarize the following article,' which is then followed by the article text that serves as the user input. The model's objective is to adhere to the instruction and generate a concise summary of the provided document.

Example of Instruction Following: Summarization

A user provides a large language model with the prompt detailed below. Analyze the model's response based on the instructions given. In your analysis, identify which parts of the instruction the model followed correctly and which parts it failed to follow. Justify your evaluation.

Evaluating LLM Instruction Adherence

A user provides a language model with the following prompt:

'Translate this sentence into Spanish, and ensure your response is exactly five words long: 'The quick brown fox jumps over the lazy dog.''

The model responds with:
'El veloz zorro marrón salta sobre el perro perezoso.'

Identify the specific instruction the model failed to follow and briefly explain why the response is incorrect.

Analyzing an LLM's Instructional Failure

While prompting is a powerful method for adapting large language models through in-context learning, it is often insufficient on its own. Additional tuning efforts are typically required to ensure that the models can accurately and consistently follow user instructions.

Necessity of Tuning for Instruction Following

Prompting offers a straightforward and effective technique to direct a single Large Language Model to execute various tasks without needing to alter the model's underlying architecture or parameters. However, for this approach to be successful, it is a prerequisite that the language model already has the capacity to comprehend and adhere to the given instructions or questions.

Prompting Without Structural Adaptation

A user provides a language model with the following prompt: 'List the three most common states of matter. Then, in a separate, single sentence, explain why gases do not have a fixed shape.' Which of the following model responses best demonstrates a complete and accurate adherence to the user's instructions?

Learn Before

Related