Google

Providing a Large Language Model with demonstrations consisting only of question-answer pairs can be insufficient for it to deduce the correct reasoning process. This limitation makes it difficult for the model to generalize and accurately solve similar problems that require multi-step thinking.

Limitation of Question-Answer Pair Demonstrations in Few-Shot Prompting

In Chain-of-Thought (CoT) prompting, large language models benefit significantly more from demonstrations that include detailed problem-solving steps than from simple question-answer pairs. The fundamental technique for creating a CoT prompt is to incorporate these intermediate reasoning steps, which allows the model to learn the process for deriving an answer, not just the final answer itself.

Creating a CoT Prompt by Incorporating Reasoning Steps

Analyze the following scenario describing a developer's prompting strategy. Based on the provided information, identify the primary flaw in the examples being used and explain why this flaw leads to the model's poor performance on new problems.

Diagnosing a Flawed Prompting Strategy

A developer is using a few-shot prompt to teach a large language model to solve geometry word problems. Each example in the prompt provides a problem description and the final calculated area (e.g., 'Problem: A rectangle has a length of 10cm and a width of 5cm. What is its area? Answer: 50 sq cm'). The model performs well on problems about rectangles but fails on problems about triangles. Based on the structure of the prompt examples, analyze the fundamental reason for the model's inability to generalize to a different shape.

Diagnosing Prompt Insufficiency

When a Large Language Model is prompted to calculate the average of the numbers 2, 4, and 9 using few-shot demonstrations that only contain question-answer pairs, it might provide the incorrect answer 7. The actual average is 5, calculated as $$(2+4+9)/3$$. This response illustrates that models can struggle to deduce multi-step mathematical reasoning without explicit step-by-step demonstrations.

LLM's Incorrect Answer (7) to the Prompt for Calculating the Average of 2, 4, and 9

A developer is trying to get a large language model to solve two-step arithmetic word problems. They use a few-shot prompting strategy, providing several examples. Each example consists of a word problem followed only by its final numerical answer (e.g., 'Problem: ... Answer: 15'). The model consistently fails to solve new, slightly different word problems. What is the most likely reason for the model's poor performance?

Learn Before

Related