Google

When a Large Language Model is prompted with a question like 'Where will the 2028 Olympics be held?' but is only given irrelevant context, such as information about the 2024 Paris Olympics, it may decline to provide an answer. This refusal stems from the provided text being insufficient to form an accurate response.

Example of LLM Refusal with Irrelevant Context

A customer support chatbot is designed to answer user questions based *only* on the contents of a specific product's user manual. The manual for a new smart camera details its setup process, technical specifications, and troubleshooting for connectivity issues. A user asks the chatbot, 'How does this camera's image quality compare to the previous model?' The provided manual does not mention any other models or make any comparisons. The chatbot responds: 'I cannot answer your question. The provided user manual does not contain a comparison of this camera's image quality with previous models.' Analyze this response. Is it the correct behavior for the chatbot given its design constraints? Explain why or why not.

Evaluating a Chatbot's Fidelity

Imagine a language model is instructed to answer questions *only* using the information from a provided text. The model receives the following user question and text:

**User Question:** 'What are the primary causes of the Renaissance in Europe?'

**Provided Text:** 'The Renaissance was a period of great artistic and cultural change in Europe. Famous artists like Leonardo da Vinci and Michelangelo created masterpieces during this time. The period saw a renewed interest in the classical art and literature of ancient Greece and Rome.'

Based on this scenario, write the most appropriate and faithful response the model should generate. Then, briefly explain the principle that justifies this response.

Generating a Faithful LLM Response

A user asks a language model, 'What is the capital of Australia?'. The model is instructed to answer *only* using the information from the following text: 'Sydney is a vibrant city in Australia, famous for its opera house and harbor bridge. It is the most populous city in the country and a major hub for tourism and finance.' Which of the following is the most appropriate and faithful response for the model to generate?

Learn Before

Related