Google

A practical error-analysis routine is to inspect about 100 validation examples the model gets wrong, group the failures by type, and use the counts to decide which problems deserve attention first.

Reviewing a Sample of Validation Errors

Why Perform Manual Error Analysis?

Should error analysis be limited to examples the model got right?

Match each part of manual error review with its description.

Arrange the manual error-review process in order.

Question:
Explain how a team can use a manual review of roughly 100 mistakes made on a validation set to decide which problems deserve the most attention.

Sample answer:
A team can look through about 100 validation errors and sort each one into a small number of categories, such as wrong labels, confusing inputs, or model failures on rare cases. After tallying how often each category appears, the team can estimate which types of mistakes account for the largest share of the total error. That makes it easier to choose the highest-value fixes first, instead of spreading effort evenly across all problems.

Key points:
- Review a sample of about 100 validation errors
- Group the mistakes into meaningful categories
- Count how often each category appears
- Focus engineering work on the categories that contribute most to the overall error

Rubric:
A full-credit response must describe sampling about 100 validation mistakes, classifying them into categories, counting the categories, and using the counts to prioritize the most common or costly error types first.

Using a Small Error Review to Set Priorities

Case context:
You are building a classifier for scanned office documents. You review 120 development examples that your system misclassified. In the manual analysis, 48% of these errors are invoices, while 7% are handwritten notes.

Question:
Based on the manual review of these 120 examples, which error type should you fix first, and how does the analysis support that choice?

Sample answer:
You should prioritize invoice misclassifications. They make up 48% of the reviewed mistakes, compared with only 7% for handwritten notes. Fixing the most common error category is likely to reduce the overall development error the most.

Key points:
- Identify invoices as the highest-priority error category
- Use the 48% versus 7% counts from the review
- Explain that the largest category offers the biggest opportunity to lower total error

Rubric:
The answer must identify invoices as the priority category and justify this choice by noting that they account for a much larger share (48% of the 120 misclassified examples) than handwritten notes (7%).

Choosing the most important error type to fix in a document classifier

Question:
What is the main reason for manually reviewing about 120 validation examples that the model predicts incorrectly?

Sample answer:
The goal is to inspect the mistakes, group them into common categories, and count how often each category appears so you can decide which problems to fix first.

Key points:
- Look through examples the model got wrong
- Group the errors into major types
- Use the counts to prioritize debugging work

Rubric:
The response must explain that reviewing about 120 misclassified validation examples helps identify major error categories and rank them by frequency so the most important ones can be fixed first.

Why inspect about 120 validation mistakes?

A team is reviewing a text classifier and wants to understand its mistakes before making changes. Which sample is the best choice for error analysis?

Learn Before

Related