Learn Before
Why Focus on the Incorrect Dev Predictions?
Question: In one to three sentences, explain why it is usually more efficient to examine only the wrong predictions in a development set instead of checking every prediction.
Sample answer: If a model is correct on most examples, the wrong predictions are a much smaller set to inspect. For example, if a system gets 97% accuracy on 2,500 items, only 75 items are incorrect, so reviewing those 75 takes far less time than reviewing the 2,425 correct ones.
Key points:
- A small number of errors is much easier to review than the full set.
- Less reviewing time means the analysis is more practical.
Rubric: Full credit requires noting the small error count compared with the much larger number of correct cases and explaining that this reduces review effort.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main reason error-checking can become biased when revising labels in a dev set?
Reviewing the Mistakes
Counting Misclassifications in a Development Set
Match Each Validation-Set Quantity to Its Meaning
Why fixing only the mistakes can distort a development set
Explain why label checks often get focused on the examples a model gets wrong.
Explain the bias created when a team inspects only the validation errors.
What problem can arise when you review only the validation examples the system gets wrong?
True or False: The 1,120 examples that were classified correctly must all have perfectly accurate labels because they were not selected for review.