Google

When people can no longer easily tell which examples the model is still getting wrong, only some human-comparison methods remain useful. At that point, progress usually becomes slower on tasks where the model already exceeds human performance, because humans have fewer reliable ways to spot weaknesses. When the model is still behind people, those comparison methods tend to be more informative and improvement is often faster.

Why Human Comparison Becomes Less Helpful After Strong Model Performance

When does human comparison stop helping much?

Faster Gains Before Surpassing a Human Baseline

When people can no longer easily point out the model’s obvious mistakes, only a _____ of comparison methods remain useful.

Put the reasoning steps in order for why improvement slows after machine performance exceeds human performance.

Question:
A company builds a defect-detection system for factory parts. While the system still makes more mistakes than experienced inspectors, engineers can quickly review examples, compare the model with human judgments, and find obvious failures. Later, the system becomes better than the inspectors on the main benchmark. Explain why further progress often becomes slower at that stage.

Sample answer:
When the system is still worse than the inspectors, people can easily recognize clear errors and use those cases to guide debugging and model changes. After the system surpasses the inspectors, its remaining errors are harder for people to notice and judge. As a result, fewer human-comparison methods remain useful, so it becomes harder to identify what to improve next and progress slows.

Key points:
- Improvement is usually faster while the model is still below human performance
- Human reviewers can spot clear mistakes more easily in that stage
- After the model exceeds human performance, its remaining errors are harder to detect
- Fewer human-comparison tools remain effective, which slows further gains

Rubric:
Full credit requires explaining both stages, describing why humans can diagnose problems more easily before the model reaches human performance, and linking the slowdown to the reduced usefulness of human-based comparison methods.

Why improvement slows once a model beats human performance

Case context:
A logistics company built a machine learning model to decide which packages should be sent to extra inspection. The model now makes fewer mistakes than the experienced staff who used to do the review by hand. After reaching that point, the team notices that each new round of improvement produces smaller gains.

Question:
Using the idea from the source, explain why progress has slowed and what that means for the methods the team can still use.

Sample answer:
Progress has slowed because the model is already outperforming the human reviewers. Once that happens, people can no longer easily point to many cases where the model is obviously wrong, so methods that depend on comparing the system with human judgment become less useful. The team can still improve the model, but fewer human-based comparison strategies remain, so advancement is usually slower than it was earlier.

Key points:
- Package screening is an example of a task where the model can move beyond human performance
- Improvement often slows after the system overtakes human reviewers
- When the model is already better, humans have a harder time finding clear errors
- Only some techniques that rely on human comparison still work well

Rubric:
Full credit identifies that outperforming human reviewers is the reason progress has slowed, uses package screening as the example domain, and explains that fewer human-comparison methods remain available.

Explain why progress has become harder for a package-screening model.

Question:
Name two applications the lesson uses as examples of tasks where machine systems already exceed typical human performance.

Sample answer:
Two examples are estimating parcel delivery times and deciding whether a bank transfer looks suspicious.

Key points:
- Estimating parcel delivery times
- Screening bank transfers for suspicious activity
- Predicting customer churn

Rubric:
Full credit for naming any two of the three examples given in the lesson: estimating parcel delivery times, screening bank transfers for suspicious activity, or predicting customer churn.

Learn Before

Related