Google

A parts-inspection classifier trained and evaluated on clean laboratory photos from a fixed camera can perform worse after deployment on factory-floor tablet images. The tablet photos are often lower-resolution, blurred by hand movement, and unevenly lit, so the original evaluation set does not represent the deployment distribution.

Laboratory Photos and Factory Tablet Images as a Distribution Mismatch Example

The evaluation set matched the camera conditions seen during deployment.

A bird-species classifier was trained on 1,800 studio images, but it did not generalize to _____ pictures taken by field volunteers on tablets.

Field Photos from Visitor Tablets

Match each part of the bird-photo classifier example to its role in the distribution shift.

Order the steps that exposed a data-distribution mismatch in an email spam filter.

Question:
In a concise analytical response, explain how the training and testing setup could make a document-classifier look better than it would perform after deployment.

Sample answer:
The model was trained and evaluated using scanned document images collected from a controlled source, so the test score reflected performance on that same kind of input. In deployment, however, the app received photos taken with phones, which often included skewed pages, shadows, motion blur, glare, and heavy compression. Because the evaluation set did not match the real input distribution, strong test performance did not mean the system would work equally well for phone photos.

Key points:
- Training and test data came from scanned documents.
- Deployment inputs were phone photos of documents.
- Phone photos often had skew, blur, shadows, glare, or compression.
- Success on the scanned-document test set did not prove generalization to deployment inputs.

Rubric:
A strong response identifies both data sources, describes the visual or capture-quality differences, and explains that the mismatch between evaluation data and deployment data makes the test result unreliable for predicting real-world performance. Partial credit should be given for noting the mismatch even if the generalization issue is not fully explained.

Why Did Good Results on Scanned Documents Not Guarantee Success on Phone Photos?

Case context:
A team trains a document classifier using scanned files collected from a desktop portal and evaluates it on a held-out set from the same portal. The model performs well in that setting. After deployment in a mobile app, where users submit handheld photos of documents, accuracy drops sharply. The phone images are often tilted, unevenly lit, and lower in resolution than the scanned files.

Question:
What should the team identify as the main issue, and why did the earlier evaluation miss it?

Sample answer:
The main issue is a distribution mismatch between the scanned documents used for training and testing and the handheld phone photos seen after deployment. The earlier evaluation missed the problem because the test set came from the same scan-based source as the training data, so it did not reflect the image conditions the app would actually receive.

Key points:
- Scanned documents were used for development data.
- Handheld phone photos were used in deployment.
- The two sources differed in resolution, lighting, and angle.
- The test set did not match the real deployment distribution.

Rubric:
Full credit requires identifying the distribution mismatch, distinguishing scans from phone photos, and explaining why a scan-based test set could not reliably predict performance on the deployed inputs.

Diagnose a document classifier that works on office scans but fails on phone photos.

Question:
Answer in one to three sentences. In the app example, what three properties set customer-submitted photos apart from the studio catalog images?

Sample answer:
The customer photos were typically smaller in resolution, more out of focus, and taken in dimmer light than the catalog pictures.

Key points:
- Smaller image resolution
- More blur or less sharpness
- Dimmer lighting

Rubric:
The response should identify lower resolution, greater blur, and weaker lighting. Equivalent wording is acceptable.

Learn Before

Related