The Invariance Illusion

Senior Computer Vision Engineer interview at Meta, and the interviewer asks:

“Your medical imaging model shows a flawless 0.99 AUC offline, but plummets to 0.65 when deployed because real-world clinic scans undergo minor 3-degree rotations and arbitrary cropping. How do you redesign your CI/CD evaluation pipeline to catch these semantic vulnerabilities automatically before hitting runtime?”

Don’t say: “Just inject random rotations and cropping into the training augmentation pipeline using Albumentations, and retrain the model. Then, add a randomized validation split in your CI/CD pipeline to verify the new performance.”