The False Positive Blindspot

Senior MLOps Engineer interview at OpenAI, and the interviewer asks:

“Your anomaly detection model boasts an incredible 0.98 ROC-AUC on an extreme 1:10,000 fraud-to-clean dataset. Yet, the moment it hits production, the team faces a massive flood of false positives. Why did your offline metric lie to you, and how do you fix it?”

Don’t say: “We just need to adjust the classification threshold on the ROC curve to favor precision, or perhaps implement SMOTE during retraining to balance out the minority class.”