The Base-Rate F1 Trap

Senior ML Engineer interview at Meta, and the interviewer asks:

“An engineer shows you a binary classification model boasting a phenomenal 0.90 F1-score on a newly curated validation set, claiming it’s ready for production deployment. Before even looking at the architecture, you flag this metric as a potential illusion. What hidden data profile characteristic are you suspecting, and how do you prove it?”

Don’t say: “A 0.90 F1-score is highly robust against class imbalance, unlike accuracy, so the model is fundamentally solid. To be safe, I’ll just check the confusion matrix, plot the ROC-AUC curve, and tune the classification threshold.”