The Linear Bias Misconception

Senior LLM Engineer interview at Google DeepMind, and the interviewer asks:

“You inherit a legacy Transformer codebase and notice the team is dropping all bias terms in the linear layers. Someone suggests adding them back to increase the representational power of the model. What production-level training catastrophe are you actively inviting if you approve that pull request?”

Don’t say: “It’s a great idea to add them back. Bias terms add negligible compute overhead but allow the linear layers to shift activation functions, helping the model fit the data distribution better and avoid dead ReLUs.”