Advanced NLP Interview Questions, issue 21, Dec 27, 2025
The PPO vs DPO Implementation Trap
Senior AI Engineer interview at OpenAI, and the interviewer asks:
โOur engineers want to rip out ๐๐๐ (๐๐ณ๐ฐ๐น๐ช๐ฎ๐ข๐ญ ๐๐ฐ๐ญ๐ช๐ค๐บ ๐๐ฑ๐ต๐ช๐ฎ๐ช๐ป๐ข๐ต๐ช๐ฐ๐ฏ) and replace it with ๐๐๐ (๐๐ช๐ณ๐ฆ๐ค๐ต ๐๐ณ๐ฆ๐ง๐ฆ๐ณ๐ฆ๐ฏ๐ค๐ฆ ๐๐ฑ๐ต๐ช๐ฎ๐ช๐ป๐ข๐ต๐ช๐ฐ๐ฏ). They argue itโs strictly better because it simplifies the stack. Do we approve the PR?โ
Why replacing PPO with DPO is not a free lunch - and how gradient saturation turns โsimplerโ into โweaker.โ
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.