The PPO vs DPO Implementation Trap

Senior AI Engineer interview at OpenAI, and the interviewer asks:

“Our engineers want to rip out 𝘗𝘗𝘖 (𝘗𝘳𝘰𝘹𝘪𝘮𝘢𝘭 𝘗𝘰𝘭𝘪𝘤𝘺 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘢𝘵𝘪𝘰𝘯) and replace it with 𝘋𝘗𝘖 (𝘋𝘪𝘳𝘦𝘤𝘵 𝘗𝘳𝘦𝘧𝘦𝘳𝘦𝘯𝘤𝘦 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘢𝘵𝘪𝘰𝘯). They argue it’s strictly better because it simplifies the stack. Do we approve the PR?”