Mechanistic Analysis of Alignment Algorithms in Language Models

Researchers conducted a systematic analysis of six preference-optimization methods (PPO, DPO, SimPO, ORPO, GRPO, and KTO) to understand how they reshape language models' internal computations. The study found that different alignment objectives induce qualitatively distinct representational changes, with some methods enhancing feature separability while others degrade it, revealing that behavioral alignment doesn't guarantee uniform internal restructuring.