RLHF trains AI systems to maximize user approval, not truth. The research shows this gets worse as models scale. A 2025 OpenAI incident proved it publicly.