Back when dinosaurs cost a dollar and 640K of RAM was more than enough for any purpose, people in my sphere would refer to this style of dressing as “Preppy” and it was a style largely relegated to wealthier people and people who wanted to appear to be wealthy. So, maybe there was a bit of an association with this appearance and “rich assholes” in general?