I think your last point is oft overlooked. No one liked Gaetz because he was reportedly really douchey. Showing naked pictures of women to people who did not ask to see them is one example. There was also that time he over threw the top GOP congressional fundraiser because he allowed an investigation against him. When that guy from North Carolina (south Carolina maybe? I don’t remember his name but he used a wheel chair) talked about drug use and sex parties it was widely understood at the time to be about Gaetz. Then that guy had a bunch of oppo dumped against him so he’s probably working for a church or some shit now.

Tldr. Matt Gaetz is such an entitled shithead that other very entitled people didn’t like him.