If the transphobes thought trans men were men then your comment would be accurate. But they just see them as confused women and easy to just ignore them like they ignore cis women.

(You are right about trans women bearing the brunt of the hate, and I think so much of that is sexual panic from cishet men about finding a penis owner attractive)