Absolutely but you’re leaving out a key concept here, which is pay attention to the reception. If they receive it badly that’s not a hard ‘no fuck off’ (well… unless it literally is), but if it continues to be received badly or even just neutrally, it’s important to recognize this.

OP is worried about how men are judged by default (and understandably so, from both sides of this kind of situation), and they are clearly a little ‘too worried’ but really it’s more like being worried about the wrong part.

People that have trouble with this kind of thing need to shift their perceptive from ‘how will they take it’ to ‘how did they take it’ and make an audible from that point instead of trying to predict the entire sequence. It’s not about the gender or anything else other than how they as a human being receive your attention which requires rational analysis in real time (which is likely where those with low self-esteem can run into trouble) and not falling back on your default perceptions to help cope with the results