In general it’s asking another person to change their appearance to suit your preferences rather than just respecting whatever mood they’re in or how their naturally resting face just looks, it’s a dick move. Forcing a smile also sucks. It takes some amount of conscious effort to maintain a forced smile and smiling does engage quite a few muscles in the face, about 43 different muscles to make a smile.

Combine all of that with the history of women very much not having any kind of power in our society, and it takes on a more sinister tone when directed at women, hinting at the idea of someone’s only valuable because they have a pretty face and it should be pretty all the time.