Unless they intentionally damage your life by influencing coworkers, friends, and family,

I hate to break it to you, but what you just described is precisely what stigmatising gayness does.