But what if I have a duplex as my neighbors? Or my other neighbors put in a granny flat for their aging grandparent??!!!

Wont you think of the neighborhood character?