Are you going to reinvent 4e? Because a lot of attempts to fix 5e turn into 4e.

I think it worked like Everyone has a few defenses. Probably Armor, reflex, will, fortitude. Write them down on your sheet. They can receive bonuses from different things. All attacks target one (or more, maybe, if you want to get fancy) defense. Attacker always rolls. Easy peasy.