Wearing armor does mean that you won’t benefit from the mage armor perk in the alteration tree, if you’re investing in that, but if you wanna wear armor you can do that for sure. It really doesn’t make that huge a difference either way honestly, and I wouldn’t stress too much over what you do or don’t put perks in. You can max every skill in the game if you want, and if you’ve got the patience for it you can get every perk as well.