I thought I had my brewing routine down to pretty much perfect, before I saw a Lance Hendrick video where he recommends sorting out faulty beans and getting rid of most of the chaff.
It may seem tedious to do for each cup, but for me it’s worth it. The cleanness and quality of my brews improved drastically, making me enjoy black coffee way more.
If you’re patient enough, give it a try, it pays off!
I’ll pick out defects if I spot them, but I don’t obsess over it. Re: chaff, after some experimentation, I don’t feel like it makes a significant difference in my cups for the most part.