if you can’t understand their language, accent, or inflections?

I understand all of those things, and I’m a second language speaker for both English and Japanese, so there’s no favoritism there.

Do you reckon maybe American voice actors aren’t treated well has more to do with the elitist attitude weebs have regarding them than their skill levels?

No, I don’t. That’s a fairly senseless hypothesis.