Downvotes are public… that guy could just look and see if that user is the downvoter. Though I guess it could be an alt.

addition: i swear i have seen it before on the site, but now cannot find it. so maybe im wrong?