I think that if you allow that question in the first place, voting on it should not have any consequences either.
Besides, despite what most people instinctively think it's better to see what you disagree with so that you can keep your eyes on it rather than forcing it into hiding and knowing nothing (again, in moderation - you probably don't want to run an actual Nazi instance, so if it does bother you you should moderate that post/comment).
And mistakes still happen; it's easy to accidentally upvote/downvote something by mistake, to misunderstand someone, etc. So yes, I do think banning people based on what they up/downvote is a bad idea.