Because some of us have fat fingers and accidentally downvote when we scroll on mobile.
One of the things I liked about reddit was that, since it saved downvoted posts, I could go through the list every once in a while and undownvote the accidents.
Can't do that here though, and I sometimes notice posts or comments I've accidentally downvoted.
Anyway, people shouldn't care so much, we don't have a karma system or the like here anyways, so why does it matter?