I'll tell you, as a man who often walks home alone at night, it's never the women I'm scared of. I know how to defend myself, and I'm not small, but I still keep my guard up when I meet a man on my walk home from work.
Yes it's not all men, but it is in the vast majority of cases men, and that merits some sense of caution.