Should one gender be more feared than another?

Consider this hypothetical: You’re walking alone at night. You become aware that someone is walking behind you. You don’t want to overtly turn and study this person, but a quick glance reveals the apparent gender of the person.

Would you feel any difference in the threat level depending on whether you perceive the person to be a man or a woman?

If you feel that there is a difference, would this be more or less valid than a difference based on racial distinctions?