I believe the overwhelming majority of these abuses have been men violating women and children, but I have also encountered unwanted attention (and a thigh grab) from a woman who's intention was pretty clear as well. Men have told me of instances where they have been harassed and grabbed by women and their experiences are discounted because "he is a man and therefore must want it anyway". This has nothing to do with attraction, it has nothing to do with mutual flirtation, it has everything to do with power and intimidation.
How do we fix this situation? I don't know. We can't force people to respect fellow human beings instead of seeing them as objects to promote their own power and pleasure. But we can stand up together and proclaim to the victims, "it is not your fault". We can believe and help victims when they come forward to tell their story. We can shame and shun those who blatantly violate others. I pray that the #metoo movement is the first step to making us more aware of the abuses, both social and criminal, that are going on around us everywhere.