Yeah I get where you are coming from with that and it's a bit of an unsealed road. It's a lot like when does freedom of speech cross over into abuse?
I think about it in whats a very simplistic way and that is when someone chooses to do or say something to another person and that person is offended or feels anxiety by the action then it's wrong. Both men and women have their threshholds of what they consider ok. Being aware of the threshholds or erring on the side of caution is probably the smartest way to go. Harvey clearly crossed a line and got rightly smashed. He should have stuck to the age old adage of look but don't touch unless invited.
