Just because you do not like a type of people does not make you racist.
The old saying is birds of a feather flock together. I.e. people like to hang out with other people who are like themselves
and that is only human nature.
Now if you have one group of people who do not like you, whose customs and way of life threaten you and do not look like you
then it is only natural you are not going to warm to them.