Is it because :
a. Each thinks the other is inherently inferior in some major trait ( like freedom, tradition etc. )
b. That the wall separates them as outsiders just like the Berlin wall or maybe India-Pakistan where the same people reside on both sides yet the hatred is there because the others are on the 'other side'.
c. The Wildlings association with Old Gods against the majority of Westeros following New Gods ( debatable since Starks are old gods keepers )
d.The distinct cultural & climatic differences.
e. Perception of Wildlings as being raiders /
Also , if I am not wrong .. the wall was mainly built to keep the Others away & not 'particularly' wildlings .
As Tyrion remarked to Benjen once in the TV show I think '' The only difference between us & the wildlings is that our ancestors chose the 'right' side '....
In some ways Credit to Jon Snow for going beyond any of these above mentioned perceptions.
So, have I missed some point or is it that my assumptions mentioned are wrong by itself ?