In my opinion, Tywin's dislike for Tyrion only strengthened over the years as Tyrion grew. I think that Tywin sees that Tyrion is the one child that is most like him, and the fact that he is a dwarf, he can't stand it. A few people have mention in the books that only one of Lady Joanna's brood that is most like their father, is Tyrion. Like father, like son. Tyrion could have inherited his love for whores from his father, although in his own way, Tyrion has always been faithful.