Sorry if this has been mentioned. But Tywin wanting to put him in the ocean but instead raising him as a son COULD point to the fact he wasn't his. Admitting his wife cheated would harm the family, and it was the point that he had done something for the family rather than himself he was referring to.