With all the kids he has piled up and so few have been married off. A real select few, is it that nobody wants their child with that wackjob of a family, or does Walder Frey most likely deny marriage to "lower" houses that he doesnt view are up to par with his house, meaning he thinks his family is in the same field as Tully and Stark etc
Freys are far from poor, but yet there is unmarried people piled up all over the place there, and it def sounded that Lord Walder more than once tried to marry some of them off.
Best 2 houses he got someone into was Bolton (taking the fattest girl so could get weight in silver). And Tully (Where Edmure was just part of a plot and he ended up in a dungeon.
So does nobody want to even associate with them or does he feel that their family name should only deserve them another prominent family to be crossed into