The offsprings of Lord Walder Frey seemed to call him with variable titles. I'll list them below and provide a possible interpretation.
aSoS 35 Cat IV, Walder Rivers called him my grandfather, while he should use father.
aFfC 38 Jaime VI, Ryman Frey called him my lord grandfather, which suits their relationship.
aFfC 44 Jaime VII, Edwyn Frey called him my lord grandfather, while Jaime knew he was in fact Lord Walder's great grandson.
aSoS 49 Ser Aemon Rivers’s daughter Walda