Interesting. We know that the Velaryons were of Valerian background but the family did not ride dragons pre conquest. We dont know why. Because the cou'dn't? Because the didnt have any dragons? Because there was some restriction in Valerian society as to which families were allowed to own dragons? They seem to have married Targaryans before the conquest so any of them could have Targaryan ancestry if that was a necessary condition for dragon riding.
I think the idea that the notable looks of Valerians means that anyone with similar looks must have some relationship with them is faulty. The same appearance is common in Lys - an area with a huge population and lots of trading connections. It seems more likely that there was a large ethnic group with the white hair/ purple eyes appearance - but that this was very unusual in Westeros so what would be commonplace in other areas of Planetos was simply very noticeable in the seven kingdoms.