So I’ve just come across some information which lets me just get to the point as to why it’s significant if Craster is a Casterly. Actually, Craster was a tangent on the Casterlys and Lannisters in general. I’ll have a lot more backup for this in the next post which also expands into other things. Jaime dreams of creatures that want him under the cold, dark water in the caverns under Casterly Rock. Casterly Rock is described as “thrice the height of the Wall or the Hightower in Oldtown. Almost two leagues  long from west to east…” so it’s entirely believable that things exist well hidden deep in Casterly Rock. Cowled is a synonym for shrouded in some contexts. Around him stood a dozen tall dark figures in cowled robes that hid their faces. In their hands were spears. " He had no choice but to descend. Down a twisting passageway he went, narrow steps carved from the living rock, down and down. ... Crouching, listening, Jaime moved in a circle, ready for anything that might come out of the darkness. The water flowed into his boots, ankle deep and bitterly cold. Beware the water, he told himself. There may be creatures living in it, hidden deeps . . . Notice here “Hooded” (another word for shrouded in some contexts), “necromancy” and “return from the grave” and “thralls”. Also note that Loreon is the first King. Given how people like to King themselves, he likely doesn't post date Lann and the Casterlys by much. Human thralls were kept at that time, but given that he was a powerful necromancer and what he did with the Lions, I wonder if his thralls were not something else. Lann the Clever never called himself a king, as best we know, though some tales told centuries later have conferred that style on him posthumously.  The first true Lannister king we know of is Loreon Lannister, also known as Loreon the Lion (a number of Lannisters through the centuries have been dubbed ‘the Lion’ or ‘the Golden,’ for understandable reasons), who made the Reynes of Castamere his vassals by wedding a daughter of that House, and defeated the Hooded King, Morgon Banefort, and his thralls in a war that lasted twenty years. King Morgon was supposedly a necromancer of terrible power, and it is written that as he lay dying, he told the Lannisters who had slain him (amongst them three of Loreon’s own sons) that he would return from the grave to wreak vengeance upon them one and all.  To prevent that, Loreon had Morgon’s body hacked into a hundred pieces and fed to his lions.  In a grisly aftermath, however, those selfsame lions broke loose two years later in the bowels of Casterly Rock, and slew the king’s sons, just as the Hooded King had promised. Dywen said Craster was a kinslayer, liar, raper, and craven, and hinted that he trafficked with slavers and demons. "And worse," the old forester would add, clacking his wooden teeth. "There's a cold smell to that one, there is." Apparently the Westerlands Chapter in TWOIAF is abridged from this full version on GRRM’s website.
http://www.georgerrmartin.com/world-of-ice-and-fire-sample/   Thanks to the other posters on this thread as their mental meanderings helped greatly in filling in some holes! I’m guessing that this goes back to some connection with the Rhoyne. The Casterlys had reason to visit the Rhoyne after their gold discovery as the Rhoynar were expert metal workers. Garin’s curse of greyscale was ~700BC, long after the Casterlys, but we don’t know that greyscale (or something similar) didn’t exist long before that point or that it wasn’t used as a threat. At the same time the Valyrians made a truce with the Rhoyne instead of fire-and-blooding them like everyone else, they were avoiding the Casterlys and then later the Lannisters while we have reports of hooded lords who can perform necromancy and control thralls in the Westerlands. Given Tywin’s and Craster’s connections to “foul humors” and rain, it all sounds Shrouded Lord-ish. The wealth of the westerlands was matched, in ancient times, with the hunger of the Freehold of Valyria for precious metals, yet there seems no evidence that the dragonlords ever made contact with the lords of the Rock, Casterly or Lannister. Septon Barth speculated on the matter, referring to a Valyrian text that has since been lost, suggesting that the Freehold's sorcerers foretold that the gold of Casterly Rock would destroy them. TWOIAF—Ancient History: Valyria’s Children In short, the names and numbers of the peoples who fell to Valyria are unknown to us today. What records the Valyrians kept of their conquests were largely destroyed by the Doom, and few if any of these peoples documented their own histories in a way that survived the Freehold's dominion. A few, such as the Rhoynar, lasted against the tide for centuries, or even millennia. The Rhoynar, who founded great cities along the Rhoyne, were said to be the first to learn the art of iron-making. ADWD Tyrion V "The conquerors did not believe either, Hugor Hill," said Ysilla. "The men of Volantis and Valyria hung Garin in a golden cage and made mock as he called upon his Mother to destroy them. But in the night the waters rose and drowned them, and from that day to this they have not rested. They are down there still beneath the water, they who were once the lords of fire. Their cold breath rises from the murk to make these fogs, and their flesh has turned as stony as their hearts." Valyria initially believed in the curses but later did not? Is that what restricted them at first?   Wild speculation here: it’s mentioned with the Rhoynar’s resistance against Valyria that they learned iron-making. Iron is placed on the statues at Winterell, Ironwoods surround Crasters’ keep, Ironwood doors on Winterfell’s crypt doors. Perhaps this is when the greyscale originated, or at least a predecessor and iron-making came along with it to contain and control it?   Notice Jaime’s connection to hoods, stone, water, fish, fishscales, and snow/all white in the quotes below. He’s a stone-like watcher of the dead, hooded, on chilly, dreary rainy day in a place that stinks (foul humors linked to the Shrouded Lord). He wears scales like Rhoynish armor and also mother-of-pearl. He's linked to snow and all white not unlike an Other. Also note another passage where Jaime sees a dead man he doesn’t recognize in the water along with a Tully trout. He saw a dead man in the water and wondered if he knew him. His Lannister hair disappears under the water as it’s cut off. He then looks at himself in the water and doesn’t recognize himself.   AFFC Cersei II Under the Great Sept's lofty dome of glass and gold and crystal, Lord Tywin Lannister's body rested upon a stepped marble bier. At its head Jaime stood at vigil, his one good hand curled about the hilt of a tall golden greatsword whose point rested on the floor. The hooded cloak he wore was as white as freshly fallen snow, and the scales of his long hauberk were mother-of-pearl chased with gold. Lord Tywin would have wanted him in Lannister gold and crimson, she thought. It always angered him to see Jaime all in white. She glanced at Jaime. Her twin stood as if he had been carved from stone, and would not meet her eyes.   TWOIAF Ten Thousand Ships The Rhoynish warrior with his silver-scaled armor, fish-head helm, tall spear, and turtle-shell shield was esteemed and feared by all those who faced him in battle. It was said the Mother Rhoyne herself whispered to her children of every threat, that the Rhoynar princes wielded strange, uncanny powers, that Rhoynish women fought as fiercely as Rhoynish men, and that their cities were protected by "watery walls" that would rise to drown any foe.   ASOS Jaime I Birds flew overhead, or cried out from the trees along the shore, and Jaime glimpsed silvery fish knifing through the water. Tully trout, there's a bad omen, he thought, until he saw a worse—one of the floating logs they passed turned out to be a dead man, bloodless and swollen. His cloak was tangled in the roots of a fallen tree, its color unmistakably Lannister crimson. He wondered if the corpse had been someone he knew. ... Cleos hacked away manfully, sawing and ripping his way through the mats and tossing the hair over the side. The golden curls floated on the surface of the water, gradually falling astern. As the tangles vanished, a louse went crawling down his neck. ... The reflection in the water was a man he did not know. Not only was he bald, but he looked as though he had aged five years in that dungeon; his face was thinner, with hollows under his eyes and lines he did not remember. I don't look as much like Cersei this way. She'll hate that. In the passage above where Jaime is described not unlike the (hooded) Shrouded Lord, his gold sword is prominent. Valyria’s sorcerers prophesied that Casterly Rock gold (not silver) would end them. Craster wears a gold ring around his wrist and asks for an axe with gold inlay. Garin was hung in a gold cage when he issued his curse.   It’s long been speculated that the stonemen, greyscale and the stone statues of Winterfell (and the stonemen of Casterly Rock?) are tied to the Others and it has to do with bloodlines. Hence Craster’s relationship with the Others. Val says that greyscale is stronger on the other (Others’) side of the Wall.   ADWD Jon XI Once outside and well away from the queen's men, Val gave vent to her wroth. "You lied about her beard. That one has more hair on her chin than I have between my legs. And the daughter … her face …" "Greyscale." "The grey death is what we call it." "It is not always mortal in children." "North of the Wall it is.   This also has interesting implications for Pycelle. He was exposed to the grey plague as a child. He does not say whether he caught it or not, though the hint is there. If Pycelle caught the grey plague as a child and is now immune and the Lannisters have greyscale or something similar at Casterly Rock, then Pycelle’s value to the Lannisters is clear. Note that Pycelle links Tywin’s doing what was needed to Quenton Hightower’s doing what was needed against the grey plague for the reader hinting that there was more to Tywin’s ruthlessness than it appears.  AFFC Jaime I Without his beard, Pycelle looked not only old, but feeble. Shaving him was the cruelest thing Tyrion could have done, thought Jaime, who knew what it was to lose a part of yourself, the part that made you who you were. Pycelle's beard had been magnificent, white as snow and soft as lambswool, a luxuriant growth that covered cheeks and chin and flowed down almost to his belt. The Grand Maester had been wont to stroke it when he pontificated. It had given him an air of wisdom, and concealed all manner of unsavory things: the loose skin dangling beneath the old man's jaw, the small querulous mouth and missing teeth, warts and wrinkles and age spots too numerous to count. Though Pycelle was trying to regrow what he had lost, he was failing. Only wisps and tufts sprouted from his wrinkled cheeks and weak chin, so thin that Jaime could see the splotchy pink skin beneath. "Ser Jaime, I have seen terrible things in my time," the old man said. "Wars, battles, murders most foul . . . I was a boy in Oldtown when the grey plague took half the city and three-quarters of the Citadel. Lord Hightower burned every ship in port, closed the gates, and commanded his guards to slay all those who tried to flee, be they men, women, or babes in arms. They killed him when the plague had run its course. On the very day he reopened the port, they dragged him from his horse and slit his throat, and his young son's as well. To this day the ignorant in Oldtown will spit at the sound of his name, but Quenton Hightower did what was needed. Your father was that sort of man as well. A man who did what was needed." ADWD Tyrion V Maesters and septons alike agreed that children marked by greyscale could never be touched by the rarer mortal form of the affliction, nor by its terrible swift cousin, the grey plague. "Damp is said to be the culprit," he said. "Foul humors in the air. Not curses." I can’t say exactly what the plot relevance might be if Craster is a Casterly because I can’t say exactly what greyscale and the Others might have in common, though they do seem connected. The Lannisters and Craster and Casterly are all linked to greyscale and Shrouded Lord things like rain/water, damp/chill, shit/”foul humors”, creatures in dark frigid water, hooded necromancers, Craster’s rumored association with demons, his confirmed association with the Others, and it’s all looped into why the greyscale is always deadly north of the Wall.