Looking at AGOT, Lord Commander Mormont didn’t want to give command to Waymar Royce because of his inexperience, but caved because he didn’t want to offend Bronze Yohn Royce. Why did Mormont cave so easily to Lord Royce? What could Royce do to him if Waymar wasn’t given command?
I'd assume there would be some number of smallfolk streaming south as Manderly gives over control of the castles to Ramsay. It's not as if it isn't common knowledge that he's a monster. Would you rather live where the "Lord" starved your former liege lord, starts a fight over land he realistically shouldn't have a claim to, and hunts and rapes peasant girls all over his land? There's open negative talk of the Boltons in the Merman court, the wine sinks, Lady Dustin mentions it, and Roose chastises his son for sowing tales in Bolton lands.
How much of the refugee population is due to that? I have no idea, but it's certainly something to take into consideration.