I'd assume there would be some number of smallfolk streaming south as Manderly gives over control of the castles to Ramsay. It's not as if it isn't common knowledge that he's a monster. Would you rather live where the "Lord" starved your former liege lord, starts a fight over land he realistically shouldn't have a claim to, and hunts and rapes peasant girls all over his land? There's open negative talk of the Boltons in the Merman court, the wine sinks, Lady Dustin mentions it, and Roose chastises his son for sowing tales in Bolton lands. How much of the refugee population is due to that? I have no idea, but it's certainly something to take into consideration.