I agree it does seem like a walking contradiction. Although House Strong and Whent seemed to flourish and be pretty powerfully until they each backed the wrong side of a civil war.
If Peter Baelish had say a decade of relative peace under a Tywin administration he might have been able to grow the local resources, wealth and manpower to match the paper strength of the title.
I always wondered if House Durandon had bent the knee to Aegon and Argillac had a male heir would Aegon have given Harrenhal to Orys Baratheon?