Season 4 still has a lot of material from SoS. Joffrey`s wedding, Tyrion's trial, Oberyn Martell, Meereen. They'll probably introduce the Greyjoys as well, and Euron should be fun to watch. I think the problem is season 5-6. On AFfC, nothing happens, really. They'll have to cut through that book really fast, because the audience will NOT like to watch Brienne, Sansa and Cersei doing absolutly nothing for 10 episodes. ADwD is a lot better, since the most appealing characters appear here (Jon, Tyrion and Dany), so its easier to form a season around these three. But again, there's really not much action going on in Meereen and The Wall, it's just Jon and Dany sitting around solving problems here and there. My opinion is they will have a really hard time coming up with interesting and action packed episodes on those 2 seasons, which unfortunetly may cause HBO to not have enough interest in making seasons 7 and 8 for the TWoW. Things get even uglier if you consider that GRRM may not be able to finish Book 7 by 2018 - which would probably mean the end of the series.
