Okay, first of all, when I found this series, I raced through(in reading of course) AGoT, ACoK and was halfway or so through ASoS when I came to know that GRRM took so long in writing AFfC and ADwD and I thought that he might take as long for the next book as well, and I decided to slow down considerably. While at the same time I got busy with my examinations. Yet, I could not help but resume the series at that same speed as before and before I knew it, I was finishing ASoS and now I am about halfway through AFfC. I was also a bit wary of getting into AFfC but not for the same reasons as you. I had only heard good things about it. That was not the point. The point was that the North had always had more fascination for me than south. That is perhaps because I always liked old things, history, castles, etc. and the North seems to have all that with the old traditions, the old gods, the old ways, blood of the first men, and the varied mysteries and objects, etc that I was drawn to. If we compare that with the South, we have so many cities, people, new gods and that is all man made. Nothing seems to be reminiscent of the old history(even though fictional). That is why i was wary, not to mention the fact that all my favourite characters were in the North, save perhaps for Arya(whose POV sometimes did get on my nerves, I agree, but still felt very promising) and the Imp(whom I liked only for his wits the most) and had all the characters that I kind of disliked(Sansa, Cersei, Joffrey who thankfully is no more, Stannis, Melisandre, etc).
But what I read on this site did give me some hope. They said that Sansa's character improves a lot, and I am starting to see the improvement. They also said that some new story lines are starting which are good, and I must say they have been very intriguing till now and have a lot of promise. Arya is not disappointing at all. The only thing that was being complained of was Brienne, and I must say GRRM is finding a lot to write about something that I would have thought would be difficult to find enough to write about. Anyways, it's not bad enough as some of the others that were there in the first three books.
Plus, many said that they did not like Bran in the first three books, which I must say is surprising for me because his were the best IMHO. And now the same people say that his chapters are fantastic as Lord Barristan(who I must say is very comically and ironically a squire here on the site, though I hope and my best wishes with you that you rise much further ahead) is testifying. And that is what has kept me going in a major way because as I loved Bran's chapters, I can not wait to get onto ADwD. Also I am loving Jon's chapters and beyond the Wall was so great!! And now he is the Lord Commander!! Wow, I want to see how he does as one.
In the end all I want so say is that I know that AFfC, and I think ADwD, are not as action packed as the first three(especially ACoK and ASoS) but I would like to compare the series with a hurricane. As you all must know, they first hit fiercely, and then the eye of the storm goes over which for some time creates a lull, a peace of sorts, and then finally the storm hits again, and sometimes more fiercely and definitely finally. So I am having a feeling that this is just the calm before the storm. A time for everyone to gather what little they have and brace for the storm, which gives GRRM some space for more story build-up. So yes, I would definitely recommend you to read on because if the little story build-up in AGoT and to some extent in ACoK(comparitively to AFfC and ADwD) brought about such a great action packed third novel, I can't wait for the final two books to be out, which are I assume A Winds of Winter and A dream of Springs which as the titles suggest are signs of a sort of a comeback for the Starks, because Winter is their time and the working title for the last book was A Time for Wolves, and we all know who the wolves are.
I just can not wait to get through to the final book, so much so that I am surprised you have been thinking of quitting the series after ASoS. Well, that's all personal I think, so I would leave it all upto you ofcourse, but I hope to have made you think atleast and be of even a little help.