You'd ignore your son for years and then expect his mother and sisters to have made him a manly man? After Sam, he prioritized spending time with Dickon as he saw what happened when he wasn't involved.
I think the shady reference of Swann's task in AFFC is enough to view it as true. Even though we are in these characters heads, GRRM still can't just let us know everything, our it would kill most of the suspense.
To be fair, surrounding yourself with competent people who are experts in their field is a hallmark of great leadership. So at least early on, you can absolutely argue that Aerys was a good king, even if Tywin did the heavy lifting. Even after the rebellion, Robert took over a kingdom with a fully stocked treasury, full harvests, and the smallfolk had a chance at living a decent life.
Getting mad over Dany killing and double crossing the Astapori would be like getting mad at an undercover cop who killed a pimp who is trafficking sex slaves. She gave the unsullied their freedom, and they chose to follow her. Dany has done some bad stuff (the Yunkish envoy), but some people stretch to paint her add the villain some times. The House of the Undying was trying to capture her and she was saved by Drogon, so to try to paint that as violating guest right there is ludicrous.