I think what we are forgetting here is that a royal decree trumps all, right? Was Ramsay documented somewhere as being Roose Bolton's bastard? I don't remember having seen that anywhere. Yet, Joffrey issued a decree legitimizing Ramsay, just so Ramsay could kill his father and become Lord of Winterfell and Warden of the North, albeit temporarily. Jon Arryn, Ned and Joffrey all knew about Gendry, so it was either written down somewhere, or accepted as truth. Either way, if the reigning king/queen decrees that Gendry is legit and a lord, then he's legit and a lord/warden. With Robert being dead along with his "heirs", and Stannis and Renly dead without living heirs, who is around to contest it or say otherwise?