I think it's important to note that when Rhaenyra was named heir, Viserys didn't have sons or grandsons. And the throne was expected to pass to Daemon. Now I'm not saying it was the Greens idea to make Rhaenyra Queen in the first place but it was totally the Greens idea to make Rhaenyra Queen in the first place.
He'd have to go get those forces first and in that time Robert might attack or their force might disperse. They'd retreat to Harrenhal, Riverrun or even march on Highgarden. Rhaegar might not die when he did, but Stannis (holed up in Storm's End) and Walder Frey (late to the Trident) would both enter the war against him. And Tywin might follow.
1) Jaeherys I 2) Daeron II 3) Viserys II 4) Aegon I 5) Maekar I 6) Jaeherys II 7) Aegon V (great king but married for love and let all three of his sons do the same making him loads of enemies as Barristan would have it and in the end his desire to bring dragons back got the better of him) 8) Aegon III 9) Aerys I 10) Robert I 11) Viserys I (both Robert and Viserys loved feasts and left the rule to their small council who turned out to be filled with snakes, but both had the realms best intentions at heart, I put Robert first cause while his personal vices were bigger, much bigger, he did put Ned and Barristan and Jon in the small council, Viserys' government enjoyed no equivalents) 12) Baelor I 13) Daeron I 14) Aenys I 15) Aegon II 16) Tommen I 17) Aegon IV (I'm putting Tommen and Aegon first because while clearly incompetent kings, at least they were funloving incompetent, rather then "kill, kill, kill" crazy, no one would prefer to have the below as kings) 18) Maegor I 19) Aerys II 20) Joffrey I