Sir Loras is a badass, yes. Ser Loras beat the mountain in Ned Stark's hound tourney. If Ser Loras can beat the mountain, he has a chance at beating Jaime. Ask yourself this: How much bigger is the mountain than Loras? Then this: how much bigger is Jaime than Loras?
The show has painted him as a pansy who likes men, but don't let your prejudice against homosexuals and his defeat by Brienne make you think he isn't a respected dualist and fighter. Brienne is a beast and a beauty, but being beat by her is only a shame in the extremely sexist pre-enlightenment culture that medieval westeros is set.
It was always clear that Jamie respected Brienne's size and form, he just never respects her femininity. He wants to fight her to prove that he can beat the strongest woman... and then go on to beat the strongest man.
Oberyn is cool, sure. He is not anywhere on the radar of best fighters ever though... because his family is backwater Martell that was close to the "mad king".
Doran is a cripple, his heir is a woman, and the leading man is "prince frog"... underestimated to be sure... but still the Martells are not held in high regard either in book or show and Jaime was not referring to any living Dornishman.