Sorry but it’s obvious the true answer here- the lemon tree Dany remembers outside of her room in “Bravos” is actually Lem Lemoncloak. Clearly Willem Darry and Viserys hired him as a guard to protect Dany, but she had dyslexia (just like her half-brother Jaime) and mistook “Lem” for “Elm”. Even though he wasn’t as tall back then, he was still taller than young Dany, so the nickname stuck, and it’s been a long time so now she just remembers him as a literal tree.
So no, she didn’t grow up in Dorme, but the Riverlands! It’s just a double fake-out that lemons don’t grow there either.