[BOOK SPOILERS] Varys' allegiance

[HodoringHodor]

Long time lurker, first time poster here. I was trying to find a discussion on this topic but since there was none I figured to register.

From the books we know that Varys is a Targ (or Blackfyre) loyalist and he's said he doesn't approve of magic because of what happened to him. But we also know he might very well be lying.

There was something curious in the final episode of 4th season concerning this. The scene in which Varys is watching Tyrion being loaded into the ship is shot with the camera focus going slowly down from the crate up high until Varys is the focus of the shot. Camera's movement stops just when Varys' back is shown, lingers for a second and cuts away. In that scene the robe Varys is wearing has a large heart embroidery in the back and the camera movement stops just when the heart is fully visible. The other decorations on the robe make the heart seem like it's ablaze.

In the beginning of the episode we see the R'hllor's burning heart symbol when Stannis' forces attack the wildlings. Because of that it's difficult to attribute Varys' clothing as an oversight.

Since D&D know where the story is going, could that short moment be a hint of where Varys' allegiance really lies? The shot of his back seems very deliberate.