I think you forgot the picture of the gems hanging out with the pirates on a ship(The hard part was getting the shark to hold its position). They've been on earth a lot longer then Mr. Universe has been alive. Also, I think most, if not all, the places they teleport to are on earth as well some of which suggest gem based civilizations that had been a round for who knows how long.
I totally did forget about the pirate ship, you're right. That doesn't necessarily disprove my theory though, they still came to earth and fell for humanity, then eventually Rose met Mr Universe.
Same with second point, forgot that Rebecca had said that it's an alternate universe earth where the Gems have been part of civilization to the point where humanity isn't really plussed by them anymore.
But again, that doesn't totally discredit the theory that they came to Earth, saw humans, found them to be worth saving, and set up camp. Gem Leader didn't like this group of renegades deviating from the plan, came to earth to capture them, giant battle, Gems stay behind on earth, Rose eventually meets Mr Universe, etc.
The Gems don't age so when they initially came to earth doesn't really affect this theory, and I guess when the battle happened doesn't matter either just as long as it happened before Steven was conceived.