I've put 200+ hrs into Oblivion and Skyrim and am pretty much lost when it comes to the lore. TES does a really bad job at furthering the lore through its plot and gameplay.
I've played Mass Effect, Ultima, Fallout, plenty of other RPGs, and I was able to get a pretty deep understanding of all the races, leaders, ambitions, territories, etc. just by playing normally.
TES is a freaking mess. There, I said it!
Sometimes I do pick up a random book and read it, and I think, "that seems interesting," but it's volume 7 and none of the other volumes are around, so what the hell am I supposed to do? Plus the game puts all its information at the very periphery, so it's hard to get to it without going out of your way. Which can make it rewarding, but it's handled better in other games where the lore comes out through quests and character interaction.