You're just plain wrong about Halo CE being a plot heavy game. I'm not saying that Halo CE doesn't have a plot, but I'm just saying as far as narrative structure goes it's a very simplistic outline of the Hero's Journey. The narrative structure of that game reminds me Nintendo game like Mario, Metroid, Zelda in that there is very little in the way of subplots or anything outside of what Master Chief is directly involved in within the levels. I'm not saying that the simplistic story telling in that game should necessarily be celebrated, but I think there's something to be gained from reducing the hero's quest down to it's most basic elements. There's certainly something to be gained from keeping all of the narrative within the world of the gameplay instead of requiring multiple external sources to understand the world.
I dunno man, Take a poll in this thread. Halo is pretty plot heavy, From moment zero the game is throwing plot points at you. Cortana is constantly giving details on the world as you explore and learn new things. Then 343GS pops up and starts going on about reclaimers and stuff. It's all very alluding and never giving definitive answers but it's constantly making the plot the focus because cortana is in your ear saying more and more about the world. The game has about 40 mins of cutscenes and is pretty much driven exclusively by the plot. Halo CE is laying lots of big groundwork and dishing out lots of plot to the players.
Sure there is something to be said for keeping a games story self contained but Halo is a unique position where it doesn't have to. They can produce fairly good efforts outside of games to extend the universe because it's microsofts golden child.
I'm fine with one game breaching into other genres, Plenty of other games are staying self contained. Some people clearly dig the expanded lore.
Your timeline regarding Halo 2 is wrong BTW. Yes it was announced in 2002 and yes the notorious campaign demo was shown in 2003, but it's misleading to suggest that the game that was announced in 2002 had anything to do with the final version that was developed following e3 2003. Bungie has admitted that following the e3 demo they had to scrap the entire graphics engine and start from scratch. It's obvious that that the plot had to be significantly altered as well. Again, I'm not saying that my thesis about the Iraq War having anything to do with Halo 2 carries any water, but it's ridiculous to suggest that this game was developed in a cultural vacuum. The tonal shift that occurred between Halo CE and 2 is significant enough that I think it's useful to consider the cultural era that the game was developed in.
The only real differences we are certain of is that Arbiter had his name changed, content was streamlined and the ending was cut eliminating a big chunk of the 3rd act which had chief return to earth, I doubt many story changes actually took place from 2002's Halo 2 concept. In fact the making of halo 2 backs this up by showing some of that cut content which was just extended cutscenes that did little more than exposition.
The whole concept of the Arbiter and the covenant civil war was very much an original draft thing. It's pretty obvious that the Arbiter was always planned for Halo 2 and so the Civil war was necessary so you didn't play the villain.