• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

343: Explanation of What Happened with the MCC & Why It Was Left Unpatched Until Now

This has popped up in a couple places such as Reddit and the Halo 5 OT from Frank O'Connor as the long promised explanation of why the MCC turned out the way it did. It's a long read but goes into detail on why things were as broken as they were and why things weren't eventually patched at the time.

reposting from waypoint:

As you saw on the livestream earlier on this week, it was a big relief to be able to share the news that we're fixing and indeed updating MCC. I was traveling this week and out of the office while the livestream was happening, but if you remember, a couple of years ago I promised to explain what the underlying issues with MCC were. Well here it is, sort of. I'm obviously not an engineer, so apologies to the deeply technical for keeping this at a level I understand.

I'd also like to be clear, up front about what's not contained here – there's no excuses. We're explaining some of the issues in more depth than we have before, in part because we now have the resources, OS and capability to make meaningful changes. So please don't mistake explanations for excuses – we're trying to be as transparent as possible, but there are loads of proprietary things we can't talk about at a granular level.
We will, however, have kind of a second half of this post when the update is released, where we can go through the causes of items that were fixed by the new update, rather than jinx them right now.

There's also no guarantees in here, beyond my guarantee that we care about this very much and are throwing our best people and best efforts into this project.

On Wednesday of this week, we announced that we're both fixing MCC and working on enhancements for the Scorpio (Xbox One X) version of the game, but I should be clear here, that in terms of chicken/egg scenarios, fixing the existing ”vanilla" Xbox One MCC was the Chicken that laid the Xbox One X enhanced version egg. Without the ability and opportunity to reconfigure and fix this thing, we wouldn't touch an Xbox One update. But a series of changes to the Xbox architecture, some of them related to Xbox One X – and others just a series of ongoing improvements to the OS and back end networking systems, have cracked open an opportunity we've wanted to seize for many, many months now. So to be super clear, these fixes will apply to both regular Xbox One version and the Xbox One X enhancements.

We're also getting a lot of help from the (wholly separate) Halo 5 team, who created a much more robust system for the launch of that game and continue to make improvements to their networking model.

From a personal perspective, the MCC launch was one of my lowest ebbs, professionally. Every angry mail I received, I took to heart. I felt like I had personally let our fans down. I have not spent a single day since the night the game fell down in matchmaking where I didn't think about it. The hardest messages to deal with were the ones driven by disbelief. ”How could you not know that matchmaking was going to break?" – fundamentally it was because we were testing it in an environment that we had set up incorrectly and with some (as we discovered later) faulty assumptions. And unlike some of our other normal testing cycles, we weren't testing for gameplay balance and stuff that the original releases already contained so our test process was radically different, and we made mistakes in some of the scenarios we asked for.


We had, with the best intentions, created a massive and ambitious project that almost read like a Halo fan's wishlist. As a player, I was incredibly excited. And as an employee, I was proud of the work and effort the team had poured into making this thing so big.


It initially started as a conversation about making a Halo Anniversary 2 – we thought about simply replicating what we'd done with the first Halo: Combat Evolved Anniversary, a polished update with some cool new graphics and features. But we kept talking about it – and the conversation inevitably led to the ”problem" of a franchise existing over multiple generations of hardware. This was built for Xbox One – and prior games were spread across 360 and OG Xbox. So we figured, why not finally put the whole Chief saga on one console? We wanted everyone to be able to enjoy his entire story.

And so the project ballooned in scope and scale and ambition. We threw a ton of resources behind it internally and worked with some trusted partners.
In our matchmaking testing we were seeing results that ultimately weren't reflective of the real retail environment, and our test sessions never got to the kind of scale where we'd see some of the looping issues I'll describe below. So we genuinely didn't know until the day it released, how bad the matchmaking in particular was going to get. I'm not going to ignore the other bugs, they were real, and important, but the way the UI and matchmaking protocols interacted with each other exacerbated many of the smaller items and amplified a couple of them in unpredictable ways.
The short version was that for Xbox One we built some of the underlying systems to work on a brand-new platform, which was fundamentally, quite different to both the original consoles the games were designed for. We also had some very new (and frankly these have evolved since then and are now much better) online systems on a new console and made some educated, but (with hindsight) ultimately faulty, assumptions we made during development and testing. To be clear here, the platform networking model was working as intended, but we made errors and ultimately approached it with the wrong strategy.
Frankly, we don't assume anything anymore. While we had some valid reasons to believe the game would function properly in the retail environment, we've shifted our development philosophy to basically assume nothing anymore. And one way we're going to avoid that in future is through a retail flighting program – testing the game fixes in a real-world environment with real players, including many of you. Naturally we'll also be doing much more rigid conventional testing with the benefit of both hindsight and new, better systems.

One of the main matchmaking issues was related to the way that the games gathered players – each title had some differences in how it sought out players, then connected them into sessions. In an attempt to unify that method, we actually introduced a bunch of (with the benefit of hindsight) several avoidable problems and some unavoidable ones. It gets really technical, and this is as much metaphor as technical explanation, but each potential player was assigned a kind of ”ticket" which would then grant them entry into a match or session – picture a virtual waiting room at a train station – when the train arrives (a match) – everyone has to board - or the train can't leave. Issues arose when folks left sessions before games had started that would cause the initial ticket distributions to fail, and that sometimes meant very long wait times for matches as tickets were issued and reissued – especially in countries with lower populations.

Now the above isn't particularly unusual or original in terms of approach, but at the time the systems were less resilient in terms of churn, and bad information could cause a lobby or match to get caught in a state where it couldn't ultimately complete a group and join them cleanly into a session.

At the time we made tons of changes to the backend server configurations to try and reduce those wait times, but ultimately it was a self-fulfilling prophecy – players understandably would leave sessions because they got tired of waiting for a match to begin, and that would amplify the issue across the board.
But there were other issues that compounded the noise and frustration players felt. For example, there's a good-sized subset of our population that has issues with Teredo, IPSEC and NAT compatibility that we simply can't troubleshoot or identify, and some of those users are encountering issues that are literally beyond our control – trapped behind corporate or academic firewalls. These feed into some of the areas we're planning to improve, and ongoing improvements to the Xbox systems have improved some of those issues, but not all.


We've loosely explained this over the last couple of years, but I'd like to reiterate here. We ended up in a situation where the game was working for the vast majority of users. That's not the same as perfect, or for some, even acceptable and that's not what I'm trying to claim. Many people who complain about the game these days have legitimate issues with matchmaking and other aspects, and we're not going to dismiss those – so even though for most, the game is stable, and the sheer wealth of content and experiences makes it – to this day – a highly rated title by players - we get it. If you're one of the people affected, seeing a statement from us that it's working for ”the vast majority of players" is cold comfort. When you're the one affected, it's as good as 100%.

The fixes and patches we'd applied were pretty delicate and we ended up in a precarious situation where there was no way to make more fixes without potentially breaking something else or making things worse. We weren't happy with that situation, but we were stuck between a rock and a hard place – most users were (by this time) able to play properly and find matches, and further tinkering might put that at risk. At that time we decided the right thing to do for the total player base was to stop. That was hard to do, especially knowing there were still some customers impacted more seriously than players who were merely inconvenienced.

But that didn't stop us being concerned about it anymore. On the contrary, in some ways leaving it was worse. I mention this not to garner sympathy, we deserve none, but to answer folks who've continued to ask, ”Why don't you guys care?"

We do. Everyone here puts their heart and soul and sweat and tears into building our games. I can tell you without hesitation that I have never heard someone here dismiss or ignore or belittle complaints. We always take them to heart. It's the internet of course, so sometimes folks take it too far, with threats or other inappropriate reactions, but I'd be lying if I said I didn't understand the anger or disappointment those came from.
So over the months we discussed and investigated other fixes. The platform itself has made some truly evolutionary improvements to its underlying technology, and recent fundamental changes mean that we might have the opportunity to make some fixes without risking everything else.

It may sound simplistic, but MCC was essentially six pretty different game engines strapped together and interlinked with highly complex and highly delicate new systems. With Xbox One X on the horizon, it was obvious that we could simultaneously update the game to take advantage of the new hardware for folks that have it and use that as an opportunity to finally rearchitect and update some of the foundational issues and networking/matchmaking methods.

And to be clear, these solutions were simply not possible until quite recently. The platform team has made numerous improvements over the last year or so, and we've internally done a bunch of research, and so our timing has been reliant on a number of systems and solutions converging rather than one single element. But these weren't easy fixes we were simply sitting on. That's honestly not a thing, even. I also understand that silence can be frustrating. You have complaints or questions, and we try to answer them as best we can, but sometimes bad information is worse.

As I said at the start of this explanation, it doesn't answer all your questions. I'm going to follow up next year after we have better detail on the fixes and the Xbox One X update, to follow through with an even more detailed technical breakdown of what broke, why and how we fixed it. That's what we owe you – that and a game we can both finally be satisfied with.
 

Inuhanyou

Believes Dragon Quest is a franchise managed by Sony
Better late than never Frankie, just get it right this time for the fans. You guys knocked it out of the friggen park with the remaster of Halo 2's campaign(you guys made halo 2 actually jump to tying with halo 3 as my favorite halo, and i never would have ever imagined that with bungies original), it was a shame that the rest of the package was such a failure of quality that it dragged Halo 2 along with it.

I wish you could go back and somehow give halo 1 the same production values you gave halo 2, even though i know that's impossible now. Halo 1 anniversary was such a mixed bag for me, only reason i play it now is cause halo 1 doesn't play well with 360's BC.
 

gamz

Member
That was a great and detailed explanation. Fantastic stuff and appreciate the transparency.


Btw - I'd rather have you guys swing for the fences everytime out. It's was ambiguous as hell and I'm sure most of us appreciate it.
 
That extra kinect overhead!!! /s

Hope the update works out. It's an ambitious and unrivaled collection, but it doesn't mean much when it's not reliable. Would like to see MCC become the marquee piece of software it should've always been.
 

HonMirin

Member
I have a feeling this is going to be a DRIVECLUB scenario. Sounds like issues have been identified and worked in to improve. I really hope they manage to pull this one back and make it the experience it deserves to be.
 

Jumeira

Banned
Thank you for the honest and sincere explanation Frankie. Whilst i own MCC i didn't play online that much I completely understand where the fans were coming from, in regards to thier frustration. Halo 5 matchmaking and online was flawless (perhaps the smoothest ive ever played) so i know 343i have learned from their mistakes. Appreciate the company are staying true to thier word to sort this out and have made fundamental changes to their workflow to prevent this happening again, thats all we can hope for. Cheers
 
It's important that they fix TMCC for the legacy of this franchise. It was going to forever remain a stain on Halo, even going as far as to negatively impact future releases. Consider how many people stopped caring about Halo after TMCC and had no interest in Halo 5 as a result. However, now there's an opportunity to correct things and show one of gaming's strongest franchises the respect it deserves.

On top of that, it's just great to revisit older games, let alone in 1080p/60fps with online support and all accessible from one unified system. If you're a fan of the franchise, you've no doubt had that itch to play classic Halo, and TMCC allows for that. The significance of this cannot be understated. Every Halo game playable on modern Xbox platforms. That's insane. Imagine being able to play every Mario and Zelda game on Switch, that's what TMCC is for many people.
 

Vinc

Member
Better late than never is right, and I say this with the least amount of sarcasm possible. This game was and has since been my most anticipated Xbox One game ever. Among other reasons, its inability to work properly made me get out of the Xbox ecosystem entirely for the first time since 2003. This might be a first step towards bringing me back. Microsoft needs to invest far, FAR more in first party though.
 
I hope they can get it to a near perfect state. I need a reason to own an xbox again and would definitely put periods of time into replaying the campaigns again and the MP of 2 and 3.
 

malfcn

Member
Better late than never. Hopefully it gets acknowledged. Thanks Frankie, and 343.

I was able to enjoy 1-3 with my best friend. There were minor and major hiccups along the way. But overall we had a great time.

Games with Gold?
 

Fracas

#fuckonami
It's cool that something is finally getting done but jesus, this game came out three years ago. I can only speak for myself but that ship sailed a loooooong time ago and with it, any passing interest I had left in the series.

I bought then later sold an Xbox One because of this broken mess.
 
Insanely ambitious package that I had a mostly great time playing through with minimal issues (I didn't bother with mp too much though). Glad they're finally (hopefully) getting things fixed.
 

enemy2k

Member
Thank you for that detailed explanation. Looking forward to the second half you mentioned coming with the update. I'm willing to give you guys a chance to redeem yourself with the MCC update.

I will reinstall as soon as it's patched and can't wait to see the Xbox One X enhancements when I upgrade from the One S.
 

TheXbox

Member
Very, very much appreciated. Frankie is a sincere guy and I love the fact that he sticks his neck out on here to address the playerbase. I understand and believe him that these network and matchmaking issues were unintentional.

But.

The game still shipped with a ton of other crippling bugs. Faulty matchmaking, as a whole, is only one of many. Broken co-op. Broken saves. Hard crashes. Framerate issues. Countless new glitches introduced by the new ports. Even if matchmaking had been 100% functional, the Master Chief Collection still launched in a dysfunctional state. I mean that literally -- the game was not functional.

I do not believe that these other issues escaped their testing environment unnoticed. If they did, then the sheer negligence beggars belief. If they didn't, then they knowingly shipped a broken product and asked consumers for $60 on a false promise. On that account, the opportunity for amends passed long ago.

edit: btw, I'm glad they're still committed to new fixes. I don't want to take that away from them. But misleading customers is a far more severe transgression than goofing on your matchmaking network.

edit 2: last one, for clarification. I'm very much aware that every publisher does this, to some extent. Games get launched with bugs, and not every ticket can be addressed before ship. That I understand. The problem with MCC was that the bugs were so numerous and so all-encompassing, even setting aside the matchmaking, that I believe shipping the game in that state was irresponsible.
 

Surface of Me

I'm not an NPC. And neither are we.
So if I wanted to play some multiplayer right now does it work?

Only way to find out is to try. It works for some, not for others. Also take in to account that there are likely few people in the matchmaking pool these days.
 

VariantX

Member
If they get this working properly, I'll get an Xbox one X sometime in 2018. MCC being bug ridden and having those matchmaking issues are precisely why I don't have an Xbox One even though I had the disposable income to pick one up. It was one of the defining reasons for me to even purchase one outside of my interest in sunset overdrive.
 

Zeta Oni

Member
Very glad to hear it.

I might have different individual games I may place higher, but Halo has always been my favorite franchise and TMCC situation ended up really effecting my outlook when it came to the future of the series.

Knowing they are returning to it to do what's right and fix things might not bring everyone back, but it gives me more confidence in 343 and where they go from here.

On a side note, I really hope they re-do the UI and just bring back the old main menu screens for you to switch between like this.
 
Man. Imagine how different thing would have been if we just got a really nice halo 2 AE in 2014. That game has such a special place in my heart.


Anyway wasn’t the original ntkrnl leak that included so much other stuff that turn out true the plan was at one point to release each game separately over a year as a bust to 5. I wonder how that could have gone.




Anyway. Happy they are finally doing it. It is going to be a hell of a big patch huh.
 

borges

Banned
If everything goes ok, game would be re-released as a definitive edition, not just a patch for existing customers. That would help to increase population as well.
Just dreaming a bit, but.... what about an E3 2018 presentation announcing a free-for-all gold/pass members MCC Definitive edition, with proper 4k, HQ textures benefits for X users?
 

TheXbox

Member
I played through all of the SP of a non-functional game at release then.
I'm glad.

We ended up in a situation where the game was working for the vast majority of users. That’s not the same as perfect, or for some, even acceptable and that’s not what I’m trying to claim. Many people who complain about the game these days have legitimate issues with matchmaking and other aspects, and we’re not going to dismiss those – so even though for most, the game is stable, and the sheer wealth of content and experiences makes it – to this day – a highly rated title by players - we get it. If you’re one of the people affected, seeing a statement from us that it’s working for “the vast majority of players” is cold comfort. When you’re the one affected, it’s as good as 100%.
Many, many others did not. The launch issues are well documented. Peruse any thread on GAF, reddit, wherever from those early weeks, or take a look at the laundry list of fixes accompanying all the patch notes.
 
There is absolutely no way they did not know the game was broken on so many levels prior to release with the plethora of problems it had. Furthermore insisting that a still glitchy frustrating mess of a multiplayer system “works for the vast majority of people” is frankly offensive.

The truth of the matter is they relied on this game to push sales for their platform when it was in serious trouble and didn’t mind pushing it out broken despite the long term damage it would cause to the franchise just to prop their sales up in the short term.

It’s also a bit ironic that it is now being dragged out again and to push yet another new platform with promise of “we will fix it this time guys we swear” in hopes of once again banking off the brand recognition of the franchise to push their new HW and so few seem at all upset that they were perfectly fine with letting it stay broken until their next piece of hardware came out. That tells me they couldnt justify spending the money and resources to fix a broken product that they sold to their customers until they could exploit the positive word of mouth again. Yet people here are lining up to praise them and give them their money yet again.

Until this game is patched and working 100% without issue then excuse me if I take anything the company that released a broken mess and proceeded to not fix it and proclaim “actually it’s working for the vast majority” with a massive grain of salt. No amount of extra pixels or frames is going to change the fact that they sold consumers a shit sandwich and told them “well it tastes fine to the vast majority of other people.”
 
I'm happy for games that get fixed after launch. It's too bad they launch that way for whatever reason, but if they do fix things, I also believe you should at least see what they fixed and if it does improve.

If they fix it correctly.
 

UraMallas

Member
Many, many others did not. The launch issues are well documented. Peruse any thread on GAF, reddit, wherever from those early weeks, or take a look at the laundry list of fixes accompanying all the patch notes.

Vocal minority is a real thing.

I wasn't on message boards screaming about it not working - I was playing it at launch.

He acknowledges the launch issues so I don't think you have to document that any further than the OP.
 

AZ Greg

Member
Good on Frankie for finally talking about it as promised.

I've been one of the biggest critics but that's due to Halo 2, and the experiences I had with it back in 2004/2005, being some of my best gaming times.

Here's to hoping the fixes deliver. And that the title can get some momentum behind it and have a nice player base.
 
Thanks for explaining it, and if we're being honest it was MCC that destroyed my, and my 4 friends excitement in Xbox One, pushing all of us to drop it and get PS4s to this day.

Now with that being said, I would pick up an Xbox One again (Or an Xbox One X) If I can get everyone to join me again. MCC sounded like the funnest game ever to me and all my friends, and I still think it can be, and I hope you are able to work everything out.
 

hollomat

Banned
There is absolutely no way they did not know the game was broken on so many levels prior to release with the plethora of problems it had. Furthermore insisting that a still glitchy frustrating mess of a multiplayer system ”works for the vast majority of people" is frankly offensive.

The truth of the matter is they relied on this game to push sales for their platform when it was in serious trouble and didn't mind pushing it out broken despite the long term damage it would cause to the franchise just to prop their sales up in the short term.

It's also a bit ironic that it is now being staged out again and to push yet another new platform with promise of ”we will fix it this time guys we swear" in hopes of once again baking off the brand recognition of the franchise to push their new HW and so few seem at all upset that they were perfectly fine with letting it stay broken until their next piece of hardware case out. That tells me they couldnt justify spending the money and resources to fix a broken product that they sold to their customers until they could exploit the positive word of mouth again. Yet people here are lining up to praise them and give them their money yet again.

Until this game is patched and working 100% without issue then excuse me if I take anything the company that released a broken mess and proceeded to not fix it and proclaim ”actually it's working for the vast majority" with a massive grain of salt. No amount of extra pixels or frames is going to change the fact that they sold consumers a shit sandwich and told them ”well it tastes fine to the vast majority of other people."

This 100%. This apology and promise to fix it is meaningless now. If it wasn't possible to release it without those issues then, it never should've been released in the first place.

I was so excited for MCC and really looking forward to playing some Halo 2 and Halo 3 multiplayer. MCC was the main reason I bought an Xbox One. Then I bought it and it was a buggy unplayable mess. But I had already bought the Xbox One, so they fooled me.

Between MCC, Halo 4 and Halo 5, 343 has completely burned all the goodwill I have for Halo. Every Halo game was day one for me. Now I'll never buy another 343 game. If its a buggy mess, I/m sure players can look forward to an apology years after the fact and a promise to fix it years after the fact.

Also a big slap in the face to say "most users were able to play no problems". That wasn't the case at all.
 

jviggy43

Member
Cool. I'm happy to hear this. Ive been critical of where this was but if this gets up and running I'm back in. Transparency goes a long way
 
Stinkles is the man. MCC was a case of biting of more than you can chew and I bought into the bite and left bitter and burned. It's too late for me, I sold my One and do not plan on purchasing an X, but I still hope this turns out to be a story of redemption. 343i have talent and passion, so I hope it all works out well. Halo is one hell of a thing to me.
 
Don't think I'll ever go back to online multiplayer but it should make some people happy. Kudos for the transparency, even if it is long overdue.

Fixing it will be worth it just for the game preservation aspect alone, if the next generation or two are going to be backwards compatible then having all the games in one place will be pretty cool.

They shouldn't have released it broken in the first place.

Fucks sake, at least pretend you read the OP.

There is absolutely no way they did not know the game was broken on so many levels prior to release with the plethora of problems it had.

It's been so long that my memory might be wrong but didn't reviewers play through the game without having any of the matchmaking problems prior to the launch? It wasn't until the game went live for everybody that it was clear the game was broken but like I said, I'm not 100% certain.

As for the rest of your post, if you want call Stinkles a liar, don't waste so many words on beating around the bush.
 
This 100%. This apology and promise to fix it is meaningless now. If it wasn't possible to release it without those issues then, it never should've been released in the first place.

They didn't know about the issues until after release though so how could they have?
 
Top Bottom