• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Kotaku: Debugging A Massive RPG Like Skyrim Means Leaving Some Bugs In

Link. Full podcast available.

When a company like Bethesda is trying to debug a massive game like Skyrim, where do they even start? Skyrim lead programmer Brett Douville tackled that question (and much more) on this week’s episode of Kotaku Splitscreen.

Here’s an excerpt from our conversation:

Jason Schreier: Something that I bet is a really interesting topic of conversation at Bethesda is how to approach bug fixing, because there are a lot of conversations about all the weird, hilarious bugs in Bethesda games, and some people point to the engine as a problem. Some people say, ‘Oh the Gamebryo engine, they need to fix that, change that,’ even though I’m sure it’s changed drastically from game to game anyway. But I’m curious—when you’re making one of these massive RPGS and you know there are gonna be a lot of bugs, is it just a matter of sitting down and triaging and deciding, ‘Alright, we’re not gonna bother fixing all these little graphical errors?’ How do you even start when it comes to the scale of that problem?

Note: “Triage” is a process in which game developers categorize bugs based on their level of priority. As a general rule, top priority is the ones that break the game.

Brett Douville: Well it’s definitely triage. You lean on your QA department, and certainly we did this at LucasArts as well, where you say, ‘OK look you need to tell us what’s important, because there’s this much time between now and ship, period.’ And we may have our own ideas, certainly, and that’s part of the conversation—‘No we want to fix this and not this.’ There are bugs you’ll just leave, like who would get rid of the Warthog stuff in Halo? That’s really fun. The Warthog jumping whatever, that’s a bug, it’s ridiculous, it looks silly, and yet it’s hilarious and fun, so why not leave it?

We tended to fix things that were obvious and egregious, and particular could impinge on a player’s ability to continue enjoying the game. If something was a little wonky-looking or whatever, eh. The tradeoff there is letting designers go hog wild and put whatever in the game, and we’d far rather have that than not. I can imagine making the games much more production heavy, ‘polish polish polish,’ sort of like Blizzard does, for example. They release games a lot less frequently, in part because, you read about their production process and it’s like, ‘Well every six months we just stop and we fix everything, and we make everything look perfect, and then we throw out all the balance stuff and start over.’

It would take forever to make a Bethesda-style game that way. It might never finish. Because they’re that big and complicated, and the combinations are so vast of things that can happen. So I think that’s at least a partial answer.

Jason: When you guys were starting work on Skyrim, did you have any crazy ideas that didn’t make it into the final game?

Brett: That didn’t make it? I can’t think of any crazy ideas that didn’t make it in. I think maybe the building-your-own-base stuff that’s in Fallout 4 might have been something that was discussed back then. I feel like that was a thing that was discussed going back to Oblivion as a possible thing to allow to happen in the games. I don’t think it would’ve been particularly possible on the consoles of that generation. We were already scrapping for memory, as you can tell playing the games. Adding this big, user-generated content thing on a console would not have been something I think we could’ve done.

And so that’s probably one. But honestly I think dragons were kind of the flipside of that. Nobody expected them to turn out as well as they did. Other games that had dragons prior to that had not had them be so central. They always felt, to me as a player they always felt a little underwhelming. And when David Diangelo and some others brought that into the game, we were like, ‘Oh man, these are good.’ [Director] Todd Howard’s talked about that before, about how you bring things into the game, and you play with them, and you as a result decide to put more of a spotlight on it than you had planned to. And that’s really the magic of Bethesda games is they throw a ton of stuff at the wall, and the things that stick, they just push really hard. And the things that don’t stick you just roll back a bit.

The other thing, and this was something that even made it to a mention at E3, was there had been thoughts of this vast economy and the Civil War stuff actually impacting individual town economies and things like that. And we had tried it and it wasn’t all that fun—you might have an opposite faction come in and mess with a town and now you can’t buy stuff at a vendor, and it’s like, who wants that? It’s not that kind of game.
 

bomblord1

Banned
Maybe they need to take the approach Nintendo had for Breath of the Wild. Large all staff on board play test sessions after any major changes to check if anything has broke. They seem to imply that would be impossible for a large complex game like theirs but Zelda is also incredibly large and complex as well. Seems like their workflow could be somehow improved to get more polish than is currently in there given other large and complex games exhibit a higher level of it.
 

AEdouard

Member
Frankly Fallout 4 or non ps3 versions of Skyrim were perfectly acceptable bug wise to me. The main issue now is that their ancient engine makes butt ugly games. They should take their mountains of money and do something about that.
 

Luigiv

Member
Translation: Tons of bugs haven't hurt our games' sales in the past so why should we spend more money on QA than the bare minimum to meet our prior standard?
 
Is comparing it to Zelda really apt? I imagine a game built for multiple consoles from different manufacturers along with PC that has a base seeking the tools to alter the game themselves has different caveats than developing BotW.

Not to mention Zelda is a different genre.
 
Translation: Tons of bugs haven't hurt our games' sales in the past so why should we spend more money on QA than the bare minimum to meet our prior standard?

Yeah and frankly I don't blame them. People will still buy their games in droves and reviewers never seem to give a shit until after the fact. So why should they spend time and resources improving their tech when it hasn't been detrimental to them in anyway.
 

KJRS_1993

Member
Frankly Fallout 4 or non ps3 versions of Skyrim were perfectly acceptable bug wise to me. The main issue now is that their ancient engine makes butt ugly games. They should take their mountains of money and do something about that.

Absolutely this. Oblivion blew my mind for visuals when it first came out. Skyrim was acceptable enough but Fallout 4 looked like a right rotter in comparison to its competition. They've definitely fallen behind on looks.

Translation: Tons of bugs haven't hurt our games' sales in the past so why should we spend more money on QA than we need to?

Maybe previously, but I'm not so sure now. Look at the slating in review scores (and sales hit) that Andromeda took. I'm wondering if reviewers will be as forgiving post Breath of the Wild now the bar has been set so high.
 

tsundoku

Member
Maybe if you haven't been making the exact same game in almost the same engine with bibles of qa and bugs to reference for like a fucking decade
 
It's ancient! It's ugly! I don't know which is worse! ♪

(The bugs don't bother me very much in Bethesda games. Is it a poor standard? Yes. But they've managed to make a brand for themselves that includes this kind of jank and now it is what it is.

My only concern is building up the expectation that games of this size can't be well-polished or well-animated or have good performance. Because they absolutely can.)
 
Thats a pathetic reason when Xenoblade Chronicle X and Breath of the Wild exist though.

I think they're different types of games though. The persistent simulation aspects of Xenoblade are non-existent, and in Breath of the Wild they're isolated (and the world constantly resets anyway). Of course that raises the question of why Bethesda need to model so much global stuff in their games anyway - the only time you ever really notice it is when something goes wrong, and some NPC disappears off the face of the earth because they've got stuck on the geometry somewhere during their daily routine.
 

Jonnax

Member
Translation: Tons of bugs haven't hurt our games' sales in the past so why should we spend more money on QA than the bare minimum to meet our prior standard?

Yep.
Quite a few games have come out that have outclassed them in different ways from your Witcher 3, Breath Of the Wild and so many other games.

And these games haven't had anywhere near the same level of bugginess at launch. Because they can't rest on their laurals knowing that they have the general consumer mindshare like Bethesda.
 
For one it's made for PC with a healthy modding community that wants to get into its guts themselves. Zelda has no such thing even close to it.

I don't see how that's relevant or how that would cause the game to have more bugs. The console versions of their games are still ridiculous buggy. And plenty of games made for PC with modding communities manage to not be bug riddle messes.
 
I don't see how that's relevant or how that would cause the game to have more bugs. The console versions of their games are still ridiculous buggy. And plenty of games made for PC with modding communities manage to not be bug riddle messes.

Because Bethesda obviously builds their games with the intent to make them easily alterable, both by themselves and others. That comes with completely different levels of complexity than something like Zelda where it's a closed system.

Edit: This by no means is meant to be an excuse for Bethesda. It is perfectly fair to ridicule and criticize them on their bugs. But do so with comparisons that actually make sense.
 

Corpekata

Banned
What do they do differently that would cause their games to have significantly more bugs?

For one, dozens more actors on schedules that are more important than the average NPC in either game and dozens more items to interact with, as well as persistence? There's a reason Skyrim PS3 got fucked, and that was because it was trying to keep every dragon corpse in memory.

They're pretty different approaches to the open world RPG.
 

Shaanyboi

Banned
Maybe previously, but I'm not so sure now. Look at the slating in review scores (and sales hit) that Andromeda took. I'm wondering if reviewers will be as forgiving post Breath of the Wild now the bar has been set so high.

I'm gonna say a significant majority of people buying Bethesda's games don't care about acouple points being docked from a score because of technical issues, as every single one of their games is a buggy mess, even in their best state.

Thats a pathetic reason when Xenoblade Chronicle X and Breath of the Wild exist though.

Uhhh these games are superficially similar to Bethesda's games at best.
 

eot

Banned
I don't think anyone is expecting them to be bug free. There's still a fuzzy line between what's acceptable and what isn't though.
 

Striek

Member
Are people serious with BOTW comments? I loved it, its a big game, but it has like 1/10th at very best the content in terms of quests, NPCs, items, variables etc. that Skyrim has. Its a sandbox, but theres nothing really persistent in the world Link can really impact.

Edit: I'm way late with this reply lol. Should've refreshed. Still doesn't excuse Bethesda, but they are really the only company that makes games as complex as they do. Even Witcher 3 is extremely simple in comparison.
 

Shifty

Member
Came in thinking 'another Schreier article lol', left thinking 'yup fair enough makes sense'.

Good stuff. I'm as hard on Bethesda as the next person for the bugginess of their games, but in fairness the only major bug-related memory I have from Skyrim is of having to ragdoll a frozen quest NPC up some stairs with unrelenting force in order to trigger progression.
Could have been worse.
 
Is comparing it to Zelda really apt? I imagine a game built for multiple consoles from different manufacturers along with PC that has a base seeking the tools to alter the game themselves has different caveats than developing BotW.

Not to mention Zelda is a different genre.

It's not, but everyone here is an armchair game designer and thinks they know what they're talking about. Zelda is a complex, systems-driven game, but not even close to the extent as Bethesda games. Just as one standout example, everything in BOTW despawns as soon as it gets outside of a certain, rather small radius around the player. There's no permanence to anything in BOTW. Not so for Beth games. And the amount of interactibles and AI actions are much greater too. There's just a much larger amount of possible interactions. That's not a qualitative statement about either game, it's just a fact.
 

weblaus

Member
Since Bethesda doesn't even bother to fix stuff that is clearly broken, but would likely be rather easy to fix (like the messed up surround sound on PS4 Skyrim - on Xbox it's fine), I really don't believe anything from them. when it comes to their RPGs
 
I think it's fair to shit on Bethesda's terrible writing and lazy quest design.

But on the technical level the games are marvels despite the bugs. Yes, you read that right, marvels.
Their titles are simulated to the degree very few other titles are, on top of being moddable to hell and back.

Both the simulation and mod support bring with it shittons of bugs.
It's a compromise I'm willing to take and wouldn't have it any other way.
 
It's not, but everyone here is an armchair game designer and thinks they know what they're talking about. Zelda is a complex, systems-driven game, but not even close to the extent as Bethesda games. Just as one standout example, everything in BOTW despawns as soon as it gets outside of a certain, rather small radius around the player. There's no permanence to anything in BOTW. Not so for Beth games. And the amount of interactibles and AI actions are much greater too. There's just a much larger amount of possible interactions. That's not a qualitative statement about either game, it's just a fact.

Are half of those interactive objects worth having in the long run? Or would their removal make it less of a 'Bethesda' game?

Full disclosure: I do not play Bethesda games and have no clue as their respective quality or what people really enjoy about them so I'm just curious.
 

Sayad

Member
Thats a pathetic reason when Xenoblade Chronicle X and Breath of the Wild exist though.

And MGSV, GTA, Red Dead, Horrizon... Even Ubi-Soft open world games are less buggy. AE are closing in fast on Bethesda games with Andromeda though.
 
Are people serious with BOTW comments? I loved it, its a big game, but it has like 1/10th at very best the content in terms of quests, NPCs, items, variables etc. that Skyrim has. Its a sandbox, but theres nothing really persistent in the world Link can really impact.

Edit: I'm way late with this reply lol. Should've refreshed. Still doesn't excuse Bethesda, but they are really the only company that makes games as complex as they do. Even Witcher 3 is extremely simple in comparison.

No, BoTW has a ton more variables and things that could go wrong with its chemistry system. Skyrim doesn't even come close to the level of physics in BoTW either so I don't see any good reason for so many geometry bugs in the game as well. Fact is the systems programmed into these games are poor, one of the big ones seems to be their spawning system.

Bethesda simply needs things to be cleaner under the hood of their engine. Best way is to start from the ground up or to strip it down to its bones and reprogram with a more modern architecture in mind.
 
Are half of those interactive objects worth having in the long run? Or would their removal make it less of a 'Bethesda' game?

Full disclosure: I do not play Bethesda games and have no clue as their respective quality or what people really enjoy about them so I'm just curious.

The games are sandboxes where the player makes their own fun and interactive objects are a big part of that.
 

barit

Member
Frankly Fallout 4 or non ps3 versions of Skyrim were perfectly acceptable bug wise to me. The main issue now is that their ancient engine makes butt ugly games. They should take their mountains of money and do something about that.

And still I'm always amazed how good Skyrim can look on the PS4 Pro. Sure character models are not what we would call next gen (especially compared to something like Horizon or Witcher) but the overall world can look beautiful even with this old engine. In my eyes at least.
 
Maybe they need to take the approach Nintendo had for Breath of the Wild. Large all staff on board play test sessions after any major changes they seem to imply that would be impossible for a large complex game like theirs but Zelda is also incredibly large and complex as well.

Thats a pathetic reason when Xenoblade Chronicle X and Breath of the Wild exist though.

Yeah I don't buy that excuse when there's a shit ton of open world games that aren't nearly as buggy as Bethesda games.

Other games don't let you pick up literally anything in the world and place it anywhere else. That's about the long and the short of it.

Not only does that mean any given room could have twitching food all over the table, or an item that launches and bounces all over the room when you knock something into it, it also means they have to account for the crazy people who decide to pick up every watermelon in the game and place them all in the same small shack.

It's primarily physics. You don't usually encounter mechanical bugs, like "oh I increased my health but actually it went down instead of up."
 
If a game is going to cost me $60 (and have an expensive season pass day 1) like Fallout 4 I expect the game not to bug out or crash every 20 minutes like it did in my time with it on PS4.

Their job being hard doesn't make my time with it any less trash.

I point at FO4 because I had more issues with it then FO3/Skyrim/Oblivion/Morrowind combined. (New Vegas was just as much if not a bigger trash fire).
 

giapel

Member
"‘OK look you need to tell us what’s important, because there’s this much time between now and ship, period."

That's their problem, right there. You don't ship until you fixed all the bugs you know of. Period.
 

nynt9

Member
Are half of those interactive objects worth having in the long run? Or would their removal make it less of a 'Bethesda' game?

Full disclosure: I do not play Bethesda games and have no clue as their respective quality or what people really enjoy about them so I'm just curious.

Let me put it this way: one of the big patches of Elder Scrolls online was "we added more interactivity to world objects" because their lack was one of the biggest criticisms of the game and after that patch the game started feeling a lot more like Elder Scrolls even if it still isn't there yet.

It's a huge part of the feel of the games, even though it's not an integral system.

"‘OK look you need to tell us what’s important, because there’s this much time between now and ship, period."

That's their problem, right there. You don't ship until you fixed all the bugs you know of. Period.

That's just infeasible and not how this kind of software development works. There's always going to be bugs. You just need to draw a line where you think the acceptable bugs are.
 

Striek

Member
No, BoTW has a ton more variables and things that could go wrong with its chemistry system. Skyrim doesn't even come close to the level of physics in BoTW either so I don't see any good reason for geometry so many bugs in the game as well. Fact is the systems programmed into these games are poor, one of the big ones seems to be their spawning systems.

Bethesda simply needs things to be cleaner under the hood of their engine. Best way is to start from the ground up or to strip it down to its bones and reprogram with a more modern architecture in mind.
Variables are more than just physics, actually encompass almost everything, objects, quest states, decision trees, NPC schedules etc. Zelda isn't even in the same league. The physics in Skyrim applying to almost anything anywhere as opposed to way less objects and almost none in areas with NPCs means Skyrim is doing a lot more on that front regardless. The "chemistry system (lol)" is barely a thing. It's just a hype term for several good novel gameplay mechanics that don't actually attempt to emulate chemistry or really interact much.
 

nynt9

Member
Variables are more than just physics, actually encompass almost everything, objects, quest states, decision trees, NPC schedules etc. Zelda isn't even in the same league. The physics in Skyrim applying to almost anything anywhere as opposed to way less objects and almost none in areas with NPCs means Skyrim is doing a lot more on that front regardless. The "chemistry system (lol)" is barely a thing. It's just a hype term for several good novel gameplay mechanics that don't actually attempt to emulate chemistry or really interact much.

This. Most of the interactions in Zelda between different systems are pre-programmed things and some behavior emerges from them. The difference is that in Elder Scrolls literally every single thing is a physics object. If you go inside an inn and use the force breath power, it will sweep up all the food, utensils, shelves, books, people and whatever else is in the scene, and if you save the game and reboot it, those items will stay there. Zelda doesn't persist physics objects when a scene is loaded back in.
 
Huh. It turns out the "lazy devs!" argument actually *does* hold for Bethesda. This is a complete rationalization for shit software engineering practices. They should be ashamed of themselves.
 

RPGam3r

Member
"‘OK look you need to tell us what’s important, because there’s this much time between now and ship, period."

That's their problem, right there. You don't ship until you fixed all the bugs you know of. Period.

Do you work in IT? There is no way to fix all bugs.
 
Not only does that mean any given room could have twitching food all over the table, or an item that launches and bounces all over the room when you knock something into it, it also means they have to account for the crazy people who decide to pick up every watermelon in the game and place them all in the same small shack.

And honestly, these sorts of bugs are more hilarious than they are troublesome. My favorite will always be when the giants club you into space. If I were making a game and saw that I'd say to leave it in.
 
"‘OK look you need to tell us what’s important, because there’s this much time between now and ship, period."

That's their problem, right there. You don't ship until you fixed all the bugs you know of. Period.

Ok all video games are now $200 and take three times longer to make.

Even the most polished games have hundreds if not thousands of bugs, the issue is that Bethesda continues to let common and game breaking bugs get through.

If Zelda BOTW crashes if I swirl the stick 97 times in a row it doesn't really matter, that bug can be let through because almost no user would ever encounter it.
 
Variables are more than just physics, actually encompass almost everything, objects, quest states, decision trees, NPC schedules etc. Zelda isn't even in the same league. The physics in Skyrim applying to almost anything anywhere as opposed to way less objects and almost none in areas with NPCs means Skyrim is doing a lot more on that front regardless. The "chemistry system (lol)" is barely a thing. It's just a hype term for several good novel gameplay mechanics that don't actually attempt to emulate chemistry or really interact much.

NPC's have schedules in BoTW as well and it doesn't matter if one game has less NPC's, a well programmed game uses a universal system for all so the pure quantity of a feature doesn't mean anything if programmed properly. Physics apply to every single object in BoTW and that includes trees, rocks, and literally everything besides solid foundations so yea Zelda takes it quite a few steps further there. Also your whole comment on the chemistry system not being a real thing is bullshit and you know that if you have actually played the game at all.

EDIT: Now I see there is no sense in trying to argue this point between these two games as no matter how you put, Bethesda has a different and frankly bad approach to fixing bugs in comparison to Nintendo. The excuses for the amount of jank their games have don't mean anything if they're not intending on being meticulous about fixing bugs that simply ruin immersion to an otherwise very fun experience.
 

RPGam3r

Member
Huh. It turns out the "lazy devs!" argument actually *does* hold for Bethesda. This is a complete rationalization for shit software engineering practices. They should be ashamed of themselves.

People calling devs lazy should be ashamed of themselves. A lot of people on these teams work their ass off, blurring the lines of good work life balance way more than they should.
 

nynt9

Member
NPC's have schedules in BoTW as well and it doesn't matter if you one less NPC's, they all use the same system so the amount of a feature doesn't mean anything. Physics apply to every single object in BoTW and that includes trees, rocks, and literally everything besides solid foundations so yea Zelda takes it quite a few steps further there. Also your whole comment on the chemistry system not being a real thing is bullshit and you know that if you have actually played the game at all.

The NPC schedules in BOTW and the amount of physics objects do not compare to the scale they are in Skyrim.
 
Top Bottom