Support NeoGAF

[IbizaPocholo]

Xbox’s Matt Booty ‘dreams’ of having AI as QA testers | VGC

“That would be transformational,”, the Xbox Game Studios head says…

www.videogameschronicle.com

Speaking at a Q&A at PAX West 2022 over the weekend, Booty explained that as games get more complex they’re more prone to bugs, and claimed that current QA testing methods struggle to keep up with this.

As such, Booty says he’s asked AI researchers at Microsoft to come up with an AI that is able to replace a human when it comes to testing games.

“Some of the processes we have, have not really kept up with how quickly we can make content,” Booty said. “One of those is testing.

“You think about a game, one of the biggest differences between a game and something like a movie, is if we’re working on a movie and you come in and say ‘hey, this ending, let’s tighten this up, let’s edit this, let’s cut that scene’, it usually doesn’t break anything at the beginning of the movie.

“But in a game you can be ready to ship, and a designer’s like, ‘I’ve got this one little feature, I’m just going to change the colour on this one thing’’ and then it somehow blows up something and now the first 10 minutes of the game doesn’t play.

“So that testing aspect, every single time anything new goes into a big game the whole game has to be tested, front-to-back, side-to-side.

“My dream – there’s a lot going on with AI and machine learning right now, and people using AI to generate all these images.

“What I always say when I bump into the AI folks, is: ‘Help me figure out how to use an AI bot to go test a game.’

“Because I would love to be able to start up 10,000 instances of a game in the cloud, so there’s 10,000 copies of the game running, deploy an AI bot to spend all night testing that game, then in the morning we get a report. Because that would be transformational.

“I always kind of laugh a little bit, people always say ‘the game shipped on Tuesday but I hear they were fixing bugs on Saturday night’ – there’s months of testing and things that have to happen before a game goes out.”

 

[mansoor1980]

AI fixing bugs should be good for bethesda game studios which is owned by microsoft , overall a decent proposal by mr booty

 

[legacy24]

so he wants people to lose their jobs??

 

[Sosokrates]

so he wants people to lose their jobs??

Yes. Every business would like less costs and better efficiency.

 

[Arachnid]

TBH that does sound like the dream. I bet an AI would catch just about everything in a fraction of the time. I dream of buggless games, and this seems to go hand in hand with that.

I, for one, welcome our new robot overlords.

 

[Hendrick's]

so he wants people to lose their jobs??

AI for testing software is already being done and will be a great tool when it matures.

 

[ChorizoPicozo]

I can see this happening faster than people could anticipate.

Machine learning to detect the most common type of bugs and glitches.

 

[adamsapple]

Considering an AI generated painting won an art competition a few days back, this might be good news in the coming years for bug testing.

But it's definitely not good news for QA testers losing their jobs.

 

[Jinzo Prime]

so he wants people to lose their jobs??

Why is it that Matt always seems to say the wrong thing? It's like he has terminal foot-in-mouth syndrome.

 

[SlimySnake]

The software development industry has long devalued QA engineers so Im not surprised someone from Microsoft devalue their contribution, and think AI can do what good QA can. Software developers are expected to architect and engineer bug free code that pass QA on the first try but ive seen the best programmers completely fucking fail to get larger updates out without QA finding something. Guys getting paid hundred of thousands of dollars getting shown up by a guy getting paid $35-50k.

It's a shame we continue to undermine what QA brings to gaming. I know for a fact that good QA can make programmers better. Our minds are simply not wired to think of the user experience. We get laser focused and tunnel visioned into developing according to specs and we need GOOD QA to snap us out of it. It's what separates good software development firms from others. An AI alogrithm created by those same programmers is going to have the same issues.

 

[bender]

Fewer humans is always a good thing.

 

[Hendrick's]

Fewer humans is always a good thing.

 

[Bernd Lauert]

so he wants people to lose their jobs??

Low tier grunt work needs to be eliminated entirely.

 

[Knightime_X]

Considering an AI generated painting won an art competition a few days back, this might be good news in the coming years for bug testing.

But it's definitely not good news for QA testers losing their jobs.

When a game STILL comes out buggy but the credits show a bunch of QA testers something went terribly wrong in that department.

 

[Robb]

That actually sounds like a great use of the tech.

 

[Jinzo Prime]

When a game STILL comes out buggy but the credits show a bunch of QA testers something went terribly wrong in that department.

It's not the testing that's the problem. Its always that the developers don't have time to fix the code before the deadline.

 

[jaysius]

It's not the testing that's the problem. Its always that the developers don't have time to fix the code before the deadline.

Well you've stated the problem clearly, people can't process the games fast enough to keep up with the pace that's demanded.

AI really is the answer, it could probably run the game infinitely more times than any one human. Not that it would find more bugs, but the number of times it could replay the same situtation EXACTLY THE SAME is superior to any human.

 

[ckaneo]

When a game STILL comes out buggy but the credits show a bunch of QA testers something went terribly wrong in that department.

No. Most QA find the bugs. Devs just dont fix them.

 

[reksveks]

Automation is coming for most current jobs and that's okay, what MS needs to do is make sure that there are new roles for those who will lose their current jobs.

The argument that automation should be stopped cause we are going to lose jobs is unfortunately a no-go in a capitalist system.

 

[Javthusiast]

AI for testing software is already being done and will be a great tool when it matures.

 

[Knightime_X]

It's not the testing that's the problem. Its always that the developers don't have time to fix the code before the deadline.

Eventually, AI can easily assist developers in fixing the very problem AI detects.
Or at the very lease show pathways to multiple solutions and the devs just pick the one they want.

 

[Chukhopops]

It’s kinda obvious since this kind of automated testing already exists for simpler processes and to automate regression tests for example. If you measure by volume the vast majority of test cases on my product (APIs for an insurance company) are already automated.

But you still need humans to run the rest of the tests (especially functional) and it will be a few years before AI can understand acceptance criterias (most of which are not written but implicit from the perspective of the tester).

 

[Negotiator101]

Another thread on Matt Booty? Let's create one thread mods and combine all Matt Booty threads and call it "Booty call".

 

[nocsi]

Just hire speedrunners as QA consultants. All those guys do is look for glitches. AI being able to find non-functional problems is a incomputable problem.

 

[Banjo64]

Not hear anything from this guys for ages and now he’s coming out with all sorts of horse shit

 

[Wandering Leopard]

It's inevitable that AI will take a lot of jobs, it's just a question of when.

 

[Shut0wen]

so he wants people to lose their jobs??

Tbf QA is probably one of the worst jobs you can get, never met anyone who said its enjoyable and has stayed in that profession for more then afew years

 

[OGM_Madness]

The software development industry has long devalued QA engineers so Im not surprised someone from Microsoft devalue their contribution, and think AI can do what good QA can. Software developers are expected to architect and engineer bug free code that pass QA on the first try but ive seen the best programmers completely fucking fail to get larger updates out without QA finding something. Guys getting paid hundred of thousands of dollars getting shown up by a guy getting paid $35-50k.

It's a shame we continue to undermine what QA brings to gaming. I know for a fact that good QA can make programmers better. Our minds are simply not wired to think of the user experience. We get laser focused and tunnel visioned into developing according to specs and we need GOOD QA to snap us out of it. It's what separates good software development firms from others. An AI alogrithm created by those same programmers is going to have the same issues.

Why not both?
Have AI support real humans test the games. Imaging having AI simulate a MMO, but more than just bots, they are able to play the game and figure out when something is broken so they flag it for a real human to review and debug.

AI could figure out speedrun glitches or game crashes and provide more data to troubleshoot.

 

[Kenneth Haight]

Whenever you feel bad about yourself, remember Matt Booty is a high level executive at Microsoft and think you can achieve anything you want. Truly remarkable.

 

[Mowcno]

But then you need humans to QA test the AI QA Tester.

 

[Pelta88]

The best games I've played are designed to immerse and induce an emotive response.

Things which AI can't pick up on.

 

[reksveks]

The best games I've played are designed to immerse and induce an emotive response.

I am a supporter of QA testers but their primary role isn't related to design. They might do it as an extension of their roles and responsibilities but they ultimately aren't there for that.

 

[PendekarRajawali]

It's a booty dream.
By the way, can AI feel whether a game is fun or not?

 

[Topher]

"Booty says he’s asked AI researchers at Microsoft to come up with an AI that is able to replace a human when it comes to testing games."

And yet in all those quotes no where does he say "replace a human". As a software developer, I would love to have an AI that can run through every single possible path of my code and find the issues before a user does. But that doesn't mean I don't want QA folks to still do their thing. QA isn't just about finding bugs. This is how I read what Booty is saying. Why the writer of the article interpreted the way he did, no idea. I would not be opposed to AI replacing some journalists, that much I’m sure about.

 

[AJUMP23]

We use AI scripting and testing all the time, just not in video games. Automated testing is a life and time saver. You will always need people in the loop for certain testing though.

 

[Loxus]

The software development industry has long devalued QA engineers so Im not surprised someone from Microsoft devalue their contribution, and think AI can do what good QA can. Software developers are expected to architect and engineer bug free code that pass QA on the first try but ive seen the best programmers completely fucking fail to get larger updates out without QA finding something. Guys getting paid hundred of thousands of dollars getting shown up by a guy getting paid $35-50k.

It's a shame we continue to undermine what QA brings to gaming. I know for a fact that good QA can make programmers better. Our minds are simply not wired to think of the user experience. We get laser focused and tunnel visioned into developing according to specs and we need GOOD QA to snap us out of it. It's what separates good software development firms from others. An AI alogrithm created by those same programmers is going to have the same issues.

These companies just need better quality control. Besides, isn't it the job developers already to make sure the game is bugs free?

I rather him divert those resources to in-game AI instead.

 

[CeeJay]

Most QA is outsourced these days anyway and a lot of those QA companies in the credits of games have a legion of casual testers on their books who do it as a side hustle to their fulltime jobs and won't appear in the credits of the game. To be honest I think Matt is right to push QA testing in this direction, it will mean that updates to games will be quicker to release and also release with fewer significant bugs generally.

Regression testing that Matt is on about here is just one aspect of QA testing and is the one that is the most mind-numbingly boring, most labour intensive, yet also most valuable for a polished product. It makes a lot of sense to automate it as much as possible. Due to the cost and time of regression testing it means that a developer will want to do it as little as possible to streamline the development process as much as possible. It will typically be done very late in a dev cycle for maximum value and often leaves little time to resolve the bugs that they do find. If regression testing can be run more often and earlier in development it will make the finished product much higher quality.

 

[jm89]

He should be dreaming of AI for management.

 

[marjo]

This is not controversial in any way. Everyone I know in QA believes that automation is where things should be.

-"People should think and machines should test"

 

[dezzy8]

Yes. Every business would like less costs and better efficiency.

I mean it would create jobs for the people managing and updating the AI.

 

[crackajack]

It's a booty dream.
By the way, can AI feel whether a game is fun or not?

In the future imho inevitable way better than any games journalists, who is just able to argument why he/she find it themself subjectively good or bad with some pretented objectivism. If it's fun to you, your personal AI test-bot (or the group you fit to) will be more accurate in their hit rate.
Current youtube algorithms know your tastes to some degree, future machine learned AI testing teams will create gamepass recommentations for specific subscribers according to what they previously liked.

Think drivatars, which are for racing just a ghost with minor personality, if at all, but that to the max.

If they aren't doing it dumb, never recommending anything outside of one's bubble, this will be great and people will not waste their time and money on games they don't like.

If they scan your fb-page, browser cookies, netflix history they might get it even more accurate, when some hints allow them to know that content xyz will suit some memory you shared and other media you consumed.

Imho not an if, just when and not everyone will want this level of being open about themselves. Taking their experience if and why they like something, or not, kind of away.

 

[CeeJay]

These companies just need better quality control. Besides, isn't it the job developers already to make sure the game is bugs free?

I rather him divert those resources to in-game AI instead.

If only it was that simple. Game development can involve hundreds of people coding specific areas of a game and as one of those developers you are focussed on delivering that small part of the much bigger whole. As Matt points out, a developer could make some changes to their own work that has an unintended effect on someone else's work who has already completed their part and had it signed off. This is an all too common situation and is increased as teams get larger.

Say that a dev is working on the in-game physics of a character and is trying to solve a problem which in certain situations results in the character being launched high into the air. They fix that bug by tweaking the inertia to prevent the glitch from occurring and when they test that change it all appears fine and get it signed it off. But, that tweak has other unintended outcomes such as a jump taking longer to perform which messes with another characters animation or prevents a character from being able to jump a specific gap somewhere in a level that they could previously make. You can't expect the guy who made the change to play through the entire game multiple times with all characters whenever they make a code change, games would never release!

Sure, this situation is likely to be found and rectified quickly as it would be an obvious bug but you can see how a small change by one developer can have affect other peoples work and not get picked up by the developers.

 

[MiguelItUp]

As someone that is extremely experienced in QA and the crunch that comes with it, I could certainly see something like this being a dream come true. QA is a great first step into the industry, but man is it not for everyone, lol. When studios combine their own QA with another external QA studio things tend to get messy. With duped bugs, inconsistent logging structure, etc. With AI it'd always be consistent, and would most likely catch things that some people could potentially miss. I doubt dupes would ever be a thing either. AI could also work as much as you need it to and it'd never be an issue. So, yeah, huge pluses across the board. I don't blame him for dreaming of such a thing at all.

 

[DaGwaphics]

Will be a bit of a paradox when the AI gets a bug in it.

Seriously, this would probably be something that could work in tandem with live humans. The AI could do a good job of finding bugs that block progression etc.

There would be some bugs that are QoL things that I can see a computer missing.

 

[lovetalkgames]

Some of the more annoying people in the gaming industry got in via QA testing.

 

[Clear]

Some of the more annoying people in the gaming industry got in via QA testing.

So fucking true. LOL.

Huge problem in the UK industry for years has been the pipeline from QA into design and production. Particularly as the latter has always been horribly cliquey and full of untalented sycophants and cronys.

 

[M1chl]

For most people in the industry QA work is how they got there in first place, so I do find this disgusting

 

[Kilau]

Considering an AI generated painting won an art competition a few days back, this might be good news in the coming years for bug testing.

You have inspired me.

Xbox Matt Booty ‘dreams’ of having AI as QA testers

 

[AmuroChan]

I'm surprised a company like Microsoft isn't already using AI for QA. My company's been using it for a couple years now at this point.

 

[hemo memo]

AI fixing bugs should be good for bethesda game studios which is owned by microsoft , overall a decent proposal by mr booty

When it comes to Bethesda glitches, I think the AI will just give up and stop working.