• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Xbox Matt Booty ‘dreams’ of having AI as QA testers

CeeJay

Member
I'm surprised a company like Microsoft isn't already using AI for QA. My company's been using it for a couple years now at this point.
For games?

Using AI for unit testing in an application has been around a fair while and is way simpler than getting an AI to first learn to play an game and then test it. I think for games it's still very much in it's infancy.
 

Sosokrates

Report me if I continue to console war
If no one has jobs no one buys your product (y)
Lol, well Microsoft doing this would not take all the jobs lol.

Anyway this is an inevitable societal and government issue.

Robots + A.i are already taking jobs, and it's only going to get worse.
 

Trogdor1123

Member
I suspect this would be a careful what you wish for sort of thing. If you got 100 million change requests from an ai it could be a serious issue to deal with.
 

RiccochetJ

Gold Member
I'm in favor of it. Especially if it's unbelievably mind numbing for a human to do a thousand times a day. I'm absolutely convinced an AI will find a hole in the map that you'll fall through quicker than a human. Leave the intricate and complex tasks for a human to work through and test where it's actually stimulating work.
 

AmuroChan

Member
For games?

Using AI for unit testing in an application has been around a fair while and is way simpler than getting an AI to first learn to play an game and then test it. I think for games it's still very much in it's infancy.

I wasn't suggesting that AI can fully replace the entire QA process for games, but surely for basic script testing they can utilize AI at this point. I was just surprised that MS doesn't use any AI processes in their QA at all and that they are still fully reliant on human testers.
 
Just hire speedrunners as QA consultants. All those guys do is look for glitches. AI being able to find non-functional problems is a incomputable problem.
Funny you should mention speedrunners, as them using bots/AI in tool assisted speedruns to find new speedrunning glitches in 30 year old games is basically the exact same concept as using bots/AI to find glitches in new games as part of Q&A.
 

NickFire

Member
Bloggers everywhere are shuddering at the thought of two more QA articles (QA tests their replacements; then QA crunches to test their replacements) and then having no more QA blogs left to write.
 

xrnzaaas

Gold Member
AI would be helpful in catching technical and recurring bugs but it shouldn't be the dominant one over human testers or, even worse, replace them completely.
 
AI would be helpful in catching technical and recurring bugs but it shouldn't be the dominant one over human testers or, even worse, replace them completely.
Yeah, it’s never going to be a replacement.

A real life example of it being used already, was a studio that when the employees left for the night they would network the computers together to run a bot through the game map in a grid pattern and make red flags when it would get stuck or fall through the map. Since the bot can run internally with just the 3D ground model alone and doesn’t need to worry about rendering frames or running at regular game speed with 200+ gaming PCs backing it up, simply put: in a single night it tested what no Q&A team could pull off if it spent 10 years trying.

The problems are still being solved by a human, but bots/AI can simply just cover a lot more ground in some scenarios than any human ever could. And with games getting bigger and bigger and more complex, tools like these are going to be becoming more and more crucial to game development.

The concern around it taking over human jobs, is a bit like being concerned that editors everywhere are going to lose their jobs because spellcheck exists.
 
Last edited:
It's a good idea. As games continue to get larger and more complex, it's increasingly difficult for a QA team to test every corner of the game constantly with each update. I mean just look at how many exploits, glitches and bugs are found in any current game of any size. An AI toolset to streamline testing would be incredible.
 
Sounds like a good idea. Hire a bunch of people to try and break your game or run endless amount of instances and have AI do it for you, with less overhead, faster and more accurate results. Sounds like a good idea.
 

SSfox

Member
Oh boy what a dream...
44nSy7g.png


Why also AI make games literraly we're at it
44nSy7g.png
 

ReBurn

Gold Member
so he wants people to lose their jobs??
AI Robots don't participate in Kotaku hit pieces when Nintendo doesn't hire them after 10 years of sitting around doing nothing but playing video games all day and learning no other marketable skills. At least not yet.
 

Speaking at a Q&A at PAX West 2022 over the weekend, Booty explained that as games get more complex they’re more prone to bugs, and claimed that current QA testing methods struggle to keep up with this.

As such, Booty says he’s asked AI researchers at Microsoft to come up with an AI that is able to replace a human when it comes to testing games.

“Some of the processes we have, have not really kept up with how quickly we can make content,” Booty said. “One of those is testing.

“You think about a game, one of the biggest differences between a game and something like a movie, is if we’re working on a movie and you come in and say ‘hey, this ending, let’s tighten this up, let’s edit this, let’s cut that scene’, it usually doesn’t break anything at the beginning of the movie.

“But in a game you can be ready to ship, and a designer’s like, ‘I’ve got this one little feature, I’m just going to change the colour on this one thing’’ and then it somehow blows up something and now the first 10 minutes of the game doesn’t play.

“So that testing aspect, every single time anything new goes into a big game the whole game has to be tested, front-to-back, side-to-side.

“My dream – there’s a lot going on with AI and machine learning right now, and people using AI to generate all these images.

“What I always say when I bump into the AI folks, is: ‘Help me figure out how to use an AI bot to go test a game.’

“Because I would love to be able to start up 10,000 instances of a game in the cloud, so there’s 10,000 copies of the game running, deploy an AI bot to spend all night testing that game, then in the morning we get a report. Because that would be transformational.

“I always kind of laugh a little bit, people always say ‘the game shipped on Tuesday but I hear they were fixing bugs on Saturday night’ – there’s months of testing and things that have to happen before a game goes out.”
Well who in the fuck is going to test the AI, Booty? Chicken or the egg conundrum if I’ve ever heard it. You know, this guys takes on shit are really booty, man.
 

adamsapple

Or is it just one of Phil's balls in my throat?
there's been an update to the article


Editors note: After the keynote, I caught up with Matt, who clarified his comments about use of AI in QA testing: "Digital tools have always been core to the evolution of video games. Our goal is to create more time for our teams to be creative and iterate quickly. AI testing is not about replacing Test and Quality teams, but allowing them to focus their skills on the complex, design-related nuances while automation handles testing tasks at scale. Our Test and Quality teams are a valuable part of the development process and much of the AI experimentation is being done by our test teams themselves, to free up others from having to spend time on repetitive tasks like scouring large open-world games for texture seams or hunting for gaps in collision boundaries."
 

oldergamer

Member
The larger the game, the more testers you need. Its called automation. They do need a way to better test games as its generally impossible to find all the issues in most rpg game titles
 

Kenneth Haight

Gold Member
Of course AI will be used in testing.

I think peoples concern about this is similar to that of a network penetration tester. You can run automated security scanners and tools against a target network, anyone can. But a proper professional will be able to break in to a network in a silent way, document it etc etc.

I imagine proper QA testing requires a “human” element at times. But if you can speed up the process by automating a lot of it, then of course you’re doing that. This is just marketing talk. AI and ML have been in use for years now so this is not necessarily new.
 

CeeJay

Member
If no one has jobs no one buys your product (y)
no one is saying that this is taking jobs away. Think about it... If you can automate tasks that would otherwise bog your team down then it frees them up to do other things and ultimately shorten development time and bring down costs. A glass washing machine in a bar doesn't mean the bartender is out of a job it just means they can spend more time serving people drinks.
 

CeeJay

Member
Yeah, it’s never going to be a replacement.

A real life example of it being used already, was a studio that when the employees left for the night they would network the computers together to run a bot through the game map in a grid pattern and make red flags when it would get stuck or fall through the map. Since the bot can run internally with just the 3D ground model alone and doesn’t need to worry about rendering frames or running at regular game speed with 200+ gaming PCs backing it up, simply put: in a single night it tested what no Q&A team could pull off if it spent 10 years trying.

The problems are still being solved by a human, but bots/AI can simply just cover a lot more ground in some scenarios than any human ever could. And with games getting bigger and bigger and more complex, tools like these are going to be becoming more and more crucial to game development.

The concern around it taking over human jobs, is a bit like being concerned that editors everywhere are going to lose their jobs because spellcheck exists.
Hello Games used a similar concept where they set a load of bots off exploring and reporting back various parameters from what they found. It's how with such a small team they could create a game on such a large scale and have some assurance that it would work.
 

Topher

Gold Member
no one is saying that this is taking jobs away. Think about it... If you can automate tasks that would otherwise bog your team down then it frees them up to do other things and ultimately shorten development time and bring down costs. A glass washing machine in a bar doesn't mean the bartender is out of a job it just means they can spend more time serving people drinks.

Another way to think about this is the fact that so many bugs in games are reported after it is released. That is because suddenly you have millions playing the game and they are performing numerous actions and playing a number of ways that a QA team cannot possibly emulate. That's where AI in the cloud comes into it. They actually can emulate those millions of gamers. If AI can report reproducible bugs then the QA team has a heck of a lot more information to work with than they would trying to do it all on their own.
 

Dr.Morris79

Member
no one is saying that this is taking jobs away. Think about it... If you can automate tasks that would otherwise bog your team down then it frees them up to do other things and ultimately shorten development time and bring down costs. A glass washing machine in a bar doesn't mean the bartender is out of a job it just means they can spend more time serving people drinks.
Which is true, but you know a few will be given the heave ho, so to say..

But thinking about it, after seeing games ship like Saints Row, Cyberpunk, Skyrim, No mans Sky, Fallout 3+4, most damn games nowdays.. are Q&A testers really doing any job to begin with..
 

CeeJay

Member
Another way to think about this is the fact that so many bugs in games are reported after it is released. That is because suddenly you have millions playing the game and they are performing numerous actions and playing a number of ways that a QA team cannot possibly emulate. That's where AI in the cloud comes into it. They actually can emulate those millions of gamers. If AI can report reproducible bugs then the QA team has a heck of a lot more information to work with than they would trying to do it all on their own.
Agree 100%

QA companies have tried to do this by hiring legions of semi-casual staff, i have done a bit myself when the times of the tests were favourable for my work and home life. Getting paid a bit to play a game upto a year or two before release sounds amazing but it has it's downsides. The problem was that a load of casuals might not follow the instructions properly and no matter how many people you get it's never enough. Also you can guarantee at least one of them will leak a screenshot. The number of times i've seen a leaked screenshot on GAF and known instantly it was from a test session the previous day! They would then add watermarks to try and stop it. It got so bad that some "bright" spark (you know who you are!) had the idea to replace all the in game textures with a checkerboard, it was like playing a magic eye epilepsy generator, fucking horrendous! I soon lost the appetite to attend much after that.

AI testing can be done in bigger numbers, more quickly, more accurately and its not going to leak a screenshot of your secret game. Where's the downside?
 

ZoukGalaxy

Member
Matt is pretty stupid, that's humans playing in the end, no AI. It's yet incapable to do all the stupid thing human can do.

That's well know, quote from Einstein "Two things are infinite: the universe and human stupidity; and I'm not sure about the universe"

What does he wants ? Game for AI ?
Hot Shots Idiot GIF
 
Last edited:

Miles708

Member
Of course AI will take over all the most mundane and stupid testing. How's that even a question.
There are already systems to automate walking over all collideable walls or constantly pressing buttons repeatedly. You will need testers for actual testing instead of grunt work.

And that's a good thing.
 
I would figure AI would never completely replace QA testing, but would act as a first pass that could do a much heavier workload, save the instances of what it thinks is a problem, then kick it up to the QA testers to make the decision to flag for the actual devs to fix.

Even having an army of people manually doing what it would take to catch every bug in modern games sounds like the definition of a living hell. I would rather work my insane hours driving a semi than deal with that kind of tedium.
 

01011001

Banned
Matt is pretty stupid, that's humans playing in the end, no AI. It's yet incapable to do all the stupid thing human can do.

That's well know, quote from Einstein "Two things are infinite: the universe and human stupidity; and I'm not sure about the universe"

What does he wants ? Game for AI ?
Hot Shots Idiot GIF

it's really hard taking your take seriously with that grammar 🤣
I automatically read this in a Borat-esque voice

Sacha Baron Cohen Thumbs Up GIF by Amazon Prime Video
 

consoul

Member
For a forum with an interest in technology, the number of posters in denial or ignorance of AI capability is a bit surprising.

Every industry is about to be transformed by AI and ML. If you haven't already recognised that, you're not paying attention.
 
Top Bottom