Let’s talk about the Xbox Live Cloud
Posted by Abbie Heppe on Jun 24, 2013
Hi everyone!
I’m Jon Shiring, and I’m an engineer working with the Cloud technology that you’ve heard about for Respawn’s game Titanfall. I have seen a lot of confusion online and I think it’s worth explaining more about what we’re doing on Titanfall and also more generally about Cloud computing and dedicated servers.
First, let’s take a step back and dive into the common multiplayer design and talk about why Dedicated Servers are better.
Player-Hosted Servers
The vast majority of games will pick a player and have them act as the server for the match. This means that all of the other players talk to them to decide what happens in a game. When you shoot your gun, the server decides if that is allowed and then tells everyone what you hit. Let’s agree to call this system “player-hosted” for simplicity.
What kinds of problems do you get with player-hosted servers?
- What if one player has great bandwidth, but it’s laggy? Games are having to choose between different player hosts, and have to make hard decisions about which one should be the host, with two different measurements – bandwidth and latency. Sometimes it will pick a host who has good bandwidth, but whose latency isn’t ideal. But we don’t want the game to make compromises on lag and we really want the game to feel the same every time we play. We really don’t want to worry about this stuff – we just want to play and have the game feel good.
- What about host advantage? The player-host has the game running locally on their machine, so they get super low latency access to the game world. You’ve probably seen this in action as some player seems to see you long before you get to see them or their bullets hit you before yours hit them. That sucks. Nobody should have an artificial advantage in a competitive multiplayer game.
- What if the player-host is a cheater? Since the host gets to make decisions about kills, XP, and unlocks and such, it’s really bad if they abuse their power to wipe out your stats, or they cheat by flying around maps and insta-killing people. It’s infuriating, in fact.
- What if the host disconnects? In the “best case” for this, you can do host migration if there’s another player who has enough bandwidth and everyone else can talk to them. If you hit that jackpot, you can migrate from the old host to the new one, which pauses the game and then unpauses when the new player-host is ready to start acting as the server. This isn’t a fun process, and it can fail.
- What if the host’s bandwidth disappears? The game tested the host’s bandwidth at some point and decided that they had enough to host. But someone at their house is now torrenting files and their roommate is streaming Netflix. That “great” bandwidth the game detected earlier is now awful bandwidth, and the other players are lagging halfway through a match.
- What if some players can’t talk to the host? You know all that “Open NAT” stuff? Your home internet router is generally trying really hard to keep bad people out, and games are sort of a weird case where the game is trying to get your router to cooperate and let other players create connections INTO your network. Games need to check if every player can talk to the host and if one can’t then that host won’t work. It makes matchmakin slower, and we hate that. Also, by telling you to open up your router, the game is asking you to reduce the security of your home network in order to make the game work. It would be great if you didn’t have to compromise your security in order to play games.
- What if nobody has enough bandwidth? You got a great group of players together, but nobody has enough bandwidth to actually host a game. You can work around this by compromising your matchmaking a little to make sure that each lobby has a player in it that can be a host. But we don’t actually want compromised matchmaking, so this isn’t a good fix.
- What about players who are paying for their bandwidth or have bandwidth caps? If you have a bandwidth cap on your home internet connection, or even worse, you’re paying for your bandwidth, what happens when you play a game and later find out that the game thought you were an awesome host? Your home internet connection is now slow or you have a huge bill waiting.
- So if I’m hosting, my machine is doing all this extra work on behalf of everyone else? Yes! You are doing more work on your CPU than all of the other players are. This means the game isn’t as cool looking as it could be and everyone else has extra CPU just sitting there. Or worse, their game actually looks better than yours! We think the game should be consistent on every machine in a match. Don’t punish the host with a worse game or leave all of that extra CPU sitting empty on the other players machines.
Okay, so player-hosted servers have a lot of downsides. So why do so many games use them? They have one really big upside – it doesn’t cost money to run the servers! Running hundreds of thousands of servers can be extremely expensive. EXTREMELY expensive. Like “oh my god we can’t afford that” expensive. So your player experience gets compromised to save (large amounts of) money.
Dedicated Servers
Dedicated servers are when a computer sitting out on the internet handles all of the host duties, leaving every client free to just be a client.
- You can get even more CPU on your dedicated servers to do new things like dozens of AI and giant autopilot titans!
- Suddenly you have no more host advantage!
- Bandwidth for the servers is guaranteed from the hosting provider!
- You can use all of the available CPU and memory on the player machines for awesome visuals and audio!
- Hacked-host cheating isn’t an issue!
- Matchmaking can be lightning fast since it’s guaranteed that everyone can connect to your servers.
- And since the servers aren’t going to go disconnect to watch Netflix, you don’t need to migrate hosts anymore!
The player experience is so much better. This sounds awesome!
But it costs a LOT of money.
This is something I have worked on for years now, since coming to Respawn. A developer like Respawn doesn’t have the kind of weight to get a huge price cut from places like Amazon or Rackspace. And we don’t have the manpower to manage literally hundreds-of-thousands of servers ourselves. We want to focus on making awesome games, not on becoming giant worldwide server hosting providers. The more time I can spend on making our actual game better, the more our players benefit.
I personally talked to both Microsoft and Sony and explained that we need to find a way to have potentially hundreds-of-thousands of dedicated servers at a price point that you can’t get right now. Microsoft realized that player-hosted servers are actually holding back online gaming and that this is something that they could help solve, and ran full-speed with this idea.
The Xbox group came back to us with a way for us to run all of these Titanfall dedicated servers and that lets us push games with more server CPU and higher bandwidth, which lets us have a bigger world, more physics, lots of AI, and potentially a lot more than that!
What is the Cloud?
Amazon has a cloud that powers websites. Sony has a cloud that streams game video so you can play a game that you don’t have on your machine. Now Xbox Live has a cloud that somehow powers games. Cloud doesn’t seem to actually mean anything anymore, or it has so many meanings that it’s useless as a marketing word.
Let me explain this simply: when companies talk about their cloud, all they are saying is that they have a huge amount of servers ready to run whatever you need them to run. That’s all.
So what is this Xbox Live Cloud stuff then?
Microsoft has a cloud service called Azure (it’s a real thing – you can go on their website right now and pay for servers and use them to run whatever you want). Microsoft realized that they could use that technology to solve our problem.
So they built this powerful system to let us create all sorts of tasks that they will run for us, and it can scale up and down automatically as players come and go. We can upload new programs for them to run and they handle the deployment for us. And they’ll host our game servers for other platforms, too! Titanfall uses the Xbox Live Cloud to run dedicated servers for PC, Xbox One, and Xbox 360.
But it’s not just for dedicated servers – Microsoft thought about our problem in a bigger way. Developers aren’t going to just want dedicated servers – they’ll have all kinds of features that need a server to do some kind of work to make games better. Look at Forza 5, which studies your driving style in order to create custom AI that behaves like you do. That’s totally different from what Titanfall uses it for, and it’s really cool! So it’s not accurate to say that the Xbox Live Cloud is simply a system for running dedicated servers – it can do a lot more than that.
How is this different from other dedicated servers?
With the Xbox Live Cloud, we don’t have to worry about estimating how many servers we’ll need on launch day. We don’t have to find ISPs all over the globe and rent servers from each one. We don’t have to maintain the servers or copy new builds to every server. That lets us focus on things that make our game more fun. And best yet, Microsoft has datacenters all over the world, so everyone playing our game should have a consistent, low latency connection to their local datacenter.
Most importantly to us, Microsoft priced it so that it’s far more affordable than other hosting options – their goal here is to get more awesome games, not to nickel-and-dime developers. So because of this, dedicated servers are much more of a realistic option for developers who don’t want to make compromises on their player experience, and it opens up a lot more things that we can do in an online game.
Wrapping up…
This is a really big deal, and it can make online games better. This is something that we are really excited about. The Xbox Live Cloud lets us to do things in Titanfall that no player-hosted multiplayer game can do. That has allowed us to push the boundaries in online multiplayer and that’s awesome. We want to try new ideas and let the player do things they’ve never been able to do before! Over time, I expect that we’ll be using these servers to do a lot more than just dedicated servers. This is something that’s going to let us drive all sorts of new ideas in online games for years to come.
I know this got pretty technical and long-winded, so I thank you for reading this far. Hopefully I’ve cleared some things up, and you can see why I’m so excited about what Microsoft has done here and how it is letting us do awesome new things for our game. I’ll see you online in the spring to play some Titanfall on our dedicated servers!
