- [car engine roaring] [upbeat music]

- A car is very hard to control. Like it is a very complex system and in a simulation engine, it has loop upon loop upon loop. A tire itself is a spring. Then there's the springs of the suspension. The car has moments of inertia and weight and the way the car moves and all of these things interact. The car gets heat and wind resistance. It's so complex. And the AI has to be able to predict that I'm Dan Greenwalt, Creative Director on the Forza of franchise. And this is how we've evolved to take better advantage of deep neural networks to make AI. [car engine roaring] [soft music] I had a pretty atypical background coming into games. I was a martial arts instructor, but it turns out that martial arts does not necessarily pay the bills that great. So I started looking for jobs and a friend of mine. My roommate actually said that there were hiring at Microsoft to test games. I loved games. I like working on games. So I started testing and I really, really enjoyed the work. I enjoyed the teams. I liked how it was so collaborative after working on a bunch of sports titles NBA inside drive, and then some racing titles like Midtown Madness and Motorcross Madness and Monster Truck Madness, and a whole bunch of madness titles. Eventually the original Xbox was announced internally. And when some jobs opened up to go full time, I started working full time and immediately started working bizarre creations on Project Gotham Racing. I ended up moving to the UK, lived there for a while and really enjoyed the work. They ended up giving me credit as a designer, which was not something I asked for, but I was incredibly honored and humbled that they did that. They were a great team. I learned so much working with Martin Schenley and the guys from bizarre. [car roaring] So we founded turn 10, about 2001. The original Xbox had been out for a little while. The group believed that we needed something to compete with Gran Turismo on Sony's box. You know, Gran Turismo was a juggernaut and that really cleared the business case and put the onus onto this very small team. There were four of us to come up with a creative pitch. We got together and put this pitch together for Forza. And it was, it was a new take. We were looking at the racing space differently. We're embracing new trends, web 2.0 customization UGC. That was something we thought was really gonna be important. And it worked so well with car culture. You know, I lived in car culture. I love cars. I tuned cars. I raced cars. I loved that stuff. And so did other people on the team. So that idea of customization and building your car instead of just buying was something we just, we loved Forza obviously it's a racing game. It's in the car genre. I think different players are looking for different things in Forza. When you think about motor sport, they're looking for simulation racing, at 10 tents competition. It's being under threat. It's driving a car under duress it's mastery, horizon is about exploration. It's about hanging out with your friends. It's about finding new places, freedom, fun, jumping the car around, having a great time. The first forts of motor sport, the original, the shift in 2005, it started on the original Xbox. [car engine roaring] So we made a prototype and that was a track that it was stitching together a whole bunch of different ideas. And we built our physics engine and quickly realized that our physics engine needed some assists on top. And at that point we were faced with a fork in the road. We could either layer assists on top that you could turn off or we could change the foundational layer of the physics engine itself. And the engine would become more "Arkady." We immediately went to this idea that the physics engine was sacred and it has to be cutting edge at all times. And the layers we add on top are things you can enable or disable that make it more controllable the controller that first prototype the physics engine was so hard to control the controller cause you were giving a 100% input left a 100% input right. You could get on the throttle off the throttle immediately, and anyone who's played racing games, you watch their hands. That's how most players play. It's incredibly digital, even though they have analog sticks. So we need to put assists on and buffers that would change your input. They would kind of interpolate and guess, what were you really trying to do? That doesn't happen in a real car, we didn't add that to the physics layer. We put that on top. So sometimes the inspiration is new tech. Like the original Xbox had a hard drive and it had a NIC card and that allowed us to create new experiences. So we were inspired by things that would use that, that actually birth Drivatar because drive Drivatar to are needed to create big data caches. And you really couldn't have done that without a hard drive. Drivatar is a AI technology that we worked on in the original Forza Motorsport with Microsoft research in Cambridge and at start, it was a machine learning system on a hard drive using a Bayesian Neural Network to record lines and characteristics of how somebody drove a car. And that's how it started since then Drivatar has evolved considerably as new technologies have come in, new ways of doing AI come in and we've learned so much from how our players play and all the data we get from how the community in general has interacted with the system. So the Drivatar that we work on today bears very little resemblance to the original one. So the original Forza Motorsport, there was this team in Microsoft research from Cambridge, and they had a thought that why is it that Juan Pablo Montoya who is an F1 driver at the time and Michael Schumacher who is another F1 driver at the time, could both be so incredibly fast, but they drive so differently. In fact, you could tell which driver was driving, which car based on how the car yard and how it drove in the rain and how strictly they went into corners. So they started looking at Bayesian Neural Network, learning networks to learn a line based on how somebody drove. So they would have people like me drive a car in our physics engine and that would give them a line. And then they would classify that line and then generalize it to a new application. So they might take a given corner, that I was in and say, okay, we're gonna classify that as a right angle corner. And you took it at 10 tenths or eight tenths of the car's capability. You came in with this much yarn. Now we're gonna go to another track that you've never driven in another car you've never driven and we're going to replay that. But of course you can't replay it because it's got new variables. So it has to then use the neural network to develop a new line. That was a really interesting challenge, but in some ways it was a mismatch with Forza, which is about building cars, falling in love with cars, which means we have hundreds of cars and they all have upgrades. So that whole like how different is one driver versus another in the same car, that's a level of subtlety that requires the cars to be same. We meet two people in cars that are radically different, a car may have 400 more horsepower or a thousand more pounds of weight that really changes how a car drives. So from the very beginning, we had an interesting piece of technology, but it was at slight mismatch with the goals of the game. Ever since then, we have encountered new mismatches that are for us to either design around as designers or implement around with technology. [soft music] It's hard to know even where to start when it comes to AI challenges. I think you've got to start with what players expect out of an opponent in a racing game. What they love is seeing their friends names and kind of a direz version of what they think their players are almost like a cartoon or a characterization. And we found that when we gave really subtle representations of how your friends drive people didn't even recognize it. They had to get kind of bombastic. They had to go beyond how they really drive. And that creates unpredictable AI, which is one of the problems players don't want. Meanwhile, you're meeting players where their skill level is. If you're a very skilled player, you can lap a car incredibly quickly. So how do we keep the opponents grouped around you or predictably around you? And what if you have a catastrophic mistake, you don't wanna necessarily restart the lap, but if you go off track, it can be quite a while to get out of the sand and back on the track. Well now the opponents are long gone, so we've got a set of problems here, but underlying those problems are a much deeper set of problems. That actually is where more of our technology is. So those are more of the design challenges that our technology then looks to match. Part of those technological problems are a car is very hard to control. Like it is a very complex system and we're amazing. I mean humans, we will throw a car into a corner and just intuit that it's gonna be able to take this corner at 60 miles an hour because I've done a couple of laps on it. It probably can go at 80, but we are hitting the limits of 80% of what the car's capable of really quickly. If you're experienced. That's amazing. That's a big challenge for AI. And then on top of that dealing with us humans on the track. So the opponents, AI opponents have to deal with this incredibly unpredictable nature of a human that sometimes we're looking for that simulation. We wanna take it seriously or you come in. And you're just trying to smash things up. Cause some players wanna do that. The opponent, the AI opponents have to deal with all of that. So the set of problems we have to deal with in AI is multifactor. It's designed challenges of how do we delight the player

- [Instructor] Action [car engine roaring]

- As technological challenges of how do we get this computer to deal with the complexity of this whole situation? So in the Xbox one the original [chuckles] I still get that confused on the original Xbox and the Xbox 360, our Drivatar system was local to the hard drive and on Forza Motorsport five, we switched that to being on servers. So these were Thunderhead servers for Xbox live. This was a pretty cool change and brought in a whole new set of problems. We had Drivatar fairly well tamed on the local box because it wasn't really being infected by other people's data. It was really about local Drivatar We trained and your own Drivatar, which dramatically reduced the amount of variables coming into the user's experience. When we went to Thunderhead, all of a sudden it was a massive amount of data, which is awesome. It's great for training in neural net like data is your friend when you're training, learning networks. However, I feel like we've had a tiger by the tail ever since then has been that your friends are coming into your game and their behaviors are showing up in your game. So technologically, it was really cool. And from a user experience, it's had a couple of sparklers that are incredible, great user experiences, but it's also had some irritants that we've had to look at. How do we shackle and how do we tame and how do we change the user experience? All of a sudden the Drivatar system wasn't just using your data. So you might train it with a hundred laps. That's pretty ambitious that you actually did a hundred laps and various cars and various tracks. And that would train up the system for how you drive. And then we had trained the data that was on the disk and was in the game based on, you know, hundreds of thousands of our own laps right? All of a sudden going to Thunderhead, we had millions of laps within what? A week. It was crazy. And that allowed us to take a system that would classify And that's what neural nets do right? They're classifying you think about the joke in Silicon Valley, right? Not a hot dog.

- That's that's it.

- That's what AI does, it classifies. And the thing that classifies best is not A [chuckles] it can tell you, it's not this, it's not that. And the truth is that's magic. I know it sounds not very ambitious to most people, but it's pretty amazing. So all of a sudden we had millions of data points for it to classify. So the AI got very fast. It covered weird and spots. It did things we didn't expect. And we had to start like clamping that we didn't want it to do this behavior. But the problem is, it's a bit of a black box. You don't know why it's doing the behavior. That's what AI is. You feed in data and it feeds out what it wants to do. It doesn't tell you why. it's like a two year old, you just kind of have to take the output and then you can put a restrictor on it and have it do less of that or modify its output in some fashion. So the data was a huge boom. We started learning about how players drive. We started getting a laps. We didn't expect things got fast in ways we didn't expect they got slow in ways we didn't expect. And weird inaccuracy showed up in our system that we really didn't expect. Now I'm gonna have to dive super deep for a second. So the way that we classify a track is using what we call a Chi value. It's basically a value from zero to one where zero is the inside of the track. And one is the outside of the track. And when we record a line, it's a series of points. Now those points are said in that Chi value. So you might say that if you are towards the outside of the track, you were at like 0.9 chi value. And if you're at the inside of a track, maybe you're at a 0.1, or maybe you're lower than that like a 0.01. So all the way around every single track a track is expressed by these chi values, which gives you three lines an outside line and inside line. And then in the middle line, the line that AI is going to drive or that a player recorded in these chi values. Here's the issue when you wanna classify and generalize, you're trying to say this corner is like that corner and that corner is like this corner. And you're starting to stitch together virtual lines based on a whole bunch of information. Now it's gonna look at the lines ahead and the lines behind and new inaccuracies coming in, because the corkscrew at Laguna Saika is a very complex corner and you can't really generalize it to any other corner in the world. And the system may try and we don't have control over it cause the black box inaccuracy show up. Because if you think about a track, it's easy to conceptualize it as there's an outside and inside. But if you really think about a track, the outside and the inside move, a curbing shows up or doesn't show up, it gets a little bit of extra concrete over here. And so all of a sudden the scale of that Chi value changes because the track's really wide. And when you go from section to section, it changes in width. So the chi value is still accurate, but you're getting little changes that make it harder for the system to deal with. What that would end up with is lines that have an imperceptible small kink as they get replayed because they go from one track, which has no curbing to attract that has curbing or whatever, right? There's a whole host of these things that can make a difference. While the line follower sees the kink and says, "how am I gonna make that? Clearly I have to hit the brakes." And so we got this breaking behavior that showed up out of nowhere. So what do you do? You say, well, Hey, don't hit the brakes so sensitively kind of overlook some of that stuff, but now you've actually told the controller, "Hey, ignore some data." [chuckles] Sometimes you do need to hit the brakes, but this time we're telling you not to do it. So all of a sudden we're putting programming on top of the black box, which is more like traditional AI, we are actually programming if this, then that. So we kind of got a hybrid now where we've got a black box training some data, and then this thing on top of it, meanwhile, we wanted to keep some of the behavior your player did. So we wanted you to make contact. If you were a player that made contact players don't always appreciate having contact made with them. [chuckles] So we then limited that to only your friends. So strangers Drivatars weren't allowed to make contact, even if they wanted to. So this is yet again, another limitation put onto this set of data. The sparklers that happened were water cooler talk. We'd get people coming around and saying, "Hey, I drove against your Drivatar and it was doing this. Or it was doing that. It was really fast. It was really slow." And it kind of reminds you of them when you play in the game. These were awesome experiences, really delightful for our players, but they came along with a bag of behavior like brake checking that nobody wanted to see or getting pit maneuvered in a corner. Here's another complexity watching all of the other cars around you is very complex. In fact, most players have very poor perception of the cars around them. If they're not on screen in front of them, they don't realize there's a car in the rear quarter panel. Even real drivers struggle with that. Great drivers have almost like radar. They know where everybody is around them. Just like a great soccer player, a great basketball player. Now give that to an AI where we have discussions of performance and we can't all of a sudden have it go exponentially that every single opponent is tracking 16 other opponents. So we have to limit that. So for the most part, our AI is only watching one car. And if it's not you, it might run into you without even knowing. In fact it might even hit you and be continuing to hit and not know that it's hitting you at all. When we get more performance, we're able to apply that performance to watching more things. But it's just been one thing after another one way you can make things more efficient is you can run something offline, maybe on a server, or it could be run during load times. So that is when the game is loading, but before you're on track, we could run something and cache a table full of information. So for example, there's a Bugatti Veyron on track. It's a heavy car. It's incredibly powerful. It's a tricky car for AI to drive because its limits are ridiculous. It's so heavy. It is so performant We could run that track through a lightweight simulation during load time and then cache the results so that when the AI has to look up, how fast can I take this corner? Oh, through my simulation, I can take this at 60. So I should throw myself at it at 60. I have to break at this point to hit that, that allows us to run more AI on the track because we're now more performant. So every one of these cars could have look up to. Now, if our cars were all the same, then we wouldn't need as much. But the truth is it's not just a Veyron. It could be a Veyron that you've thrown upgrades on maybe racing tires or downforce or some other piece of technology. So we have to run in load time because the number of permutations across 700 cars and all of those different upgrades is innumerable. We could not cache that amount, even if we did it on a server. So we have to do that in load time. And we choose to do that and load time so that we're more performant in real time. But now there's other AI on the track and maybe there's a puddle or maybe there's rain, or maybe there's a human on the track. How am I gonna deal with this? Should I pitch, should I not pit am I damaged? I mean, there's a whole lot of things that are part of what we call race craft. Now, each one of these have their own set of problems. Places where you can apply learning neural networks, deep neural networks, places where you're better off with caching and then places where you look to simplify. And we've got numerous places that we have looked at simplifying over the years. And honestly when I say, when we get more powerful boxes we get greedy. That's what we remove. We remove simplifications on things like the AI or the physics engine itself. We will do new research projects to find new complexities and real-world physics and integrate them into our game. And then in low time, real time or even offline, we will use caching techniques to reduce some of that complexity. So we can run at 60 frames per second. So in most racing games. There's this concept called rubber banding. It's often highly maligned by players, but I'm gonna try and impartially explain why it exists. And then I can tell you how we use it. So in racing, you don't restart very often, right? So you start a race let's say it's five minute long. You run off the track and getting back on track is a little bit difficult so as a result, you fall way behind the opponent players. We needed a way for you to catch up to those players or else you feel like all is lost. And we have a rewind system in Forza Motorsport and Forza horizon, which also helps take care of this. So a catastrophic failure, you can actually rewind and redo it, Drivatar and rewind help us because we've got a myriad of lines to follow. And we have the ability for you to fix your catastrophic mistake, but you still end up with wanting to have more pack around you. It's just more entertaining racing. So we've looked at a couple different ways of doing rubber band. We don't change the opponents characteristics, but if they're far ahead of you, we start scaling back their torque. We start scaling down their friction. We start adding weight. We play with aerodynamics. The reason we have to play with so many parameters, because if you play with any one of them too much, it does crazy stuff, right? You can't double the torque. The tires will just spin. You can't do too much air friction or car slows to almost a crawl in like 160 miles per hour. So we've tried to be very sparing of where we use it. Because if you come in contact with the car that has, let's say 20, 30% more friction in its tires, it's not gonna look right. It's not gonna feel right when you collide them. Cause they have more grip. I think the reason it's maligned by players is when it's put on really heavy and you feel like you are gonna win the race and on the final 500 meters, some car comes whizzing up from behind you and pass you at the last second. That is really irritating. That's something we've avoided, but there's a connection between rubber banding and the inaccuracies that have come out of our black box. We have a setting in Motorsport, it's in horizon as well called unbeatable unbeatable AI be is in fact beatable all of our professional eSports players toy with that AI, they are so fast and they are so good that they can beat that AI without problem, we could not make it fast enough because of these kinks in the Chi value, it just would break itself a little bit. It would sort of stop occasionally. [car engine roaring] The other issue is the line follower So it sees, it wants to follow this line at a certain speed, but our line follower was digital. So that would mean it would full break or it would full throttle, creating a controller that intuitively knows exactly how much throttle and how much break to give on a very complex system is pretty hard. So we were doing it that way. And then we would let the AI use our abs system and our traction control system to limit how much power it got. But that's not the fastest way of driving our pro drivers myself even I don't drive with traction control. Often it don't drive with abs, which means I get more out of the tires, more out of the car, which means I'm fast with that car. The AI is already driving with one hand tied behind his back by using these assists because it's so binary. So we ended up having to use rubber banding, not just in the pocket when it's around you, but actually to close the gap to drivers, not even pro drivers, but drivers like me since then, we've done a ton of experimentation. This is what's so exciting. We've been experimenting with a new controller. That's able to give analog and we've turned off the assists. So the AI now drives it can do modulator throttle. It can do modulator braking. That's awesome. We also got rid of the Chi value error by going to a new system of classifying the track, basically as a result, her test opponents and our test bed have gotten so crazy fast. It's ridiculous. [car engine roaring] Our hope is to get rid of rubber banding for difficulty entirely, have it completely off. And the only reason there'd be any rubber banding ever would be outside of the pocket when you've had a catastrophic failure and we don't wanna stop the opponents. We never do that. We wanna slow them down a bit. That would be a great use of rubber band. [soft music] So we have organized a team. That's just been doing experiments on Drivatar as well as on neural networks and how they could be applied to different places in our game, our build system, our tools all over the place. And then we see the results of how much faster the opponents get with and without cheating. What about a new controller? What if we cache do things in real time and they just try new experiments. We see what works. We design something. We think players are gonna react this way and it turns out they react that way. [car engine roaring] [player cheering] You gotta be really careful when you ask players what they want. And even when you ask them what they did. And we found that over the years that players have told us they use rewind x amount. And then we look at the data and they actually use it sometimes 30, sometimes 50, sometimes a 100% more than they remembered using it. The feature was incredibly well used by people, but it wasn't always well loved. It broke up the action. It caused problems with our line recording for AI. So that was one of those features. We had to look at what a players perceive versus what a players actually do. Same thing with the self reporting from players on how rough the AI was. We had to look at what was really causing the AI opponent to run into them was it that the AI didn't see them? Which is a problem. Was it, the AI didn't have control of the car? Because it was being too binary with its inputs. Or was it that the AI thought I should hit this person because that's what I was trained to do by my Drivatar? Well, we can take care of the third one. We can, you know, get rid of that, but we needed to do radical experiments to figure out how to get the car to control better and how to give it better vision.

- [Commentator] Oh yeah, drivers coming from everywhere now its a party

- We've constantly had to grapple and wrestle with what players perceive and what they expect and what we're capable of. I love what we figured out over the years though. I think we've made a much more robust system. [sheep bleats] And in I think it was Forza Motorsports six, we added a parameter where you could toggle on or off AI aggression, just on your friend opponent. Yes, we found players that really loved it. They loved having the rougher race and they loved having the reminder of Oh yeah, that drives like John versus I need a predictable field that never hits me. The biggest thing we're tackling is we've been improving our controller. don't all the way break or all the way accelerate and turning off those assists, the Drivatar and the actually AI is able to control the car in a very nuanced fashion the player can a more complex simulation of the car heading into our caching of that player data so that when the AI is trying to play the car back, it's trying to control that car. It has a much better intuition of if I brake the car is gonna slow down at this rate judging what is a collision and what's not a collision and whose fault it is. That's a great thing for deep neural nets. And we've been applying that. The thing that's impressed me, the most that we've been playing with is by turning off all the assists, all of the scalers, any sort of cheats or anything like that, but improving the line value. The AI has gotten so fast. So it destroys our unbeatable setting, which was the one that I could sometimes compete with in the unbeatable settings cheating. And now it's being destroyed by this car controller and fixed line follower. These are just cool things to see happen in technology, and so we're doing that with the research team in Redmond, and we're just playing around with this stuff and seeing where it goes and where it's useful. We started looking at neural networks pretty early. it was part of the original Xbox and the hard drive 2005. I mean, there weren't a lot of game teams looking at neural networks as a result. We maybe we got jaded. Maybe we got experienced. I don't know how you want to say it. But when all of a sudden everyone was talking about solve everything with AI, we had a very realistic eye on that. There are some things AI and DNN, Deep Neural Networks are incredibly powerful for, but they're not like peanut butter. They're not good on everything, right? You got to decide where you put them. So a lot of the learning has been the right amount of experimentation to make these features come to life. AI doesn't just have to be applied to opponents. It can be applied to build systems. It can be applied to optimization. It can be applied to a whole lot of different areas of your pipeline and your experience. So I think a lot of the learning we've had is going really eyes wide open into deep neural networks. It's a black box. You put in a ton of data, but what you're gonna get out of that is gonna be a bit of a black box. You've got to take what you get. And then you're gonna have to add scalers and things on top. So it's not this like one size fits all solution. You're kind of doing a deal with the devil and it's powerful. It's powerful, but you got to know what you're doing, [car engine roaring] but it's just been super interesting. We learned so much about our players and what they love. We learned more about racing in the real world. And on top of that, we learned more about what our technology can and can't do. And what's really magic about it. [upbeat music]