The rise of the audio-only video game

0:00Support for this show comes from MongoDB. You're a developer who wants to innovate. Instead, you're stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It's a flexible, unified platform that's built for developers by developers. MongoDB is ACID compliant, enterprise ready, with the capabilities you need to ship AI apps fast. That's why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at mongodb.com slash build.

1:01This week's sense is hearing. Sound. And I want to start by telling you about a game I've been playing a lot over the last few weeks. It's called Blind Drive. Start driving. What the hell? Wait! Hey! You have ears on me. Here's the story of Blind Drive, which is available for lots of platforms, but I've been playing it on my phone. You're the main character. You thought you were taking part in a scientific study, but suddenly you find yourself driving a car against traffic with a blindfold on. You can't see anything. The only way to survive is by hearing the cars, trucks, and cops, and bikers, and whatever else coming at you, and then tapping on the phone screen to steer away from them.

1:44The longer you make it, the higher your score, and the happier you make this scary-sounding guy on the phone. I've never played a game quite like this before. It's really intense and really fun, and I find myself way more focused on it because it's all based on sound. I'm on edge all the time playing it. It's kind of a lot, honestly. It's been fun to play this too because it feels so different than what I'm used to. We live in such visual times right now. We scroll vertical video, we look at our TVs, we put on VR headsets, we play these huge, gorgeous open world games. There's just always so much to look at all the time.

2:23But there's something really powerful and different about audio. I mean, you get it. You listen to podcasts. Podcasts are proof of how cool and powerful audio can be. But I'd never really even thought about audio as more than just a feature of video games. It's a way to make explosions sound more real, or at the very most alert you to when someone is sneaking up behind you. But there's actually a rich history of audio games, video games without the pictures, and some reasons to think those games might be coming back. One of the people responsible for a big part of that history is this guy. That's right.

2:57As a baby, on a BBC Model B, because I'm that old, and it was my first game. It was, in quotes, my first video game. Paul's route into audio games was a windy one that actually makes perfect sense in retrospect. He was really into games as a kid. He was writing code on a BBC Model B in the 80s by actually copying it out of a magazine. That was how you wrote code back then. Then Paul got into the music business and the broadcast radio business, and then he actually ended up directing a video game version of the game show You Don't Know Jack. It's time for the show where high culture and pop culture collide.

3:36Yeah, woo! All right, we got ourselves one, two, three players on board the flight today. Making that game, he told me, was mostly about making the audio. All of the effort of that game sort of went into phenomenal writing, good game design. and smoke and mirrors that would make it seem like you were talking that you the player were inside a live tv game show and doing that kind of reawakened my my love for technology and creative technology and i've really been doing that ever since at this point paul had built lots of different kinds of games but he had always thought it would be cool to make one that was all audio for a couple of reasons actually there was a broadcaster in the uk called channel four They were originally sort of looking for the games that were accessible, the games that could be played by communities who were generally excluded from playing games.

4:27And so I didn't want to make a game for people with a quote unquote disability. I wanted to make games that everyone could play and that everyone would love without any compromise. Eventually, he decides to do just that, build an audio only game. At this point, it's about 2010, which is important because there was a big technological development that happened in 2010. The iPhone 4 came out. It had a bunch of new features like the retina display and the selfie camera, and it eventually was the one that went through all that antenna gate stuff. Remember that? The whole Steve Jobs, you're holding it wrong thing.

5:01But for Paul, there was a significantly less publicized new feature that really, really, really mattered. iPhone 4 was the first of Apple's devices that had hardware acceleration for floating point math. Now, why is that important? It's important because we were doing real-time binaural synthesis on a mobile device. So let's talk about binaural audio. I have two ears. Most people have two ears, but not everyone has two ears. And you might think that therefore that we listen in stereo. Well, we don't. The shape of your, the fleshy bit of your ear, the pinna, has got these folds in it, these lumps and folds.

5:37You probably know the basics of this. Sound hits your ears at different times, your brain does a huge amount of work to both make all of that sound make sense and also to intuit where that sound is coming from based on when it hits all those lumps and folds. What it turns out that you can do is that you can get an artificial head with two good microphones where the ears would be, and you can blaze some white noise at that head from lots and lots of different directions. And you can work out how the human head hears those sounds from different sources. And you can express that in terms of a big table of numbers.

6:14Now what I can do is I can take away the white noise and I can apply those numbers to an arbitrary bit of audio, an arbitrary audio file, and I can get the right bit of the right number from where I want this sound to sound like where it's coming from. I'm way oversimplifying here, but basically once you can map sound the way Paul is describing, you can manipulate it. Delay sound coming from a spot on the right just a tiny bit, and it'll sound like the thing over there is further away. Increase sound in the front and decrease it behind, and boom, you have something right smack in front of your face.

6:47This is a simple enough concept, but it's really, really tricky math. And you've got to do that, by the way, 44 ,000 times a second, or 48 ,000 times a second, because that's a typical sampling rate for digital audio. So as you can see, that requires a lot of math. Pre-iPhone 4, there wasn't a device that could do it. And Apple helped us out a lot, by the way, with the Accelerate framework, which came in at that point, which enabled us to do this. So it's actually pretty simple. It's the same as in any video game, right? Imagine you've got an X and Y coordinates. You've got a grid in front of you.

7:21There's X and Y. And as objects move relative to the player, you can change the X axis and you can change the Y axis and you can calculate where the object is. So it's just the same in that regard between a video game and an audio game. In a video game, you're just saying, well, you know, that the monster has walked diagonally away from you. So now I can change the X axis and the Y axis position and I can just render that object and, you know, I'm watching it on my screen. Everyone's happy. And you can do exactly the same thing with audio games. It's just that instead of rendering it using whatever technique you're using to render an object in video.

7:54You're just applying a bit of math to the frequencies of the audio. And now all of a sudden the human being hears it. It really hears it. The pictures are better on radio, as they say. What he's describing, I should say, is super normal now. It's just spatial audio. The idea that you can anchor something in place and then as you turn your head or the object itself moves, the sound moves with it. Lots of devices have this now, and you can use it to all kinds of good and dumb effects. But at the time, it was revolutionary, especially when you also had a mobile phone with you that could accurately track your movement in space.

8:30We had to have an input scheme, an input control scheme in terms of changing the relative position of the player and the audio that was incredibly precise. You could do it with a mouse, but there were some drawbacks to that. It's far, far better if the human being themselves is doing the turning like the head is turning itself or the body is turning and with vision pro and with the current generation of spatial airpods and so on well we're used to that now we actually understand that now back in the day that didn't exist so we kind of had to kind of create the hardware effect that you get from spatialized headphones without the existence of specialized headphones and the only way that we could really see to do that was with the accelerometer and the magnetometer inside iPhones.

9:15So with all this suddenly possible, Benin and his team at a company called Something Else set out to build an audio-only game. They ended up calling it Papa Sangre. Now do as I say, or very soon you'll forget everything and everyone you've ever known. The plot of Papa Sangre is pretty simple. I'd feel bad spoiling it for you, but the game is 13 years old and you literally can't download it anymore. So here goes. You're dead. You're stuck in Papa Sangre's kingdom. And the only way out is by completing a journey through five different palaces. All the bad guys and monsters are everywhere and they're trying to stop you.

9:55All those bad guys and monsters respond to sound. So you have to move as quietly as you can, while also looking for clues and talking to characters and generally trying to figure out what's going on and where to go. Because you can't see anything. There's nothing to see. The ice is thin. You need to be careful. Making a game like this work is on one level not that complicated. You can sort of do anything you want in audio. You can make audio pong just by using spatial audio to help you understand where the paddle and ball are. You can make audio Tetris, Paul also says, which seems wild. You can kind of make audio anything.

10:36In the second version of Papa Sangre, there was even a Duck Hunt-style shooting game where you just followed the sound of the ducks, and you turn and shoot the ducks at the right moment. Paul and his team built a few of these games over time, before deciding that wasn't what they wanted to do. It didn't feel right. We gotta take a break, and then I'll tell you why. Thank you.

11:27network and get qualified candidates that you can manage all in one place. LinkedIn's new features can help you write job descriptions and then quickly get your job in front of the right people with deep candidate insights. Either post your job for free or pay to promote. Promoted jobs get three times more qualified applicants. At the end of the day, the most important thing to your small business is the quality of candidates. And with LinkedIn, you can feel confident that you're getting the best. Based on LinkedIn data, 72 % of SMBs using LinkedIn say that it helps them find high-quality candidates.

12:00Find out why more than 2.5 million small businesses use LinkedIn for hiring today. Find your next great hire on LinkedIn. Post your job for free at linkedin.com slash track. That's linkedin.com slash track to post your job for free. Terms and conditions apply.

12:20Welcome back. So at this point in our story, it's right around 2013. After working on Papa Sangre, the team at Something Else made a game called Audio Defense, which is just a straightforward shooter game. This is the kind of thing Paul meant when he talked about Audio Pong and Audio Tetris. It's just a normal game with normal mechanics, but with no pictures. In Audio Defense, there are waves of bad guys coming at you. You shoot them. That's the whole bit. And it's all done just in audio. there are zombies you can hear them but you can't see them turn carefully until the zombie sound is in front of you it was a very interesting game in that regard it was a much more sort of casual game sort of pick up put down and it was effective and it worked i think what what we realized after releasing that game was that the natural benefits of audio games meant that more artistic more carefully considered more complex experiences, certainly for our team with our skills, would probably find a better market fit than that game.

13:26You would think that that game, if people like the other Papasangro games, that audio defense would just find a larger audience. After all, it's the same kind of mechanic, but it's much, much more simple and it's much easier to understand. Funnily enough, that didn't happen. The game didn't do as well as the other games. The Something Else team really thought audio defense was going to be huge. But in retrospect, Paul seems to think that it kind of makes sense that it wasn't. Because the best audio games aren't just audio versions of video games. There are some cool accessibility upsides to that, for sure, letting more people get access to these kinds of games.

14:01But he's more intrigued by the idea of what you can only do in an audio game. It's related to the fact that the earliest way that humans told stories to each would of course be around you know be verbally orally around around the campfire and there is a mode and a pacing to that kind of storytelling and a way that words can be used to conjure up images and ideas in the listener that all of a sudden if you're just purely concentrating on in terms of an audio game you can also use and a story told orally is very different to a story told graphically. There are reasons of pacing, there are reasons of how different ideas are introduced and worked on.

14:46But mainly, again, it's because there's a complicity between the listener and the storyteller that has a different kind of suspension of disbelief or a different kind of enactment of belief, a different kind of purposeful volition on the part of the audience. And it means that you have a, I'm not going to say it's better or worse. I personally think gets more interesting. You have a broader palette of things that you can do. Again, I've sort of said it's more like a novel. The subtlety and the complexity of the ideas that you can introduce and develop are just different from what you contend to do in video games.

15:21There's this long-standing debate in gaming about the role of storytelling. There are those who think it's super important, that cutscenes are crucial, that you can do as much or more good narrative work in a game as you can in a movie. And then there are those who just smash the A button to get through cutscenes. I should confess, I'm mostly one of those people. But in an audio game, there seems to just be more space, more freedom to just let the story be the whole thing. Really, you have no choice but to be more thoughtful about it, or the whole thing will just devolve into chaos. So if I'm playing a video game, think about how many different pieces of information I can have on that screen at the same time.

15:58You know, if you think about something like any modern highly polished triple a game you've typically got n numbers of different things about the status of you as a let's say it's a first person shooter or first person game anyway you've got n number of things that demonstrate the status of your avatar of you in the game plus you've got the relative position of n number of different objects plus you've got the different choice of weapon or object in your in your hand there's just hundreds of different pieces of information you know relative position of your other players and so on and human beings aren't that great when it comes to discriminating usefully discriminating lots of different sound sources at the same time we tend to try and focus on a small amount one or two sound sources at the same time and there's an upper limit to how many you can really have simultaneously before the human being just going to go you know it's fine if i'm at a dinner party or if i'm in a nightclub or if i'm on the street because i have a bunch of other things in terms of situational context knowledge contextual knowledge that help me focus on the different things that are actually important at that moment but if i'm creating an entirely artificial experience then i have to be much much more directed and much much more careful in terms of what i'm presenting to the player for them to to be able to focus on and get information which is going to help them enjoy the system because it It needs to be fun.

17:18And trying to, you know, playing a game of what should I be listening to now is not fun. Normally, when you get to a big moment in a video game, there is so much going on at all times. But here is a climactic moment in Papa Sangre. Just listen to how it goes.

17:37Feed the hogs chicken meat, not your flesh. This all sounds like kind of a niche thing now, but Papa Sangre was a big hit, and Papa Sangre 2 even more so. When that second game, the sequel, came out in 2013, it was one of the best-reviewed games of the year, with a 92 % score on Metacritic. For most of the year, it was literally the best-reviewed iPhone game of the year. And folks who classified themselves as blind lost their shit. Someone had gone out and had tried to to assemble a team of talented people to make an experience that had absolutely no compromise in terms of its playability and its enjoyment.

18:21I just went back and looked, by the way, and it was actually tied for number one in mobile games that year. The company behind it, Paul's company, something else, made a couple of other games and ended up selling to Sony. And then Sony, for contractual reasons Paul didn't really want to get into, but did seem sort of sad about, just shut the whole project down. That was almost a decade ago now, and Paul has gone on to lots of other things. But the more he and I talked, the more it became pretty obvious that he's still thinking about audio games. And he thinks that actually 2024, this incredibly visual world we live in, might be the perfect time for it.

18:55I'll tell you why right after the break.

19:07Support for the show comes from Crucible Moments, a podcast from Sequoia Capital. We've all had pivotal decision points in our lives that, whether we know it or not at the time, changed everything. This is especially true in business. Like, did you know that autonomous drone delivery company, Zipline, originally produced a robotic toy? Or that Bolt went from an Estonian transportation company to one of the largest rideshare and food delivery platforms in the world? That's what Crucible Moments is all about. Deep diving into make-or-break moments that set the course for some of the most important tech companies of our time, with interviews from some of the key players that made these companies a success.

19:45Hosted by Sequoia Capital's managing partner, Rule Off Botha, Crucible Moments is back for a new season with stories of companies as they navigated the most consequential crossroads in their journeys. Hear conversations with leaders at Zipline, Stripe, Palo Alto Networks, Klarna, Supercell, and more. Subscribe to Season 3 of Crucible Moments and catch up on Seasons 1 and 2 at cruciblemoments.com, on YouTube, or wherever you get your podcasts. Listen to Crucible Moments today.

20:21Welcome back. So, let's go back to 2010, when the iPhone 4 came out and it was this perfect confluence of technology that made a really cool spatial audio game possible for the first time. Remember that moment? Well, Paul Benin thinks we're at another one of those moments right now. You know, I've never been more excited about the possibilities for spatial audio in games. I really haven't. There was this alchemy that happened at the beginning of the Papa Sangre titles where creative idea and technology and the ecosystem around it sort of came together to make those games happen. And there's another point like that right now.

20:54Spatial computing, for example, in quotes, VR. Meta has sold more Quest 3s and Xboxes in the last couple of years. The hardware is out there for experiences like this. And by hardware, I also mean things like small headphones that have really good noise cancelling. You're the first person I've ever talked to who thinks about the Vision Pro as an audio device. Yeah. And I love that. But I'm curious kind of how you think about what that could be. So I think it's interesting if you look at the difference, like the philosophical difference between something like Vision Pro and Meta's excellent hardware, which has got lots and lots of benefits different ways.

21:29But the overall difference where Apple is really concentrated has been on being able to bring virtual objects into the space that you are in. It literally is spatial, you know, spatial computing is their shtick. And it's very good at that. And those objects don't need to be visual. They can be audible and still be in line with Apple's design philosophy. And if you think about what the early Papasangro games were about, that's what they were about. They were about objects that you couldn't see in the physical space that you were in. And that's how those games worked. It's literally how those games worked.

22:03And that fits very closely with Apple's design principles for Vision Pro. It doesn't exclude it from things like Quest as well. And I'm sure that if and when the Quest Pro 2 comes out, or the new hardware that we know that's licensing, Horizon OS, I'm sure that some of that's going to be able to compete in terms of low latency, high definition, pass-through video. So we'll see how long that advantage is maintained. But these devices are great for the kind of thing that we're talking about, I would say. I think they're phenomenal. They're a little heavier, right? Sure. And you don't need to have that thing strapped to your head.

22:37The AirPods Pro that I'm wearing right now are an ideal solution for this kind of thing. Yeah, what are the kind of specs that matter there? I mean, you know, again, as we talk about screens, we're in this incredible GPU race, and everybody's trying to make the higher density screens with higher refresh rates. I'm sure there are other kind of measurable specs that are getting better over time. But what are those things that you would look at as sort of raw technical advancements? I think that we've reached a stage where that's no longer an issue. Okay. In terms of speakers in your ears, that's no longer an issue.

23:08Bluetooth, I mentioned earlier on that you need to have an output that's proportional to input. And latency is very important. But Apple is targeted, I think it's 12 milliseconds or it might be 8 milliseconds or something like that in terms of response to input. and that is fine. It's fine. It's more than just the audio stuff too. Think about what you can do with great voice recognition and generative AI that can create a personalized game script on the fly. Or the fact that your phone's location services work indoors now, so you can actually move through a game by moving through your living room.

23:43And think about what's even possible in multiplayer gaming, when instead of rendering someone's weird avatar with no legs or a crazy looking face you can just bring their voice into the space with you to stumble across your friend in a space and find out that they're there is the kind of surprise and the kind of magic that would be phenomenal in one of these games and the ability to to shout to your friend that you need some help and not just like if you're playing call of duty but in a way that's weirdly more practical because it's the only way that you can communicate like if i'm playing call of duty of course i can leave and I can leave some loot for anyone, for a random person.

24:19But if I'm running out of ammunition or something, or you have a resource which I need to unlock this door in the next five seconds, and I have the resource, the tension of me throwing that object to you or getting over to you to give you that object is a set of really interesting things that are fun to explore in a playful experience. They're absolutely on the roadmap. One game I think a lot about in this context is Zombies Run, which is sort of a hybrid of an exercise app and a video game. The idea is that the game tells a story as you run, and then when the zombies get too close, you have to pick up the pace and run faster.

24:52And the game actually knows and responds to how fast you're moving. Yeah, go. Raise the gates, please. You know what? Just raise the gates. Alrighty then. I love Zombies Run, by the way. Cannot recommend it highly enough. And now I'm just imagining an audio AR game that follows you around as you go about your day, and then every once in a while just adds zombies and makes you hide or run away. I don't know, maybe your whole office could play together and it would interrupt all your meetings. This sounds awesome to me, and it only works in audio. There really haven't been a lot of great audio games in recent years.