#200 Trevor Back: How Speechmatics is Shaping the Future of Conversational AI

0:00To me, speech is clearly going to be a core part of any future AGI stack. Speechmatics has the best in class technology. And so there's a huge opportunity here for us to really define what does it mean to have speech in AGI. We've got hundreds of large enterprise customers that are transcribing millions of hours every month. Speechmatics is a very horizontal offering. We go out to a lot of different industries, a lot of different use cases. Hi, I'm Craig Smith, and this is Eye on AI. Today, we're diving into the world of voice-powered AI with Trevor Back, Chief Product Officer at Speechmatics, which just launched Flow, F-L-O-W, a groundbreaking conversational AI tool that combines speech recognition, large language models, and text-to-speech synthesis.

0:53In our conversation, Trevor shares insight into Flow's development, its potential applications across various industries, and their vision for a future where voice becomes the primary way we interact with technology. I hope you find the conversation as fascinating as I did. Hi, listeners. It's a pleasure to talk to you from Speechmatics, and I'm joined today by my friend Humphrey. Hello, everyone. I'm Humphrey. Nice to be on the Eye on AI podcast. Delighted to be here. Thank you, Amelia. So, Humphrey, we're going to listen to Craig and Trevor talk about Speechmatics. Speechmatics is actually known for their speech generation capabilities, not just recognition.

1:32They've developed some remarkable technology for generating natural-sounding speech, haven't they? I hope everyone enjoys this conversation between Craig and Trevor. And now I'll just sit back and listen. Why don't we go ahead, Trevor, and you can give us your background, and then we'll talk about Speechmatics. Yeah, sure. So it's great to be speaking to you today. Thank you so much for the time. I guess my background is basically I loved space as a kid. And so, you know, I was the guy with the space posters up all over my bedroom's wall. And so I ended up doing a PhD in computational astrophysics, simulating the early universe.

2:12But as I got near to the end of my PhD, I kind of realized that about five to ten people in the world would care about what research I was doing. and because I was simulating the early universe, it's probably never observable to prove the simulations I was running, let alone in my lifetime. And so that's why I went into AI looking for a bit more real-world impact. When was that? When did you make the switch? Yeah, so it was back in 2012. So I finished my PhD in the University of Edinburgh in 2012 and randomly got introduced to the co-founders of DeepMind. back when the company was about sort of 15 people or so.

2:54And I joined them as the first product manager there. Oh, wow. That's wonderful. Yeah. And Edward, then you were at Jeff Hinton's alma mater, is that right? Well, so I was based up at the Royal Observatory. So I had the cycle up the hill every day, which kept me nice and fit. And I like to say that I really should have befriended somebody in the computer science department because, yeah, having somebody that really knew how to code would have got my PhD done a lot faster than me doing it by myself. Yeah. And Jeff then that year was, he and Ilya and Alex Kurjevsky were making the big breakthrough.

3:37Were you aware of that at the time? So, yeah, I think I joined DeepMind around the 12th of April. And I think it was literally a week either side of that. AlexNet came out on ImageNet and took the world by storm. And so, yeah, it was good timing for me to jump into the AI domain. Yeah. And who did you meet at DeepMind that took you there? Yeah, so I initially met with Mustafa Suleiman, who is one of the co-founders there, and spent some time with him and with a guy called Ben Coppin, who was the first sort of applied AI person and engineering manager, and then interviewed with Demis Hassabis and Shane Legg and a few of the other senior leaders there.

4:32And yeah, I was grilled for a few hours, and then an offer was made. It was great. Wow, that's really fantastic. I mean, what an incredible group of people. And I'm surprised that you left. So how did you get to Speechmatics? Yeah, so I spent almost a decade at DeepMind, always on the applied side. So in the very early days, we were looking for a variety of different applications. We ended up working on an iOS game as Demis' background is heavily in games. I spent most of my interview with him talking about theme park and how many hours of my youth I'd wasted on that. And we were also working on image recognition for fashion back in the day, looking at shape, pattern, color type aspects from the image.

5:26And then when we were acquired by Google in 2014, I ended up working on a whole range of applications, including YouTube recommendations, Google ads, self-driving car efforts, wearables, activity recognition. And then around 2015, DeepMind decided to start up its healthcare project. So I helped sort of co-start that and spent sort of four or five years working on a range of applications in healthcare, including in ophthalmology with Moorfields Eye Hospital. In fact, when I heard your podcast, Eye on AI, we had that same play in our project there. But then I moved and transitioned to DeepMind Science and ended up working with the AlphaFold team and explored sort of the early commercialization efforts of that, which morphed into isomorphic labs.

6:23But essentially, I kind of felt like Isomorphic Labs was going to be another quite research focused effort. And I'd spent a lot of time working on incredible projects with amazing algorithms, incredible people, just absolutely fantastic minds. But I always felt like we had not quite made that leap from the research into having that real world impact. And so I had an opportunity to start my own startup. So I left DeepMind to do that. Spent a couple of years there working on applications in supply chain, in manufacturing, and in e-commerce, where I was particularly focused on the application of these algorithms to real-world problems.

7:10But then the Speechmatics offer came around. And to me, speech is clearly going to be a core part of any future AGI stack. speechmatics has the best in class technology and so there's a huge opportunity here for us to really define what does it mean to have speech in AGI yeah well that's that's fascinating um the um uh yeah I've had Oriol Vinalis on the podcast if you were working on AlphaFold I'm sure you know him uh the uh speechmatics you you are focused on speech recognition is that right not speech generation that's right yeah we we have a mission to understand every voice and so research here is very focused on how do you get the most out of audio coming in um and uh there is a lot that goes into recognizing text from speech.

8:15But obviously, there's just a huge amount of other information in that audio domain as well, as well as just the words that were said. Equally, I think there's a fallacy that sort of speech is solved. I like to say that there's kind of two people in the world. There's people that think the speech is solved and there's people that have used Siri or Alexa. Right. Right. And so, you know, we're still a long way from, you know, really being able to have a speech Turing test. Right. A way for us to have a Turing test via our voices where we can't tell whether it's a computer or or a human on the other end.

8:55listening, you mean, or, or a human on the other end or a computer speaking, because, because with, with, you mentioned Siri and, and I actually built an, an app, um, an LLM based app for iOS and you're, you're tied into the, uh, iOS, uh, speech to text and it's terrible. Not, you know, it's not terrible, but it's, it's, it's very little nuance. So I find that, yeah, so the built-in iOS dictation and a lot of the other providers out there, everyone's generally good at recognizing voices like mine and yours, right? With a clear English accent in a nice quiet background with not a lot of other speakers going on.

9:48Now, we know that that's not the case for a huge amount of people in the world. There's thousands of languages spoken in the world. There's millions and billions of non-English speakers who don't have the same experience through speech recognition. Then you don't have to go further than the UK or the US to realize there's a huge range of different accents and dialects and ways of people speaking. And so really, it's about how do you get to the point for the speech recognition to be able to understand every voice and not just the standard voice. And you also want to be able to do that in real world scenarios, right?

10:32So if you think about sitting in a cafe, and if you're having a conversation with people on WhatsApp or any tech service while you're sat in that cafe, it's a send a note, receive a note, send a note, receive a note. It's a very sort of structured conversation. And so you can imagine lots of AI tools being helpful in that scenario. But if you imagine them bringing your friends into that cafe and having a conversation with them, it's much more dynamic. It's much more fluid. You can imagine closing your eyes, still having the same conversation, recognizing speakers from other tables, but ignoring them.

11:13If you dropped Siri or Alexa or a lot of services, it would just completely fall over and fail. So the challenge becomes, how do you understand, recognize, and be responsive in an environment like that in a way that is human? And so for that, you need to be able to, in my eyes, understand what was said, who said it, and how it was said. And so we're still not there with all three of those things. And so that's what we here at Speechmatics are really focusing on doing. Yeah. And I'm really interested in the research and science behind it. But for application, where do you see the largest application?

11:58Is it in transcription or dictation? So we already provide services to hundreds of different customers. So SpeechMatic has been going for just over a decade, although the founder who was working on recurrent neural networks back in the 1980s for speech recognition would say it's been going for a lot longer. we've got hundreds of large enterprise customers that are transcribing millions of hours every month speechmatics is a very like horizontal offering we go out to a lot of different industries a lot of different use cases some of our biggest industries includes the media things like live captioning on sports games those types of things we've got a lot of customers in call center applications, that side of things.

12:56We've got a lot of customers in ed tech, making their lectures and other things accessible for a broader audience. And then we've got government defense and a lot of other areas that we can go into. So I think there's a huge number of use cases for ASR. But I do think that as AI has progressed, and particularly the impact of large language models, means that suddenly you can take a large amount of unprocessed raw transcript in text form and very quickly and easily pull insights, value, actionable information from these raw large text files. And so suddenly there's this wealth of information that can come from turning your audio into text and then using large language models to do something value with it.

13:53And so we're seeing just a massive increase in the number of use cases and the number of customers that really want to leverage ASR for a huge array of new domains. Yeah. So talk a little bit about the science behind it. I mean, in what advances has Speechmatics made to make your system stronger or superior, whichever superlative you want to use? Yeah, so our favorite metric is really focused in on that high accuracy and more recently at low latency in particular as well. And so the company has always built the system to be focused on how do we achieve the highest accuracy? That is for our core service, but it's also for any additional new language that we build.

14:54We offer across 50 plus languages now, but we don't just release any language. We only release a language once we've got to a certain accuracy that we're happy with. Now, the way we get there is quite different from the way a lot of others go. So a lot of our competitors build end-to-end models where they build a huge data set, pump it through one large single end-to-end model and get the output out. That enables them to provide one single model that's multilingual, but there's no free lunch in machine learning, right? So any breadth that you add there will take away from the depth that you get.

15:35And so you see a decrease, a quite rapid decrease in accuracy as you get out to the sort of lower range of languages where there's less data, less representation of that language in the data set. What we do is we utilize multiple techniques to get the best out of both worlds. And so we train a very large self-supervised, what we call body, which is unlabeled data, millions of hours. It's a bit like an end-to-end model, but it's the raw audio that goes in. And this learns a lot of the sounds, like a lot of the noises and sounds that we make when we speak are similar across languages. And so it learns the representation of the audio data.

16:24we then have a separate body that we call the acoustic body and that's where we train on labeled data but it uses the self-supervised learning body to like as a feed into it so it uses the embeddings from self-supervised to help improve it this enables us to get very high accuracy on a low amount of labeled data which is why we get such high accuracies because even in english where there is a large amount of labeled data, it also means we get high accuracies on the less well-represented data in that. We get better accents, we get better dialects, we get better localizations. Then for languages where there's a lot less labeled data, and labeling audio data is very expensive, it means that we can get to very high accuracies on 100X less data than our competitors.

17:16So we get that best of both worlds in our single one. Yeah, on the less, on languages with less data available, I've spoken to people that work on, excuse me, that work on indigenous languages, languages that are disappearing. Do you augment the supervised learning, the label data for those cases? Yeah, so we don't do indigenous languages at the moment. One of the reasons is because when you use this low amount of label data, you want the data to then be as accurately labeled as physically possible. And so we use a lot of third-party providers to provide very high transcripts, very high-quality transcripts of the low amount of data.

18:22And that is a lot harder to do for some of the less well-known. We've got out to about 50 languages, And we're continuing to try and expand that. We hope to get another 10, 20 by the end of this year, for instance. But we tend to still be focused on the languages where there is also some commercial value at the moment. But our system could very easily be used to help save some of these languages if the right data was available for us to train on. Yeah. And the unsupervised model, is it looking at phonemes or different parts of speech? I've talked to people that are building music generation software and they focus on individual notes of individual instruments and then build up from there.

19:18That's right. So we, yeah, that is very much focused on the phoneme level, phoneme level of understanding at the current time. And so, you know, the team has worked on that since I think 2018 was the first time we started really playing around with transformers for doing that type of methodology. And so we built a lot of useful structures and ways of sort of augmenting the data such that transformers work really well on it. One of the things we are quite excited about for the future, though, is multiscale representation. So currently, we sort of break the data down into phoneme type temporal cuts.

20:03But obviously, what we're saying has some impact on the word level in terms of recognizing what words have been said. but it's also very useful to know longer timescales what's going on there and so we want to explore and this is essentially how large language models have got so good as well like we want to explore those different timescales get different representations of those multi-scale timescales and see whether that will also greatly improve not just the recognition of words from what was being said, but also potentially understand more of the other audio aspects like intonation or emotion or sentiment, or hopefully, the thing on my bucket list for this is understanding sarcasm.

20:54Yeah, that's interesting. Yeah. And you, on that note, you referred to being the speech component for AGI at some point. Is that kind of the end goal that the Speechmatics system becomes so sensitive to things like sarcasm or emotion that it will plug into whatever AGI system eventually emerges? Yeah, so I think for the last decade, Speechmatics has been really focused in on how do we understand the most from that audio coming in and we'll continue to do so. We've been playing around with large language models to provide sentiment or topic detection or summarization or chapterization and things like this.

21:54But I think one of the most exciting use cases for large language models, as we've seen from the takeoff of ChatGPT or Claude or others is this chatbot type persona, right? And also over the last couple of years, we've seen massive generational leaps in text-to-speech synthesis. We've got to the point where we can now synthesize a lot of different voices, make them much more realistic, not the sort of monotone that we're used to with some of the more historical synthesis voices. And so we got to the point where, you know, you could argue that we passed the Turing test on a keyboard-based system playing with a large language model.

22:40But what if we put an audio in, so ASR combined with an LLM, combined with text-to-speech, you get this audio in, audio out type agent that could provide you this seamless, responsive type interaction. And so here at Speechmatics, we've been building something that we've called Flow, which is this audio in, audio out, conversational AI tool. Now, a lot of people have plugged together ASR and LLM and text-to-speech, but what you find is those systems are often quite stinted. It's often quite slow. If you've ever played with the voice aspect of ChatGPT, it still takes a long time to respond. You're still having that WhatsApp-based structured conversation where it's waiting for one to go and all that side of things.

23:38What you want to get to is much more of that from the movie Her, like that really seamless flowing conversation that you can have with another human. That requires low latency responses. It requires being able to interrupt and requires being able to understand not just text, but, you know, sneezes or coughs or sighs or that type of stuff. And it also requires understanding who is having the conversation. And so Flow uses our best in class ASR. It includes our diarization, which means that it can recognize who it's speaking to. And we built a conversational engine part that enables it to have this very fast, rapid interaction with the large language model and text-to-speech component to enable a truly responsive type of conversational assistant.

24:33So, yeah, we're really excited to be hopefully launching this on Wednesday this week, so in July, and bringing it to some of our early adopters and customers very soon. And the core of, excuse me, the knowledge base and the reasoner is, are you, do you have your own LLM in there? I'm sorry if you just said that, but, or are you making an API call to ChatGPT or Claude or something? Good tech solves problems that you've thought about. Great tech solves problems that you haven't even thought of. What can the commerce platform trusted by millions of merchants do for you? It's time for Shopify, the commerce platform revolutionizing millions of businesses worldwide.

25:28Whether you're a garage entrepreneur or IPO ready, Shopify is the only tool you need to start, run, and grow your business without the struggle. Shopify puts you in control of every sales channel. So whether you're selling satin sheets from Shopify's in-person point of sales system or offering organic olive oil on Shopify's all-in-one e-commerce platform, you're covered. Shopify powers 10 % of all e-commerce in the United States. And Shopify's truly a global force, powering Allbirds, Rothy's, and Brooklyn, and millions of other entrepreneurs of every size across over 170 countries. Plus, Shopify's award-winning help is there to support your success every step of the way.

26:17So sign up for a$1 a month trial period at shopify.com slash IonAI. That's Shopify, S-H-O-P-I-F-Y dot com slash IonAI, that's E-Y-E-O-N-A-I, all lowercase, all run together. Go to Shopify.com slash IonAI to take your business to the next level today. Give them a try. They support us, so let's support them. Yeah, so we've built it to be LLM agnostic. You know, I think what we've seen in the LLM space is rapid development from a lot of different players. And I imagine lots of customers have a preference as to which one they would like to use. And so what we've done is we've brought Llama 3 in into our service so we can self-host it ourselves.

27:15And so that can adhere to any privacy, security, compliance concerns that anybody may have. But you can easily plug that into GPT or Claude or Cohere or Gemini, whichever one you would prefer to use. We don't plan on doing our own LLM. I think there's a lot of resources and other people building bigger and better LLMs. Our domain expertise is really understanding that audio and that speech, and I think that's the opportunity for Speechmatics is that these LLMs are only going to be as good as what they can hear. we want to make sure they can hear really really well yeah uh and i i want to talk more about flow but uh on the research side uh do you uh can you uh handle multiple uh voices simultaneously and parse them.

28:18I had a guy from Google on a year or so ago who was working on bird calls, and they were working on, you know, when you're listening to cacophony of birds in the jungle or deep forest, of, you know, pulling apart that sound and identifying individual streams. And I also am sort of a backyard bird watcher. I have the Cornell Ornithology Lab app. I don't know if you've ever used it. It's really remarkable. And if there are 10 birds, it will list all 10. So it's able to differentiate. So can you do that with human speech? Yeah. I would say there's a spectrum. We definitely are able to understand different voices.

29:20If I got outflow now, it would recognize my voice, it would recognize your voice, it would be able to say who said what. In a busy environment, it would still recognize our voices despite an array of other people around us. Then there's, so you get to the noisy environment and then there's really strong cross talk i don't know maybe in some u.s political situation where there's just lots of people shouting at each other it's hard for a human to recognize uh all of that's going on so that that's still a research project for us to get to in terms of how do you truly pull apart the words that were said by all the different voices at that one time um so we haven't got to that level of true pulling apart, really deep cross talk.

30:09But in a call center, I don't know, with an irate customer where they're talking over the employee that's trying to calm them down, we're able to do that. And so there's a research trajectory on that front. But I would say that we have the best in class diarization for speech recognition. And I would also argue that that is a really critical step in having an assistant that can actually help you in real world scenarios. So if I was walking down the street with talking to ChatGPT and somebody else walked past, it would pick up what they were saying and it would think that that was what I was saying as well.

30:52Whereas with diarization, it knows to listen to only me. You can also then use that to set up things like only respond to me so you know you've got an assistant it's hearing the whole conversation that's going on but it'll only take an action or respond to a question if i say it rather than if anybody else in the room says it actually uh uh you know i wear hearing aids uh and that's one of the challenges of of hearing aids is uh in a in a party or something you just you can't uh differentiate between voices very well. Are there applications like that for hearing augmentation? I think that's a very good application for us to think about for the future.

31:42I know that our founder, Tony Robinson, also wears a hearing aid. And so it's certainly a strong area of focus for him. I think that one of the opportunities here is currently we've built this first incarnation of Flow to be very focused on the mobile device space. I think that's a great place for people to test it, explore it, find what works for them. But I'd be really excited about thinking about putting it into something with multi-directional microphones. So then you can start really getting an extra layer of information out from the diarization, which I'd imagine will work very well in the hearing aid case, where it would listen primarily to people that maybe you're looking at, and maybe tone down some of the stuff that is happening in your periphery.

32:39Yeah. So Flow is being released as an API or as an iPhone or a smartphone app? or how are people going to interact with it? Yeah, so we're very similar to our ASR in that we provide an API such that people can build speech recognition into any product that they're building today. We have a online portal on our website where people can go and play with it and test it themselves, but we don't do a vertically integrated product ourselves. We instead enable others to build that into whatever products they are taking to their customers. And I think that's ultimately what we want to do here is we want to say, we want to enable any product, any technology to be seamlessly integrated with the thing that comes most natural to us, like our voice.

33:39So how do we enable any product to suddenly become keyboard-free, mouse-free, touchpad-free, and suddenly become usable using your voice? I know that a lot of people don't use Siri because it's frustrating sometimes when it doesn't recognize what you say. And so it's kind of surprising that we don't have more products today that are leveraging our voices. We think flow is our inflection point where we get to the point where technology can be used with our voices. We're going to release flow as a conversational AI API so that anybody and everyone can integrate their product and enable their product to be interfaced with through their customers' voices.