The first real-time voice assistant

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to another fully connected episode of the Practical AI podcast. In these fully connected episodes, we try to connect you with various things happening in the AI space and connect you with maybe some learning resources or talk about some subjects that will level up your machine learning game. My name is Daniel Whitenack. I am founder and CEO at Prediction Guard, where we're enabling AI accuracy at scale. And I'm joined as always by Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Doing great today. It's dog days of summer here in the U.S.

1:19It is really hot and humid. Yeah, super humid and nasty. I'm looking forward to AI control, you know, like weather control from AI. And it will keep all of us at just the right temperature. Right. I can't see anything possibly going wrong with that. Of course not. Only positives there. Regardless, that is in the distant, distant future of 2025, I'm sure. Yeah, exactly. Exactly. Let's focus on the next two weeks for now. That's right. Which is important. I think one of the things that caught me off guard this last few weeks, which you and I try to stay plugged in to various things. And, you know, maybe people think and listen to this podcast that we're, you know, keeping plugged in with every single thing happening in the AI space.

2:09But I was a little bit surprised when I saw the release. I guess I just hadn't really been following along with what the company or research lab Qtai was doing. So this is an open research lab that researches AI. and they have funding and some support in terms of infrastructure and all of that. But they're a nonprofit research lab, in my understanding. And they actually, so we talked on a previous show, we kind of got fooled a little bit, or maybe it was a little bit of a fumbling in terms of marketing. But it seemed like when OpenAI GPT-4.0 came out, people were hyped because a lot of the demos were voice based but at least you know at the time of that recording i'm not sure all of what everyone has access to in the the paid and unpaid and enterprise version but the actual voice assistant for open ai was not out and at least as far as the release date of qtai's voice assistant which is called moshi they were the first to actually release a version of their voice assistant which it's similar to, in my understanding, what GPT-4.0 is on the multimodal side in that it is a multimodal model.

3:33So it's a real-time multimodal model that supports a voice assistant. And this research lab, I think it's like eight people or something like that. Of course, they have resources that are supporting them, right? Like this, I think it was a thousand GPUs or something. They have resources, obviously, but they were able to beat, you know, what is now the the Goliath of the AI space, beat them to market with this real time voice assistant, which I think took a lot of people maybe by surprise or maybe some people were following it closer and expecting it. But I think in this sort of six month or whatever time period it was when they were working to get this out and beat the kind of Goliath of what is OpenAI, which I think in and of itself is pretty interesting.

4:26It is. I mean, you know, so many try and some of the other Goliaths, you know, the second tier Goliaths, if you will, are continually trying to compete. And they may touch it. They may fall short. I always love hearing when a smaller group, especially if they're focusing on open solutions, comes out and is able to do well. And they got a cool name, by the way. Yeah, yeah. And it is interesting because this does run. So when you see the demo, and we can pull it up here in a second and maybe ask a few questions. But when you see the demo or the prototype, it obviously still has some rough edges.

5:04So I think you have some rough edges that aren't fully kind of productized version, like maybe what you get with the OpenAI voice assistant in the forms that it's in. But it is very impressive also because this is a model that I believe it's models that are of a size that you or I could run them on even a single GPU. And they're going to open source these models. I don't know what the time frame is on that, what exactly that will look like, what licensing, all of those things. They do have a few talks online. So if any of the listeners know that information and I just haven't run across it, then they can maybe update us.

5:46But yeah, they will be open sourcing this, which I think will drive a lot more experimentation. And of course, as we saw with the first open LLMs that were released with Llama and other things. There was, of course, a huge explosion of innovation and experimentation going along with the release of the open versions of those things. And so I expect that there'll be a similar thing with these models and what I assume will be other versions or other families of these types of models moving forward. Yeah, I noticed, going back to your point about being able to run it locally and potentially on a single GPU.

6:26They talk about in their press release, they just say compact. Moshi can also be installed locally and therefore run safely on an unconnected device. To extend that a little bit, I think that there are a lot of larger organizations that are worried about IP concerns. These are topics that we've covered quite a bit on the show in days past. So Moshi may very well find a home in corporate environments, first of all, where they don't want to send information out and they want to get the advantage of that because it can probably be run on a single GPU. A lot of edge devices make it possible. So great thinking there in terms of what's possible.

7:06And then finally, thinking of my own industry in the defense space, since it can be run in an unconnected or disconnected environment, there's all sorts of things from a government standpoint that they may be willing to do. So it's a great strategy. I love hearing these small companies that might be able to have a big impact in industry by accommodating those concerns. Well, Chris, I find one piece of this whole Qtai Moshi thing very interesting, which is almost like it feels a little bit like deja vu because we back in whenever it was, I forget what year OpenAI came about. is like there's these big players in the AI space and they were doing certain pre-trained models and all of this stuff and robotic things and all of that.

8:04And then OpenAI came along and said, oh, we need a open, transparent, nonprofit-driven research lab to really promote innovation going forward. And of course, as we have moved forward through that, we've seen open AI kind of get away from that sort of pure nonprofit status with a little bit more of a complicated corporate structure, right? Which we've talked about on different shows. But then also, you know, just their release of their work and their research and their models and their data and those sorts of things, of course, has become very not open. And they, of course, have their own reasoning behind that, which at least publicly they would say is related to...

8:53Microsoft. Sort of, yeah, well, at least publicly they would say is related to safety of the use of these models. Of course, there's various people that might guess certain other motivations. Microsoft. Yeah. But yeah, I do find this whole thing sort of like deja vu. I don't know if you're having the same feeling here. You and I both have a long history in the more than six years now that we've been doing the podcast of supporting open engagement from different organizations, whether they're corporate entities or nonprofits or whatever. And we've seen that from others. I mean, famously, Jan LeCun talks about he works for Meta, you know, which is Facebook's parent and talks about nonetheless having open models and all that.

9:45And so we tend to shine a spotlight on those organizations that do that wherever possible. we certainly went through that because we've been doing about just after we started the podcast which was back in 2019 or 2018 actually 2018 and about a year later open ai closed up so we actually covered that in the early shows you know it is what it is they've done that they remain an amazing corporate leader in the space but yeah they did close all up and and we tend to turn more spotlights toward others like this. So I'm pretty excited to see what Qtai is doing and is able to do going forward here. And I hope they're able to viably play against that top tier competition.

10:33I think that would be wonderful to have multiple. Yeah. Do you think that there's any chance for this sort of like open research in the AI space or in the technology space to survive as a sort of bulwark of open, transparent research and open source within the pressures that come, of course, when you release this sort of technology and you're a leader in the space and there are actual dollar signs and corporate concerns and certainly partnerships that are necessary. So, you know, partnering with companies to do this work is almost a reality, I think, in the space, because we talked about this a little bit with the Stanford AI index, where they found that, you know, the bulk of AI research is still happening from the industry side.

11:29So I don't know, what are your thoughts? Do they stand a chance at staying the course with this? I think there's certainly a chance at it. And I would argue, it's the same argument I've made in previous shows where we talked on similar topics, is that we're seeing is the AI industry has been maturing these years at an incredibly rapid pace. But we're still seeing many of the things occurring that we saw when the software world was really maturing over several decades. And the place where open source has really, really worked are in common touch points where all organizations or many organizations need a common thing.

12:12And they might build something differentiated on top of that for their revenue to drive profitability. but there's so much that is underneath that point of differentiation that they and many other organizations can get the benefit out of a lot of effort, a lot of work. A lot of times they'll have paid employees do it. So there's a point where working together and doing open stuff makes sense for business and it drives profitability. It may not be your single point of differentiation, but if it's anything under that, why not? Why not share the costs and pull expertise for the best possible foundations.

12:49And so what I'm hoping is that we continue to see that play out in the AI space. We're seeing, you know, if you look at Hugging Face, we've already talked about the fact that a couple of months ago, they announced that they were hosting a million models. Those are all open source. Really, really impressive. And so I think that there is a good chance that a vibrant, open community around AI can and will continue, and it will have a lot of corporate players involved in it. So I'm very optimistic in that way.

13:29Hey friends, this episode of Practical AI is brought to you by our new friends over at Plum. Plum is a low-code AI pipeline builder that helps you to build complex AI pipelines super fast. You can easily create AI pipelines using their node-based editor, iterate and deploy faster and more reliably than coding by hand without sacrificing control. Deployment is easy. Pipelines are live API endpoints. They eliminate the need for constant code redeployment and debugging by deploying complex AI pipelines as API endpoints. Team collaboration is easy too. Plum's declarative node-based editor enables you to build quickly while empowering non-technical roles to iterate on what you've done without breaking it.

14:16You can build advanced AI features, get structured output every time, transform data and leverage validated JSON schema to create reliable, high-quality structured output. So Plum is built for builders. Early stage product teams are using Plum to go from idea to validation in record time. To get started, go to useplumb.com. That's Plumb with a B, as in plumber, to request access today. That's U-S-E-P-L-U-M-B.com. Again, useplumb.com.

15:10All righty. So as we, uh, as we change gears just a little bit, I had noticed a couple of interesting things. So I, I spend a lot of time talking to different folks in the kind of in the fortune 500 fortune 100 world. You know, I, I work at a big company, but I have a lot of friends and former colleagues at other companies and we chit chat about these things. So something has really come up in a whole bunch of conversations lately for me. And I thought, wow, if I'm talking about it this much with different friends of mine, it probably is a good topic to talk about on the show. And that's an interesting observation.

15:48And that is, for those of you who are familiar with the organization Gartner, and that organization does a lot of prediction and kind of identifying different technologies and things where businesses can use them effectively. And famously, they put out the Gartner hype cycle. And what that is, is it is a life cycle for technologies. And they basically, across all technologies that they track, which is many, they put them on this hype cycle and track where they are in their life cycle. And the short version of what that is, it has a steep upward curve that looks like an ocean wave sort of that plunges down into a trough behind it.

16:35And then it kind of comes up without so much steepness midway to kind of a sustainable plateau. And so what they would argue is that for any given technology, there is an innovation trigger, which is this rocketing up on amount of hype associated with the technology, and that it gets to a peak, which they refer to as the peak of inflated expectations, where it's really high. Everyone's talking about it, but maybe not a lot of productive work has happened yet. Super cool. You can probably already recognize how AI might fit into this, with all the things we've talked about over time. But then those expectations have not been met, and people become frustrated with the technology, and it plunges down into what they call the trough of disillusionment.

17:25And that's where they kind of go, wow, I thought that thing was so great, but boy, it really didn't pan out. And we wasted a lot of money on it, and it's just not really worked out well for us. But then calmer minds come along and they say, well, wait a minute. This technology has some really good uses. We just need to be a little bit more practical, pragmatic about it and not lose our heads over the hype. And that's called the slope of enlightenment. And that reaches a point that's called the plateau of productivity. where basically for the long term, a technology lives out the rest of its life cycle, being a productive technology, but without all the craziness in the early hype days.

18:08So now that I've introduced everyone to that life cycle, going back to the conversation that I've been having repeatedly with multiple people, that I had noticed that so many organizations, especially large organizations, are just plowing money into generative AI with mixed results. Some are getting some decent results within the context of it being early days in the corporate sense. But I noticed that after peaking and holding a peak on the hype cycle for quite a long time, generative AI is now beginning to plunge down into the trough of disillusionment. And what that would imply, according to Gardner, is that people are beginning to get a bit frustrated.

18:53And I would say that's panning out because I've noticed many articles and social media posts over the last few weeks that people have been kind of going, this isn't going to lead to generative AI. This isn't quite as good as we thought. It's not magic. It's all the things that you see with people being a bit frustrated with it. And those are increasing in the number that I've seen. So it got us talking about what does that mean in a corporate sense, especially when you have a technology plunging down into the trough of disillusionment. And not only that, but it's a technology that has received a lion's share of funding relative to other technologies that go through the hype cycle.

19:37It's the coolest of the AI, you know, over the last couple of years, the coolest of the AI tools in the toolbox. and with corporations always lagging, they're now plowing money into it and yet expectations are falling. And so not getting to the point of it, we'll obviously find that slope of enlightenment and that plateau of productivity eventually. What does it mean over the next few months as we're looking at organizations that are still plowing money into generative AI, but maybe not in the most productive sense or not as productive as they could given the dollar value that they're putting in.

20:15So I've asked a lot of people what they think of this. Daniel, what are your impressions of that? It's an interesting place to be if you're in corporate America or corporate anywhere these days. I do think it's interesting. And I think that in some ways, some of these feelings are healthy. In particular, what I mean is I noticed earlier on so maybe in 2023 or you know last fall still talking to a lot of people with a misconception that oh we have somehow what's going to happen is we're going to get access to a large language model or we're going to get access to a foundation model in our company and somehow that kind of equates to a solution to them.

21:04Like this will now be a thing that solves problems. And I think that, of course, is a bunch of baloney because basically a model does nothing. It's how you implement it, how you integrate it, how you use it that actually makes it a solution. And so I don't know how else to describe that other than people thinking that AI would provide a different type of solution than other technologies, which are softwares that people deploy within their companies, right? And so some of this, I think, is really healthy in that people are realizing, oh, wait a minute, there's still a need to think about how we integrate a call to a large language model in the context of a larger engineering project.

21:53And actually, there is engineering around the edges of the integration of AI in some ways different than traditional software engineering and in a lot of ways the same. Whether that be hosting services or testing and evaluating outputs or versioning the way that we call these models or other things. There's a lot of those best practices that are still really valid from the software world. And so to me, it's not so much, and maybe this is just because I, of course, have a vested interest in the technology because I'm building with it every day. But I think it's not so much a disillusionment about AI functionality in the context of what people are building over the next year, but disillusionment around how that integration happens.

22:50Whereas before it was sort of this fuzzy thing that we're going to bring AI in and somehow that's going to solve a bunch of problems without really an understanding of how you would actually see return around that. Now people are saying, well, yes, we're going to bring in AI, we're going to bring in LLMs, but that's going to live still in a software stack that we have engineers developing. And we're going to develop that on some lifecycle. And yeah, there's still going to be, if anything, maybe increased engineering spend because there needs to be extra engineering around these models. And so it is enabling efficiencies.

23:27It is enabling net new kind of features or net new products. But these are still products driven by software that requires engineering. And so that realization, I think, is a really healthy one. And so maybe that thing that has the disillusionment wasn't really ever a real thing that could have been gained, I guess. I think that's a fantastic insight. I think in a perfect world, if we can help people along, kind of get through their own trough of disillusionment very quickly to climb back up onto the slope of enlightenment by following that guidance is essentially what I'm getting at. Before diving into it, I know that over time, as I've talked to people, it reminded me of amplified beyond what I've heard before, but of previous technologies that were supposed to solve everything.

24:18Blockchain was going to solve the world, if you recall. Blockchain was amazing. We were going to have it everywhere. It was going to be everything. And by having since reached that plateau of productivity at the end of the life cycle, blockchain has a fantastic place in the technology world and a vibrant community. But it, of course, doesn't solve all things. And I think people need to realize that the same with these kinds of models is that they can do that. So I I know that one of the things that I'm trying to get people to do is to get through their own trough of illusion, but quickly and start recognizing in a really productive sense, how to fit it in with larger systems.

24:55We've always talked about, it's really the software system around these models that makes it all work and that makes the value for the user. And even extending that, if you're not in the cloud, it's the hardware. If you're out on the edge, it's all about what do you have on the hardware and how does it integrate and how does it integrate with the systems you already have in place? And what special value are you expecting generative AI to bring to bear that you haven't already been trying to design and solve for? And so I think as people really stopped and they kind of got out of their New Year's Eve party moment and they said, OK, I'm an engineer.

25:31I need to start being an engineer again and thinking about it. And they thought, well, maybe it doesn't solve everything like I thought, but I can identify some pretty cool things that it would help on value. And I'm hoping that people will start focusing on that and bring engineering, to your point, back to bear on this and solving it, but solving it in that larger ecosystem that includes the overall stack that you're in, the software. And since we're moving ever more out onto the edge, into all the devices that we use out there beyond just our cell phones that were always ever present, that we can find some good uses.

26:07So maybe this is a chance for a bit of a resurgence, yes, of engineering, but also this triggers all sorts of data science-y things in my mind. Because as a data scientist operating in that industry for however long it was the thing, it was about choosing the right sets of data tools and models to come up with a solution. Or at least that's how I think a lot of people viewed it. And that may have been a gradient boosting machine plus a SQL database plus some sort of data pipeline. and connecting that into infrastructure and then eventually into products that get out into the world. Now, some people might view data science differently and have different views because it's sort of an ambiguous term in and of itself.

27:01But I see one interesting thing on the hype cycle that you were mentioning. There's a shorter time period that they talk about this like composite AI reaching the plateau than quote generative AI, which is interesting to me in that I actually had to look up this term because I have no idea what that term means. There's actually a number of terms on the Gartner height cycle that I have no idea what they mean, which I wonder where they come from. And I'm right there with you. So neither one of us knows what composite AI is. So I'm sure that there are a few people out there that are very familiar with it and are snickering at us.

27:34And we welcome your education and feedback on such. Keep going though. Yeah, but I looked up the term and this appears to just be like almost a term describing data science, which is just like using different types of AI or machine learning together to solve issues or create solutions, which is sort of just descriptive of data science and kind of what it was for many years. So I don't know that it'll be called data science. Maybe it's called AI engineering. I don't know. But I do think that we'll see kind of a return to this idea of composite solutions and And a multifaceted way of looking at doing these things, not just with Gen AI, but that plugged in as an option into the solution mix.

28:19I couldn't agree more. Despite being an AI podcast, I know you and I are always a little bit eye-rolly when it comes to all the hype around it. We try for our listeners to cut through the hype and talk about it. So, yeah, a return to engineering and taking advantage of some of these capabilities in a holistic system. that is highly productive and gives your end users what they need is the way to the future.

29:09OutShift blends startup agility with corporate strength to develop next-gen technologies from the ground up in AI, quantum technologies, cloud-native, and more. Their newest AI innovation, Motific, addresses a critical challenge in the rapidly advancing world of Gen AI. Bridging the gap between concept and deployment, this model and vendor-agnostic solution supports the entire Gen AI journey. from assessment and experimentation. Motific accelerates deployment from months to days while safeguarding against Gen AI security, trust, compliance, and cost risks, all while empowering business function and IT teams to rapidly configure end-user assistance powered by organizational data.

29:56Motific provides advanced, customizable policy controls to prevent unauthorized access to sensitive data and helps ensure compliance throughout the entire process. With deep visibility into operational and business metrics, Motific enables you to track ROI, optimize costs, and make informed decisions. By offering a centralized view, Motific deters shadow AI usage and empowers teams to innovate responsibly. So move beyond the traditional constraints of AI implementation, utilizing AI deployment that is both responsible and is revolutionary, ensuring your projects are not just quickly launched, but built on a foundation of trust and efficiency.