#162 Vasi Philomin: Behind the Scenes of Amazon's AI Breakthroughs

0:00ML has been foundational for Amazon for over 25 years. They take the technology and they put it to use in real-world business use cases at scale. The reason why Gen.ai is being talked about so much is because of its potential to automate work. It is going to help humans do things at scale that they've never been able to do before. So Generative.ai essentially is bringing in a whole new set of people that can now start benefiting from it. Hi, I wanted to jump in and give a shout out to our sponsor, NetSuite by Oracle. I'm a journalist, and getting a single source of truth is nearly impossible. If you're a business owner, having a single source of truth is critical to running your operations.

0:44If this is you, you should know these three numbers. 36 ,000, 25, 1. 36 ,000 because that's the number of businesses that have upgraded to NetSuite by Oracle. NetSuite is the number one cloud financial system, streamlining accounting, financial management, inventory, HR, and more. 25, because NetSuite turns 25 this year, that's 25 years of helping businesses do more with less, close their books in days, not weeks, and drive down costs. One, because your business is one of a kind. So you get a customized solution for all of your KPIs in one efficient system with one source of truth. Manage risk, get reliable forecasts, and improve margins.

1:38Everything you need, all in one place. As I said, I'm not the most organized person in the world, and there's real power to having all of the information in one place to make better decisions. This is an unprecedented offer by NetSuite to make that possible. Right now, download NetSuite's popular KPI checklist designed to give you consistently excellent performance, absolutely free at netsuite.com slash IonAI. That's IonAI, E-Y-E-O-N-A-I, all run together. go to netsuite.com slash IonAI to get your own KPI checklist. Again, that's netsuite.com slash IonAI, E-Y-E-O-N-A-I. They support us, so let's support them.

2:42Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I talk to Vazi Philemon, Vice President of Generative AI at Amazon Web Services. Vazi shares his journey and the pivotal role his team has played in launching Amazon's array of AI capabilities. As the mastermind behind services like Lex, Poly and Code Whisperer, and now Amazon Q, Vazi provides a peek into the transformative potential of generative AI across industries and how Amazon's unique approach to AI integration and customer-centric innovation is shaping the future of enterprise AI solutions. I hope you find the conversation as fascinating as I did.

3:36So my background is actually AI ML. So back in the 90s when I did my PhD in computer science, I chose to specialize in AI. It was not a common thing to do back then. Not at all. Yeah, most people would talk about databases or networking and operating systems. Those were the hot topics then. And this was sort of an obscure topic. So all of my friends and family that thought I was nuts back then, they think I'm a genius now because I picked the right topic. So it is my background. And I joined Amazon about six and a half years ago. And if we have to talk about my earlier role outside of Amazon, that's going to take a while.

4:13So I'm going to pass on that. Yeah, from Amazon. Right. Yeah. So I joined Amazon exactly when they were setting up a team to focus on AI ML as a business in AWS, on the AWS side of the company. And so my team's the one that has launched majority of the AI services over the last six years. So all of the language services and I've managed language services like Lex and Polly and Transcribe and Comprehend and Kendra and then we have vision services like recognition and textract and we've got some industrial services like Monitron, Amazon Monitron and things like that. And I was my team was the one that launched all of this so I've had a very busy six and a half years every year at reInvent, launching a whole bunch of new capabilities in AI for our customers.

5:03And that's led to widespread adoption of our services. And I'm sure you've heard, we've got more than 100 ,000 customers that use AI ML on AWS. And so I've been part of the journey the whole time, having launched all these services. And I think this was at Remars last year, if my memory serves me right, when we had actually envisioned CodeWhisperer by that time, we knew that generative AI was there. And so the first killer application is really helping developers be more productive. And so we ended up coming up with the concept of CodeWhisperer and then we launched it at Remars. And that's kind of how I fell into generative AI.

5:46And very quickly we realized the potential of generative AI. We think at Amazon, we think it's going to simply transform pretty much every single function in every single business in every single industry. And if you probably saw the keynotes, Adam's keynote yesterday and Swami's keynote today, it was all full of generative AI. So we're investing heavily in that space. And it made a lot of sense for me to just focus on that as an emerging topic. There's a lot of ambiguity. There's a lot of complexity. And there's a lot of potential and challenges. So the company wanted me to focus and work on just generative AI, given the broad scope of it.

6:29And so I had to leave everything, all the other stuff behind and focus now fully on generative AI. And it's been a great last year. We've been on a very accelerated pace. And we're not even waiting for reInvent to announce anything anymore. It's like immediately. We've got something that we think is good enough and it's going to help customers and we put it out immediately. So we didn't even wait for the GA of Bedrock. Normally we would have waited for the GA of Bedrock for Readman, but we didn't and we launched it like a couple of months ago. We put it in GA. Yeah. I'm curious about Amazon's development because I'm a journalist, not a practitioner.

7:09And I started paying attention after writing about Jeff Hinton in 2017, so much later than you, obviously. But at that time and for the first few years as I was getting to know the space, you know, OpenAI obviously was a player. Google was a player. Microsoft to a certain extent was a player. Amazon, it seemed to me all kind of behind the scenes. You didn't talk a lot about your AI work. It was in the products, but not as publicly facing services. So for Gen AI, when did you build the first foundation model at Amazon? Let me actually answer the question. There were a lot of things that you said there when you asked me the question.

8:16I think one of the things most people may not know is that ML has been foundational for Amazon for over 25 years. And I think what Amazon does best with any technology, especially AI ML, is they take the technology and they put it to use in real world business use cases at scale. And I've chosen these words really, really carefully because that's what's lost in all the hype. And we do it quietly because in the end, what matters most is nobody's going to look at pieces of the puzzle. They want the entire thing. They want to be able to do something differently that they've been doing before. And to enable that, you have to think end to end.

9:03And I think that's what we do really, really well. And that's what attracted me to the company in the first place. And that's probably the reason why I probably worked there for a very long time. It's because we think end to end. We think real business use cases, which means, you know, it can't just be a toy. It can't be an experiment. It can't be just for show. It needs to be adopted at scale. And so the cost has to be lower than the value for this to be adopted at scale. There's all kinds of things you have to think about. And I think you have to think at different levels of the stack. That's why we keep pointing out the different levels of stack we have for AI ML.

9:37So that's important to point out that that's what Amazon does really well. And we have lots of examples of it. So, for example, back in the 90s, if you bought a book on Amazon, people who bought these books also bought those books. And that is a classic ML technique called collaborative filtering applied at scale. Back in the 90s when I was still doing my PhD. And then fast forward now, you look at Alexa. That's like 100 million users and a billion interactions every week. That's ML applied at scale. And if you look at our fulfillment centers, if you've ever been to one of the fulfillment centers, you will see that we ship 1.6 million packages a day.

10:14and that's humans and robots working together, doing the same exact tasks. I've got videos of it, but best is to go and see it live. They're doing the same tasks in the same space at scale. So we've got a history of taking technologies like AI ML and applying it at scale to real world problems, business problems. That's exactly what we've done with AI ML and now that's what we're planning to do with generative AI. The majority of the cloud workloads happen on AWS, more than 100 ,000 customers using our services. Nobody's even close. And I remember the same kind of thing being said when we started six and a half years ago, when we started exposing the AI ML capabilities that we've used internally for a long time for external customers to use for their own companies.

11:01People said the same thing. You know, I know about Google. I know about some of your competitors. But then you just fast forward. We just execute. We work backwards from customers, make sure that it's going to fit in to what actually they want to do at scale. And that's why most customers gravitate towards AWS. And we're doing the same thing with generative AI. To come back to your question, when did you exactly start? A lot of the things that we've been doing have been tried out at various parts of Amazon already, many businesses. So there are sets of capabilities that we expose to customers through AWS some of those could be from another Amazon business that has actually applied it at scale.

11:48So it could be things like that. Other cases, we work backwards from our own customers, AWS customers. We look at their problems, and then we may have things happening already within the company, or we may choose to do something new. And so Code Whisperer, I think we started working on it already like two years ago, two and a half years ago, before all the hype around generative AI. because we saw it was a natural evolution of what was happening in AI. And if you're familiar with it, as the models got bigger and more sophisticated and they were exposed to like web-scale data, the models started having some emergent properties.

12:26And these were properties sort of like latent in them and you can unlock it very easily without a whole bunch of scientists. In the past, we would build specific models for specific things and you would start from scratch all over again and again. And you would collect the data, you would build the model, and you would make sure it worked for that specific niche use case, and then you would put it out. And then you would go and do the same thing for the next use case. And some of the use cases may have multiple models involved. Now you've got these more sophisticated models and more capable models that you could simply tweak it to kind of do the thing you wanted to do, the task that you wanted to be good at, by just giving it examples of inputs and outputs.

13:07And this is something that anyone can do. You don't need to be a scientist to do that. And so these are the kinds of capabilities now that... So generative AI essentially is bringing in a whole new set of people that can now start benefiting from it. You don't have to be an ML expert. You don't have to be a scientist to do any ML at all. And you can get these models to do what you need to do. And so Bedrock, we created this new service called Bedrock because of that reason. And if you look at it, We also don't think it's going to be a single model to rule them all. You probably heard this phrase.

13:39Some other companies want you to think that. They want you to think that it's just the model that matters, but it doesn't. There's a whole bunch of other things that you have to care about. And especially in such an early phase of a new technological trend, you just can't bet on any one thing. And I don't think that any one thing is ever, like even for a certain use case, we end up using multiple models behind the scenes because you have to take into account other things like cost and accuracy and latency and it's always a trade-off and I think in the keynote today the Intuit person the person from Intuit mentioned that they've been they've been always cutting edge leading adopters of all of our services but also all of the technology out there and she was talking about how there are lots of choices you have to make and you can't just think that the one model is going to help you solve all of those.

14:32So what we've done with Bedrock, for example, is we made sure that our customers have different choices. And there are different families of foundation models. So some foundation models are like text in, text out. Other ones are like text in embeddings out, which are useful more for search. Then there's a third group, text in, image out. And we announced some new Amazon built models there. But the main point I want to make is we think that it's not just going to be one model provider and it's not going to be one model. And that's the reason why not only do we have, we've assembled the best startups out there and the best models out there as part of Bedrock.

15:12We've gotten our partners to come and offer their models on Bedrock, but we're also building our own model. We're not saying, you know, it's just third-party models that'll be available on Bedrock. And I think ultimately all this just benefits customers because they have the choice and no one knows how things are gonna go and you've got you've got all of these things available in one place and so it's very easy for you to figure out what works for you for your particular business cases and it's always been about choice yeah so how many well well actually the just quickly how many models or model families of models does Amazon have?

15:52You don't have a marquee model. I don't know if there's anything like a marquee model, right? And it sort of helps to look at this whole thing holistically, right? And it also doesn't matter if we have them, if there is one, it doesn't matter if we have it or if, you know, one of our other partners have it. It doesn't matter. It only matters what customers have access to. Right. Right. And so let me just talk through the different model providers we have on Bedrock right now we've assembled. We've got AI21. There was an early startup that created some of the first models. And then we've got Anthropic, which is very, very popular.

16:37Right. And they just announced Cloud 2.1, which has an industry-leading 200K context length. And what that means is that you can feed it the largest books you can think of in just that one call, in that one context. And then you can ask it all kinds of things about that book. And you can even make it write another book. So it's got that. I think they've reduced hallucinations dramatically compared to their previous version. And so we've got Claude from Anthropic. Then we've got Llama 2's extremely popular open access model from Meta. And so there were a lot of customers asking us for that on Bedrock.

17:16And so we did that. We offered their 13 billion parameter model, chat model, a while ago. And I think we announced today the 70 billion chat model from Lama, which is very, very popular. We have that. We've got Cohere, which is another interesting startup. And they've got their own text in, text out models, but also an embeddings model. And then we have Stability, which is one of the hardest startups when it comes to images and videos and things like that. So you've got Stability's SDXL 1.0, which is their latest model, which generates high-quality images. We have that update as well as part of this re-invent.

17:55And then finally, the Titan family of models, where if you noticed, some of these model providers, they probably stick to one family, like text-in, text-out. Anthropics models, for example, are text-in, text-out models right now. And Stability's model on Bedrock today, they are text and image out. With Titan, we're taking a holistic view of the whole landscape here. You've got different families of foundation models, right? Text in, text out is just one. But it's the one that is most popular and most visible because of what ChatGPT did, right? I think consumers could relate to it. And so it became very, very popular very quickly.

18:37But that's just one family of models. You can't do much without the text in embeddings out models. Because if you want to do any sort of retrieval and searching and finding information, you're going to need those. And it's usually behind the scenes, so nobody talks about it much. But it's a super important component of the whole thing. And so we've got Titan text and we've got different versions of Titan text. The smaller ones, we've noticed that you can take a smaller one and you can fine tune it. We didn't talk about customization yet. That's a key component of our offering. But you can actually fine tune it and you can get it to perform on that one task.

19:14You can get it to perform close to the best models out there. If there is a marquee model, you can make it actually perform as good as that. But now you're doing it with a much smaller model. So your latency, your costs are going to be dramatically different from if you would use this hypothetical marquee model that may be out there. And so we've got that family. right and then we've got our Titan also has text embeddings model and today we announced the Titan multimodal embeddings model this is where the text embeddings model allows you to search documents but the multimodal text embeddings allows you to do image combined with text so if you have a product catalog you can either search with an image or you could search with text or you could search with both and for example I could just say here's here's a picture of my living room, recommend furniture that fits with it.

20:08So it's a combination of you're saying something, but it's linked to the image that you may be uploading as well. And so that's kind of when you would need this multimodal embeddings model. So Titan just announced that in GA. And then you've got the text to image, which is where it's useful in creative situations where, let's say you're a marketing company or an advertising company, you've got a new product that you want to create a marketing campaign on Instagram for, then you may want to create images that are reflective of what your message should be about the product you're trying to sell.

20:42But again, you'll use that in combination with some of the other models. So I think that pretty much every workflow will require access to different families and even within a family there'll be differences in the way the models were trained. For example, the text to image, the Titan image generator model, which was announced in Swami's keynote today, we focused a lot on those ad creation, marketing campaign kinds of use cases. So we put a lot of effort into doing very high quality images of faces and things like product placement, allowing customers to isolate products and put it in different backgrounds.

21:20So we focused on specific workflows. And I'm sure another provider that's looked at the same problem would have focused on other use cases. So this actually speaks more to what we are doing, which is offering all these choices so that customers can figure out what works for their particular use case. Yeah. I wanted to ask, presumably you're working on video. I just saw Andrew Karpathy on Twitter highlighting just an incredible piece of software by a company, I'm looking for it now, called Pika, Pika 1.0. Are you guys doing something like that? Yeah, so we're constantly working on, you know, as per our strategy and as per what are customers looking for, 90 % of what we do is really working backwards from customers.

22:15Right. And the reason why we built things like Titan Image Generator is because we heard from some of our customers that are in that space saying, you know, the current solutions are okay, but, you know, they've got some issues. And so that led us to actually go back and build something specific because we saw the potential and we saw how it could change, help change their innovations in their business. And so we're constantly working on new technology. I don't have anything specific to share today on video, but you can assume that we're working on a lot of different topics in this area. And I think we'll put things out when we feel like it addresses a business problem.

22:55and we'll put it out also when we think it is something that can be adopted at scale and not just something that can be demonstrated in a prototype or a proof of concept. Yeah. Yeah. The other thing I wanted to ask is there's sort of this direction to building agents off of large models. Yes. Where are you guys with that? And to me, I mean, all of these models are amazing. Yes. but when you get to agency, that's when the world will change. Absolutely, absolutely. And so many customers would think that Bedrock is just a hub of different models, but that's just the beginning, right? And what we've done, I think, in the last several months is we've thought about workflows around these models.

23:46So I've talked to a lot of customers, and one of the questions that a lot of customers have for us is, you guys are making these capabilities available also to others and to my competitors who are part of the others and that means that you know I need some way of differentiating myself from all those others because they have access to the same capabilities this is why when we started with Bedrock we didn't just say okay it's going to assemble the best models in the world we said it's going to do that and then it's going to allow customers to make these models their own with their own data and their own IP in a super secure and private way, right?

24:23And that's crucial for enterprise adoption. It's so different from the consumer side of things. And I think it was nailed. I think Adam talked about it a lot in his keynote, security and privacy and how you want to give customers the option to do this on their own in a very private way that even AWS see what the data is, right? And that's the kind of controls we've always given our customers. And we're putting it to use here so that they can actually make these models their own in a super private way. So there's workflows around customization, and there are many different kinds of customization.

24:58But then there's also workflows around putting, actually automating work, which is exactly what Agents is all about. And the reason why Gen.AI is being talked about so much is because of its potential to automate work. And for me, automating work scares people sometimes. But for me, it's more like it's transformative productivity. That's what it is. It is going to help humans do things at scale that they've never been able to do before. So I think that is how I think about it. And so agents, here's a quick overview of what we announced agents in GA right now. And here's what agents allows you to do.

25:38If you've probably, you know, taken ChatGPT, if you've used ChatGPT and you asked it, like, who the Oscar 2023 winners were. You know, I'm sure you've seen ChatGPT tell you, my knowledge ends, there's a cutoff date for the knowledge. It says my knowledge ends December 2022, and I can't tell you who the winners are. And so what we believe is that knowledge, especially knowledge that is changing fast, should stay outside the model. It shouldn't be part of the model. And you don't want to do fine-tuning just because you want to give models knowledge. You want to do fine-tuning because you want models to perform tasks, right?

26:16And so agents solves two problems. It solves that whole workflow of keeping the knowledge separate. You create an agent, which is powered by one of the foundation models on Bedrock. And then you have to go through this workflow of creating your data sources, your knowledge bases, where all of your knowledge resides. And then you've got to point the agent to it. And now suddenly you've got an agent that is very good at all of your knowledge. It knows about your knowledge and it's going to be very useful when you actually try to get it to do stuff, right? And so that's the first, that whole, it's the RAG workflow, the retrieval augmented generation workflow.

26:56With knowledge bases for Bedrock simplifies it, the knowledge base part of it. And then now you create an agent and you point it to a knowledge base. Now you've got this digital worker is the word that I use for an agent to describe the agent that is very, very fluent with all of your business knowledge. Now, the next step is you want to automate, you want to break down complicated tasks and actually make it happen. You want the agent to actually do things, not just be able to answer questions. That only gets you so far. You actually want to be able to do real work. And that's what I mean by automating work.

27:33And so a good example I want to give you is, let's say that I've just been shopping a lot for, you know, because of Black Friday and Cyber Monday, right? And so you often, you buy, let's say I buy a shoe and I buy a shoe of a certain color and when it gets delivered to me, I don't like the color. And so now I want to send it back, but I want to replace it with a shoe of another color from the same brand. So I'm triggering, essentially as a consumer, I'm triggering an order replacement workflow. And now you can, behind the scenes, what you can do is create an agent. And obviously, you've created knowledge bases around all of your products and information that you may have.

28:14And the agent knows those things. But also, you can teach the agent APIs, the business APIs you may have in the back end for your own business. So, for example, there may be an API to say, does this brand even carry this shoe of this color? And there may be another API that spots the price difference, if there is any. And then there may be a third API that actually triggers the order replacement workflow. So it's going to give you a label to return your previous thing. And then it's going to set up the order for the new, to send the new shoe, the new color to you. So you can teach the agent all of these APIs in natural language.