#252 Jeffrey Hammond: How to Build Scalable GenAI Products (AWS Strategy)

0:00Executive coaching is something that has traditionally been done person to person. It's very high touch. as a result, it usually only happens at the executive level. There's a very, very small number of people that organizations are willing to support. There's all these kind of low value added activities that even though I like our expense reporting system, it's actually a lot better than anything else that I've used in the past, there's still opportunities for improvement. Net-net they were able to get to a point where they think they can get a 70 % improvement in their product team execution.

0:38That's pretty significant. There's a growing expense eating into your company's profits. It's your cloud computing bill. You may have gotten a deal to start, but now the spend is sky high and increasing every year. What if you could cut your cloud bill in half and improve performance at the same time? Well, if you act by May 31st, Oracle Cloud Infrastructure can help you do just that. OCI is the next-generation cloud designed for every workload, where you can run any application, including any AI projects, faster and more securely for less. In fact, Oracle has a special promotion where you can cut your cloud bill in half when you switch to OCI.

1:25The savings are real. On average, OCI costs 50 % less for compute, 70 % less for storage, and 80 % less for networking. Join Skydance Animation and today's innovative AI tech companies who upgraded to OCI and saved. Offer only for new U.S. customers with a minimum financial commitment. See if you qualify for half off at oracle.com slash IonAI. That's oracle.com slash IonAI. E-Y-E-O-N-A-I. All run together. oracle.com slash IonAI. So go ahead and introduce yourself, Jeffrey. Okay. Thanks, Craig. So I'm Jeffrey Hammond, and I am a global ISV product strategist at AWS. I work primarily with software company customers, and I've had a long career in the software development space.

2:28I started out as a developer for a number of years and then spent more than a decade building software development products, primarily in product management, product marketing roles, from small startups to large, high growth, and then acquisition, in my case, by IBM. After that, spent a number of different years working with a lot of different software companies as they built their products and worked through multiple technology disruptions. Mobile technology is an example, cloud technology. And now here we are again with disruption in the AI space. And so that's really what I do as a product strategist.

3:09I work with our software company customers to help them understand the opportunities that they have and then how Amazon can help them take advantage of those opportunities and accelerate the way that they seize them to drive profitable growth. Right. Yeah. I mean, you work with software vendors, not with people who are integrating AWS or building on AWS. else? Yeah, yeah. Amazon has five different segments that we organize customers by. There's enterprise, makes sense, small, medium business, startup, digital native businesses, and then the fifth one is ISV. And the reason that we do that is because, as you're probably well aware, ISV software companies have particular challenges when it comes to building products.

3:59They live and die by the success of the software that they build. And because they are serving their own customers, those customers often have specific requirements on them that create demands on the products that they build. So with respect to AI, you're not just building a service that you're deploying internally inside the firewall to your own employees or maybe to consumers. You're building products that you might have to deploy to thousands of businesses that all have their own unique demands with respect to what they want to do with those AI capabilities. And so you've got to be able to prepare for that when you design your products.

4:40Yeah. And these are independent software vendors that are building products on AWS or with AWS using RedDock or SageMaker or whatever the... That's right. Yeah. Or even considering it. you know they may not be building on on aws yet but uh they think that there's an advantage that they can get by doing that yeah you did a survey with forester research uh about uh how successful isvs are in uh in working uh with generative ai can you talk a little bit about that Yeah, yeah, happy to. One of the things that we do when we're thinking about strategy and serving our customers here is we like to dive deep.

5:37And data is one of the ways that you can dive deep into understanding what's going on in an industry segment. And so we commissioned this survey in the fourth quarter of 2024. and we went out and Forrester designed the survey, ran it, got over 650 responses around the world, spread across three different regions, North America, EMEA, and APAC. So it gives us a good perspective on what's happening around the world, not just here in North America. And what we asked the Forrester researchers to do was to ask about implementation challenges, plans, future goals, what these software companies were trying to accomplish.

6:27It gives us insight into how we should help them, but we also wanted to make sure that that data was more broadly available because it helps inform decision-making with our customers as well. So it was a lot of fun to take a look at the details, you know, I often say with surveys, the data is the data and it's interpreting the data that is what I like to do and then help our customers work through that. Yeah. And there's there's been this rush to implement generative AI. A lot of experimentation, a lot of pilot programs, but they've a lot of people have ended up with premature solutions and unproven business models high development costs so they need to prioritize value-driven use cases right to succeed can you talk about that about about how they do that priority and prioritization?

7:38Yeah, yeah. So first of all, I think it's important to understand that that sort of rush to build isn't necessarily unique to generative AI. I've seen it in lots of other major technology disruptions. I think about when the iPhone came out in 2008, we had lots and lots of organizations building apps as quickly as they could without a clear understanding of how they were going to drive value. And then, you know, they kind of learned what worked and what didn't. And so I think that we're working through that process. It's one of the reasons that I think it is so important with respect to generative opportunities to work backwards from the customer.

8:23And that's just something that's like second nature at AWS. You know, there's a quote from Jeff Bezos in our 2016 shareholder letter that talks about the fact that customers are always beautifully, wonderfully dissatisfied, even when they report being happy and their business is great. And so if you go out and talk to a customer and you ask them a question like, where do you have to do things today in this product? And it's still a pain, a pain wherever you want to call about it. And I'll use expense reporting as an example. I've been doing expense reports for over 30 years. And I think about how I used to do expense reports when I had to tape receipts to pieces of paper.

9:08And then I had to scan those pieces of paper. And then I had to mail them in. That was a high toil process. But even today, you know, there's still a lot of toil involved in how I do expense reports. Anything that's over$50, I've got to have a receipt for. And sometimes that means I've got to go search my email for the receipt that was sent to me by the hotel. And so there's all these kind of low value added activities that even though I like our expense reporting system, it's actually a lot better than, you know, anything else that I've used in the past. There's still opportunities for improvement.

9:40And that's really the starting point of being successful in driving business value in AIs, working backwards from the things that customers want. So toil reduction is one of those. There's lots of good examples of very successful use cases in that world. Accounting is a really good example of a business where there is high toil and there's also a shortage of labor. If you look at the research, there are just not enough accountants to fulfill the demand. And so you're seeing practically every accounting software company that we work with making large investments in generative AI to reduce the toilsome activities and give accountants more leverage.

10:28Healthcare is another example of that. I was talking with a company last week that's a customer of ours. And one of the things that they said is whenever a lab runs a blood test, you know, there's a whole bunch of figures that come back. And yeah, a doctor or a clinician can take a look at those and spend, you know, seven or eight minutes kind of looking through the lab and trying to extract that information. But what if we can automatically extract the most important information, give them the summary at the top and reduce the amount of time that they spend scanning through that report from eight minutes to three or two.

11:05And what happens if we can do that dozens of times a day? It's a small example of toil reduction that adds up in the long run. So very useful use case. Retail. I worked with a company a couple months ago where they had a case where third-party selling is a really big deal in retail now. And you think of all the third parties that sell on Amazon. And one of the things you have to do to enable that is you have to map products from one catalog to another. And that can be a really, really tedious process if you've got a thousand products to map. So one of the things that they said was, well, if we could get to 90 % accuracy with a generative process, we could automate this.

11:51And so taking that identification of toil and then starting to work backward from that was one of the things that they did. There are other examples, content generation, that's what we all know of, executable artifact generation. To me, the perfect example of that, code is what we all think of. Code is an executable artifact that computers can run. But there are so many other things like a calendar appointment, the ability to automatically create that, send it out and synchronize it. A data pipeline, SnapLogic is a really good example of a company that has a strong meta model underneath its product.

12:27And so you can either understand those pipelines that exist and summarize them so that they can be explained, or you can create them. And there are just dozens of areas in the software space where there are these executable artifacts that are ready for a higher level of abstraction. So these recurring use cases, and there are different types of them are the basis for creating a product strategy, but it starts with understanding what's giving the customer heartburn. Yeah. Yeah. And you were talking about the time savings for doctors reviewing medical tests. I've talked to a lot of people, a lot of enterprises about, you know, if you use generative AI to focus on time saving, on productivity, it doesn't, across a large organization, it certainly is great for the user, for the person in, you know, in an accounting function or something.

13:38that can use Gen.I. to summarize and it saves them a few minutes. But those incremental time savings don't necessarily flow to the bottom line. You know, if I'm an employee, I've saved five minutes. I'm going to check my personal email or I'm going to do something else. And there's something in the notes here that were sent over by your team where you talk about finding real value-driven use cases that are innovative, not simply time-saving. Can you talk about how you do that, how you identify those deeper value situations? Yeah, yeah. I think one way to do that is to start by asking the question, where are there opportunities to serve customers or to do things that we have not been able to profitably do before?

14:45And I'll give you a good example. And just for listeners, when you say customers, you're talking about AWS customers. No, I'm talking about the ISVs customers. So again, I'm always working backward from the ISV and thinking about their customers, because if they can serve their customers well, they're going to drive profitable growth. So here's a good example. Executive coaching. Executive coaching is something that has traditionally been done, you know, kind of person to person. It's very high touch. As a result, it usually only happens at the executive level. There's a very, very small number of people that organizations are willing to support with that model.

15:34Well, what if we could open up, you know, that sort of career development capability to a much broader set of employees in the organization because we could do it at a much lower cost? That's an example of where an innovation, being able to take best practices, being able to take the identified competencies that an organization values and expose them through a generative approach, maybe an agentic approach, could really unlock opportunity that to this point in time is just underserved. If you look at what's happening right now in marketing around hyper personalization, or client telling, you know, the reality is, is client telling was only you, you know, really done with with your top tier customers, because it was too expensive to do it for your more casual customers.

16:26Well, if we can change that equation, then you can use it to drive additional value and expand the market. So that's where some of the big ideas lie. And I'm not saying you do those just instead of the little things, because the little things can add up. I think cogeneration is a great example. I shared the stage at reInvent this year with the group vice president from New Relic, Siraj Krishnan. And one of the things he talked about was that they've been able to drive a 15 % increase in developer productivity by deploying, you know, essentially code, co-pilot capability. When you're talking about a couple hundred developers, 15 % is nothing to sneeze at from an expense perspective.

17:16So you've got to do both. yeah you you also want to be careful that you you're doing uh implementing these ai strategies in a careful way so that you don't lose trust of your uh customer can you can you talk about that yeah yeah uh for sure um this is actually i think one of the takeaways that that was a little bit surprising from the survey that, you know, I thought that we would see a little bit higher priority placed on customer trust and on data security and privacy in the survey results. And I think that that was a little bit of a miss from some of the ISVs because we do see that as important.

18:07It's one of the reasons when we built Bedrock by design, the way that Bedrock is deployed it's designed so that the data stays private, that it never leaves, you know, the private, you know, environment that the customer set up. Because, you know, that that's a huge concern from our customers, customers. And, you know, again, I don't think it's necessarily unique to generative AI. I remember 10 or 15 years ago, when we first had the opportunity to put source code up on the cloud with something like GitHub. And there were a lot of organizations that were like, well, I don't want to put my code up on the cloud.

18:45What if I lose control over it? That's my core IP. And then over time, you know, we learned to trust it. We learned that it was that, you know, it could be implemented in a private way, that my code wouldn't get exposed. And that trust level got to the point where essentially storing source code in the cloud is just the way that most companies do it now. We have to get to that same level of comfort with the data that customers are using because one of the things that we see over and over again is one of the best ways to improve the accuracy of these generative use cases is to include data through retrieval augmented generation or cache augmented generation.

19:27And so making sure that that data is safe, critically important. So yeah, I think there's probably some more work to be done there. Yeah. And is the issue that ISVs adopt AI solutions in their software stack and overlook privacy issues or security issues? or is it that they're not communicating them adequately to their customers so that customers trust the the this new ai uh powered solution yeah i think it's part of the learning experience what i'm seeing is like when we work with a customer uh and we go through a poc so much of the focus is on just seeing if we can get to the right economies of scale you know where the costs are low enough, the accuracy is high enough, the performance is fast enough, and then they start to worry about the additional illities.

20:35Or sometimes what they'll do is they'll put this out as a tech preview, and then the enterprise customers start to ask these questions. And then it's like, okay, you know, we better make sure that this is, you know, that we've got these capabilities as well. So I think it's mainly that it's a priority that we need to be more proactive about, so we design it in from the start. Yeah, and AWS, when you're working with a software vendor, are you advising them of this, or you're recognizing through the survey that these are things that they need to be paying attention to? Yeah, when we work with a software vendor, we definitely call this out.

21:26And it's, you know, part of the value that, you know, getting some of our subject matter experts involved in the customer when they're building provides. So as an example, if we're doing an engagement with a software company in our generative AI innovation center, it's just part of the discussion of working through the process of the use case and then building it out excuse me yeah bless you

21:57excuse me comes in threes usually uh yeah yeah and i should also say it's one of those things that um that we build uh in to the product strategy as well you know the idea of secure by design so So a great example of that is beyond the design of Bedrock itself and how it treats customer data and information is the capabilities that we layer on top of it, something like Bedrock guardrails as an example, specifically designed to reduce the instance of hallucinations, to reduce the possibility of objectionable content sneaking through. You know, some of the work that our partners do to be able to potentially identify issues like data loss that might be flowing through the prompts that are coming in to an LLM.

22:46So, yeah, that's where the value of using a cloud platform and helping to accelerate the release can pay big dividends because otherwise you've got to solve the problem yourself if you're doing it all on-prem. Right. Can you talk then about how do you work with an ISV? I mean, all these guys are, unless they're Gen AI native, they're trying to figure out, you know, how to use generative AI in their products. Right. These presumably are people that have already built products on AWS platforms. Do they come to you? Do you reach out to them? If they come to you, is there a process that you go through with them?

23:35you know, a checklist or yeah, how do you, how does that work? Yeah, yeah, absolutely. So first of all, it starts at the account team level. I just came back from a week and a half in Australia and New Zealand meeting with customers. And I was talking to some of our account teams out there. And one of the things they mentioned to me was that every single AM that's responsible for working with ISVs is currently going through the certification process to be an AI practitioner. So it's the same external certification that our customers get. So they're the ones that have those discussions. When the customer has questions, then they bring in the subject matter experts that are aligned with the particular services.

24:19They also then will identify opportunities to help accelerate those. And so I already mentioned the Generative AI innovation center. That's one way that we would take an opportunity and define it and then move to a proof of concept. There are other things that we do. There's a program that I actually co-run. It's called the Roadmap Acceleration Program. And we do workshops with an ISV where we'll say, let's take a particular use case that you think is high value. And we'll go through the five questions that we use internally at Amazon, which are part of our working backward process. From those five questions, we'll go ahead and draft a PR fact, press release, frequently asked questions document.

25:07This is the way that we build our products. As part of that, then we also have reinforced that standard working backward process with some specific questions around building generative AI functionality. Like as an example, why is generative AI the best way to meet this customer's need, because it isn't all the time. And I think that that's one of the things that the Forrester survey kind of showed that there was a little bit of hype here. And it's like, I need Gen AI. It's my solution. Let's go find the problem. What you really want to do is say, what's the problem? What's the toil? What's the dark data?

25:44What's the content? What's the area that we've never served before? And then is Gen AI the right solution? And what specifically in Gen.ai? Is it generating content? Is it automating tasks? Is it, you know, being able to use natural language so that a larger user set can use the product? And then, you know, that question about data is critically important because one of the things that we see from a pricing perspective, if you want to drive business value as a software company. If you've got unique data that is hard to acquire, if you've got control of the customer through a user interface or a hero product, if you are making a creative user or a user that's in short supply customer, if you're making them more effective, you're giving them leverage, those things are more valuable and allow you to to look at user-based pricing models instead of consumption-based models they let you look at potentially outcome-based models they let you charge a higher price premium so a great example of that one of my my favorite software companies out there is canva if you look at their hero product there are just dozens of little places inside their product where they've implemented generative capabilities, which collectively speeds the whole user experience for the creative.

27:16And the net result of that is that they've been able to steadily increase what they charge for their product because the value that it provides is apparent. Yeah. Yeah. The pricing models, that's something I've seen are also changing. I know in customer service, the old model is you charged by headcount. And with the integration of generative chatbots and seamless handoff and all that stuff, these companies are starting to charge by outcome. 100%. Yeah, which is wonderful for their customers. Yeah. Rather than paying a fixed headcount cost. Can you talk about, you know, as these companies are shifting to hopefully sustainable growth and profitable, sustainable growth, are there steps that they should, is there a checklist that you go down?

28:28And also, before we get to that, you mentioned a certification. What certification was that that you were talking about? It's the AWS Certified AI Professional. I see. Is that for ISVs or is that for... It's general. It's for anybody. It's similar to what we have as our Certified Cloud Practitioner, which basically every Amazon employee, when I joined a number of years ago that was customer-facing, was required to get. But it's the same thing that our customers, you know, take as well from a training perspective. And this training is required or it's just service that Amazon offers customers? No, it's a service that we offer our customers to help them get up to speed to develop their own skills.

29:17But, you know, it's kind of like physician heal thyself. The best way to be able to advise, you know, customers is to make sure that all of our folks have that same level of expertise. And that's actually something that I've actually found really invigorating here at Amazon that the expectation that the sellers are going to have the same level of skill and competency that we would normally expect out of software architects or technical sales support is one of the ways that we help identify those opportunities so that we can progress them. Yeah. You mentioned the Generative AI Innovation Center.

30:03Can you explain a little more what that is? Yeah, that is a set of investments that we made at a corporate level to put together a set of specialists that are designed to accelerate the big ideas that our customers have. So it's not unique to ISV, but we have a number of ISVs that have taken full advantage of that. I'll give you one example, Deputy, which is an up and coming ISV out of Australia that, you know, our team has worked with extensively. In their case, the use case that they wanted to focus on was what I would call an optimization use case. They wanted to get more effective in how they built their own products, which are for small businesses, workforce management, that sort of thing.

30:50And so, you know, the Gen AI Innovation Center worked with them to really push hard on toil reduction for their development teams and their product teams. So code generation, but also collaboration within the product teams, how they prototype. And net net, they were able to get to a point where they think they can get a 70 % improvement in their product team execution. That's pretty significant. From an external perspective, BMC is a company that worked with the Gen AI Innovation Center specifically around the price and performance of the models that they were working with and the use cases that they had developed because, you know, they wanted to get to this point where the cost of goods, the cogs for delivering this inference was low enough that they could, you know, make money off of it.

31:38And so in that case, the engagement with the Innovation Center was really focused on performance and cost reduction. And in that case, I think they reduced the costs on an order of 40 % or more, I think. So, yeah. So, you know, those are just examples of the types of engagements that happen at that level. In my case with, you know, the work that I do with our individual ISVs, there's a kind of a stepwise process that we go through. we start with this idea of the high priority use case. In some cases, it's an internal use case. They want to increase and improve their sales effectiveness. I did one where churn reduction, identifying churn reduction early on in the renewal process was critically important because the customer had a churn issue with their existing product.

32:38And so we worked on that. In other cases, it's product embedded. So we want to focus on one of these total reduction use cases. I think of a case around environmental health and safety. And there's a lot of opportunity there through processing video, as an example, where you can identify safety risks or where you can look at the result of an inspection and start to look at failure trends and identify, you know, what might need to be done to reduce the number of failed inspections around a particular class of asset. So you can start with that use case and then start to build out what has to be done to make that use case successful.