Building LLM-Based Applications with Azure OpenAI with Jay Emery - #657

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Jay Emery, Director of Technical Sales and Architecture at Microsoft Azure. Before we get into today's conversation, be sure to head over to Apple Podcasts or your listening platform of choice. And if you enjoy the show, please leave us a rating and review. Jay, welcome to the podcast. Thank you, Sam. Thank you for having me. I'm excited to talk to you. You spend a lot of your time working with organizations, startups, and fast-growing organizations in particular that are building around large language models.

0:46And I'm excited to kind of pick your brain around what are some of the things that you're learning, that you're seeing them struggle with, that you're helping them with. Should be a really interesting conversation, especially given how quickly this field is moving and some of the cool things that people are doing. Absolutely. Buckle up. It's constantly changing week by week and sometimes hour by hour. It's fantastic. It is. And I guess I'll mention, I don't necessarily often kind of talk about the things happening behind the screen or behind the veil or whatever. This is the second time we're having this conversation.

1:20We had a technical glitch. Doesn't happen very often, but it happened last time. And the audio quality wasn't where we like it to be for the conversations. And so we're having this one again a week and a half later, maybe. And you could almost say the world changed like two days ago, three days ago when OpenAI had their Dev Day. A lot of the things that we were talking about is maybe possible futures like we actually heard about at Dev Day. And we may touch on that. But before we jump into the conversation, I'd love to have you kind of introduce yourself to our audience, share a bit about your background and what you do at Microsoft.

1:59And I think we have St. Louis in common. You spent a bunch of time there? We do. I spent almost two decades working for Anheuser-Busch in various engineering and IT leadership roles. And funny to note, I actually spent like five or six years abroad in Europe doing various projects and strategic initiatives. So finally brought me back here, California, where I joined Microsoft and have been with Microsoft for almost four years now. Like you mentioned earlier, really partnering with digital natives and startups. The team really focuses on what we would consider strategic partnerships within the startup ecosystem.

2:39So we partner a lot with venture capital firms and some of their portfolio companies and finding ways that the Microsoft relationship can ultimately grow their total addressable market or make them more successful by getting them access to some of our enterprise customers. Is generative AI and the Azure OpenAI products, is that what takes up all of your time? Is that just one of the tools in a broader tool chest? It is. It's interesting, Sam. So now it is. I think that a year ago, two years ago, a big portion of what was attractive to startups was our partnership, our ability to kind of reach into the enterprise customer, the large Fortune 500, Fortune 1000.

3:26And lately, it's been everything AI, the advent of the large language model and Azure OpenAI and OpenAI has really just busted open the doors for opportunities for us to find ways to partner together. And many times, some of the startups that weren't answering our calls are actually calling us directly. And so it's been a very refreshing and exciting time to be at Microsoft. That to call the OpenAI partnership strategic seems like an understatement. It is. I will say, and I mean this with all sincerity, I think the cultural change that Satya brought to Microsoft has just been fantastic. And it's evident even in these partnerships with OpenAI, where we were very likely developing our own large language model, but he was willing and able to lean in, notice when somebody is doing a better job or is maybe ready to go to market much sooner.

4:19And to ink that strategic partnership has just, you know, one done wonders, I'm sure, for open AI, but it's just really revolutionized Microsoft and ready to take us into the next century. And in fact, Microsoft historically has been very well positioned from a research perspective. You know, for a while, many of the best conversations I've had around NLP were folks at Microsoft that all had this like common root in having worked on Bing. Just a lot of expertise there, not to mention, you know, some of the achievements coming out of the Microsoft research. And I think one of the things, obviously, Azure OpenAI gets kind of the lion's share of the press and attention.

5:02But there's even a bigger story around how it ties into maybe our cognitive services portfolio in general. So it's truly a one-stop shop. Azure OpenAI is obviously kind of the crown jewel in it at this point. But we have some really capable products and services that can help enhance or aid in developing your own large language model or figuring out ways to kind of integrate it in with your existing IDE. So try to characterize for me the kind of maturity level, I guess, of the companies that you're working with. Are they all coming to you knowing all about LLMs and, you know, having built products and trying to get some help optimizing, kind of putting the cherry on top?

5:47or are they earlier stage and needing help kind of getting started? Like, yeah, I imagine that, you know, they come in different flavors, but like, how do you think about the folks that you work with and kind of where they are from a maturity and knowledge perspective around LLMs? We benefit from getting to work with startups and or what we would consider digital native organizations. I think that they have taken a more aggressive and maybe thought forward approach to large language models. I think if you look at a traditional enterprise customers, with rare exception, most don't have a large language model expert or maybe even an AI expert in staff and rely pretty heavily on consulting firms and external service providers to bring that to bear.

6:35Most, if not all of these digital natives have someone on staff that's familiar with it. And I think it's just the clairvoyance of knowing that's where the industry is going. And so what we tend to find with startups is they are ready to go feet first in. And a lot of times we're super early adopters. We, of course, have seen some shifts and trends with large language models probably the past year, year and a half. But by and large, the customers that we interact with are very thought forward. They're either figuring out how to integrate generative AI into their existing product or platform, or maybe they're actually developing their own and need to leverage some of the GPU capacity that we've fortunately been able to build out and share to our customers.

7:19So that having been said, what do they need you for? Like, what are the challenges that they're running into that cause your proverbial phone to phone line the ring or your teams to start blinging? Exactly. We kind of tend to see a lot of times there's this, I would say, misconception around security or data privacy. I think that with any new technology. The CISO's door is, you know, bell is ringing off the hook and there's a lot of concern, especially in the startup space. IP is their lifeline. And there's a lot of concern about IP leaking or being leveraged by a competitor. So one of the very first things we do is try to dispel any myths around their data or your data being used to train new models.

8:04And it's simply not the case. I think it was stories like that Samsung story continue to live on in people's minds. They do. It is. And so, you know, if you look back at 3.5 Turbo, that model was trained up until I want to say it's like September 2021 data. And it was only leveraging openly available, thus an open AI, things like Wikipedia or publicly available novels or articles or books. And in essence, the way that the models work is there's data put into a prompt and then the model does its stuff and brings back a response. And it's always done in memory. And after either your session terminates or you end your session, that memory is re-released to the system and is ready to bring on the next prompt and response.

8:52And so there's no point in time where that information that you're putting in is stored to disk and used to retrain. The one exception that I would call out is maybe our content moderation. I think one of the benefits of kind of having the power of Microsoft behind various models is around our content moderation. We take things very seriously. So we purposely screen both prompt and response for things like violence or hate speech or sexual content. When it's detected, it's actually stripped from the response. So you don't get that back. But then also it's stored where one of our human individuals can come and review it because there might be a reason for the language, right?

9:36It might be, I love to use the example of you could go in and ask for the lyrics to a Nine Inch Nails song and it will not come back with all the language in the songs, but there might be a reason why you need that language. And so we also put in different places. You could bring your own content moderation. You don't need to use ours and we have options to kind of opt out of it. You have to go through some process to do that. But things like health data, PII, you know, it's really not acceptable to record anywhere. And so we can you can opt out of those things through one of our internal processes.

10:07One of the things it sounds like early on folks will engage you with is just really understanding how privacy, security, those types of things work. And it sounds like that's like the C-suite folks trying to understand if they can even do this. Like if it is within bounds, given whatever privacy framework that they have in place. Beyond that, I'm imagining, you know, engineers are running into kind of technical challenges focused on their various use cases. When you think about folks trying to implement a particular use case and trying to get the LLM to do what they want, basically, you know, is that something that you get a lot of calls about?

10:52Well, it is, you know, I'll paint with a broad brushstroke here, but people tend to are getting better, I would say, at thinking maybe a little bit further out. You know, I think the initial gut response is, oh, I can create a great chatbot that leverages natural human language. But it's the companies that are really kind of thinking beyond that and how are they going to integrate that or tailor it to their specific business or business processes. We see three main ways that our customers are able to leverage large language models to really drive that business impact. The first, and it's actually probably the most common, is with prompt engineering or prompt tuning, prompt chaining.

11:29I mean, it's really about putting in the right level of detail and information into the prompt, getting a response back. The customer getting more and more innovative, I would say, on how they can actually take the outputs of a prompt and actually feed that back into the prompt of the next one. And that chaining or that tuning ultimately delivers a much more rich and robust answer. And we've got some first-party tools, and I'm sure there's open-source tools as well that are available, but things like PromptFlow, which it does a little bit with kind of like a graphical flow or workflow tool that can actually chain together different models.

12:08So you can pull up Hugging Face and use the output of Hugging Face model as an input to 3.5 Turbo, or you can use Llama 2 and feed it into GPT-4. So there's a whole host of tools and capabilities that are available to the customers when they kind of buy into the ecosystem. And the other thing that is really interesting is they have this concept called prompt variance. and you can actually use AI to tweak your prompting questions in just tiny different ways that will actually, believe it or not, produce, you know, in some cases, dramatically different answers. And so you can actually go in and fine tune how the questions are being asked.

12:48You don't have to think of it. The AI can rephrase it for you. And so what we're finding is a lot of startups leverage that tool to make sure that they're getting the most out of their prompts. And I love to use the example, we work, we have a lot of ed tech companies. And I think of any industry, I think if you watched NBC Nightly News or, you know, name your news network, you know, people were prognosticating the death of college and kids education, and now everybody's going to be cheating. And I think what we found is the companies that kind of lean into the challenge and try to think about how do they embrace the technology versus fight it are really coming out with some really cool examples and use cases.

13:29And so we spoke with one, EdTech, and the concept behind it was imagine you are a high school history teacher and, you know, you've got your normal lesson plan and state mandated things that you have to teach your class. But imagine if you want to kind of pivot a little bit and maybe look at, I don't know, Abraham Lincoln's early life. You can actually leverage these large language models to create an outline for you using prompt chain. You can feed it back in. So you can have it create an outline, feed that outline back into the model to generate actual content or course material. And then also leverage that output to generate quizzes and answers.

14:06And ultimately, you can feed essays and responses back into the tool and it can grade it based on like a matrix of scoring that you can also feed the model. So it's just a really brave new world and have no doubt even places like education are going to look drastically different than what we've seen in the past. That was prompt engineering. I forget the name of the company, but that sounds a lot like one of the examples that was shown at the OpenAI Dev Day in the keynote of a company that was building. I forget if it was an assistant or a GPT. I think a GPT. But the example that they showed was a teacher going in asking for like a custom lesson plan or something.

14:48Yeah, it's amazing what we're going to be able to do and ultimately the impact on everybody from us down to our kids. It's a really, really fun time. And this prompt variance, is that a tool that is available? It is. So we have a product as part of our cognitive services. It's called Azure Prompt Flow. And in essence, it's a graphical, you can chain together different modules. You can take the output of one open source model and feed it as an input to a next. And part of that is there this, it's called variant. I think it's called prompt variance. And it will actually allow you to come up with maybe if you have a prompt, tell me about Abraham Lincoln's early life.

15:25You can put in that into the prompt variant and then it will generate up to 10 or more, set the value, different ways of asking that question. So Abraham Lincoln was important in his early life. Tell me about it. So it gives you a couple different or several different ways to ask the same question. And then believe it or not, there are differences in the outputs because it's so non-deterministic. And some of the ways that you can ask those questions are. And then one of the challenges that I hear about all the time is, hey, we try all these prompts. It's kind of like throwing spaghetti at a wall.

15:56The output's different. When we were doing old school machine learning, we built out these experiment management tools so we can kind of track our results and compare AUC for different hyperparameters, et cetera, et cetera. Like we'd kind of just figured that out. And now we're doing all this tech stuff and evaluation is really hard. Like, are there tools that you are aware of? Or I'm imagining that comes up with the folks that you're talking with as well. You know, there's no lack of open source tools that are popping up everywhere. I think Ling Chang is a popular one that we see a lot of maybe early adopters using.

16:31You know, we have our own kind of first party tools and solutions. So Azure ML AI Studio is kind of our foray into that space. It tries to take a more graphical workflow project type of spin on it. Feedback from those customers that have adopted have been very positive. But I think when you look at what I would call like the picks and shovels of AI, I think we're really in that early stage. And it's actually where we're probably seeing most of the startups come up is creating tools to make it easier. The challenge, I think, Sam, to be honest, is you want to make sure that your startup isn't one feature away from being irrelevant.

17:12And so we are actually starting to see where some of the larger language models like Azure OpenAI or maybe Llama 2 or Hugging Face, they could actually add a feature such as PDF input. And that is actually negating what some of the startups are doing or innovating on. So you just have to be a little bit careful. Is your tool or product a feature or is it actually a product that has long-term legs? Yeah. So that's prompting. It's prompting. The other one is fine-tuning. So fine-tuning, this is always one of the other kind of big misconceptions that we get is, you know, you have to fine-tune. And I think where that came from is really from early adopters in this large language model space.

17:57I think that the prompt sizes were small enough that you couldn't get enough into that prompt response to be as effective as you wanted to be. But by and large, that's kind of left the that horse has left the barn. If you look at 3.5 Turbo and 4, it's what I would say, 16 ,000 and 32 ,000 tokens, respectively. And then coming out of the OpenAI Dev Days this week, it was like 128K tokens, which is essentially, I want to say it's like a 300-page book. And so, you know, the ability to just ingest large amounts of data just using prompting likely going to continue to grow. The other thing that we talk a little bit about with fine-tuning or those folks that are interested in it, there's absolutely a place for it.

18:45You know, I don't want to I don't want to just lead the audience, but it's expensive. It's time consuming and it's research intensive. And so, you know, you really want to work with your solution architect or your SI or your hyperscaler to really understand, you know, when is the right time to use it and not. We've got one startup that's a little bit top of mind because we just got done doing a briefing with and some work with some of my solution architects. But it's a software company that does essentially workflow. So you can create flow charts or business charts or business processes. And essentially what they've done is they've taken GPT and they have taught it their proprietary language, so their domain-specific language.

19:26So what will happen is customers will feed in maybe complex descriptions of projects or processes. They'll actually feed it into 3.5 or 4 to get kind of that rich summary back. And then they'll feed it into a fine-tuned model that can then translate that into their own domain-specific language. So in the grand scheme of things, they basically taught GPT a language that it didn't know existed. And the language, the domain-specific language, doesn't change a lot. So they're able to train it once and not have to, you know, continually train and update based on knowledge that becomes available. It's a pretty static environment.

20:03It sounds like that's an example of both the chaining that you were talking about earlier and the fine tuning. So they use off the shelf three five turbo faster, cheaper to do some manipulation to get the input text into a better format for the downstream task. And then they have this downstream task that they fine-tuned that is taking that intermediate description and turning it into a workflow language, essentially. You're 100 % right. It's never or very frequently is it just one way of doing things. You know, I think what we're seeing is this hybrid approach of leveraging the power of GPT that comes out of the box, but then also bringing in maybe intellectual property or data in an external corpus to really take it to that next level.

20:55And I think that's where the third one on the way we're seeing things is called retrieval augmented generation or RAG. And ultimately, and I'm sure you've covered it with your audience in the past, but the prompt goes, looks at that external corpus of information, retrieves it, really augments the prompt. And then the generative response comes back and it's much more rich. It's much more specific. And we're actually seeing a lot of companies leverage RAG much more than the fine-tuning. I'll throw an example out. We've got a startup. And again, we actually had a really interesting open hack with them just recently.

21:33But they're a sales enablement platform. And they use RAG. They actually have made available all their proprietary marketing and best practices and sales data to ultimately generate customer sales collateral. And they leverage retrieval augmented generation to do that. So they can continually add to that corpus of information and then not need to obviously retrain if they had to happen to do fine tuning. And so it's able to pull that relevant content and then feed it back. Meaning in their case, this is an internal use case as opposed to a product. They've got their sales and marketing team that has created their own collateral and they want a salesperson to say, well, I want something that talks about this product for healthcare vertical clients and it'll go and pull that information and kind of create something.

22:23Totally. So as they were developing that use case, they also came and found another one where as they hire new sales engineers or sales staff, you know, how do you effectively as their manager coach them on what a good pitch is versus not? And so what they're actually also doing is leveraging like teams meetings or getting the transcripts from conference calls and then feeding that transcript in and then kind of grading it or scoring it based on a matrix of what, you know, how many times they talk about business challenges. How many times did they let the customer finish before interrupting? And they're actually able to coach and guide the sales team to be much more effective earlier on in their career.

23:05It's really cool use cases that we're seeing kind of come out. Yeah. One of my favorite things about talking about LLM use cases is for each one you talk about, like you come up with five or 10 more that are like just beyond that, that would be really cool. Yeah. It's funny because I mean, I could go on all day, but one of the ones that we were able to, we had like a private preview of our Microsoft 365 co-pilot. And so we had several large enterprise customers take part in that. And we really took their feedback to heart. But one of the interesting one was there was a senior leader at this ISV or independent software vendor that would use the transcript to see was her team being inclusive in their meetings.

23:47And it was really focused on, OK, did Bob talk over everybody? Did Bob talk 80 percent of the time? How did Billy, did he speak up? And so she was able to leverage it to really provide feedback and coaching to her team in the space of diversity and inclusion. And so it was a really different use case after you saw it come out. And I think every organization, as they gain access to these tools, are going to find some really interesting ways to make use of them that maybe you wouldn't initially have thought of. Right. So kind of talking about RAG or returning to RAG, how do you see folks evolving their RAG deployments?

24:27Like, I guess my experience with it is that conceptually it seems easy, but it can be harder to get to the desired end product. And not to overload the use of the word tuning, it takes a lot of tuning or fiddling with to get it right. Do you see that as well? Honestly, I haven't. It's not to say that it's not prevalent in the space. Part of the challenge with RAG is, you know, really vectorizing that external corpus of data. And I know that there's some, you know, third party solutions we see and use Pinecone quite a bit. Microsoft has its version of vector databases. And so I think that what we're probably going to see is people become more fluent and adept at kind of vectorizing their data that would ultimately kind of speed up or helped improve the performance and response and effectiveness, I think, of that retrieval of that external corpus of information.

25:27But I haven't seen it be a mass issue, at least across these startups and digital natives. And so that's kind of the things that folks are running into just trying to tailor LLM responses to their particular use case or business. Another area that from our last conversation, I remember we talked about was the whole idea of incorporating LLMs into like core business systems and core business workflows and some challenges that you have run into there. Talk a little bit about that. Yeah, I think when I see some of the digital natives try to weave it into their core product or service, the performance, the inference performance can cause some challenges.

26:11I think that one of the misconceptions that we run into is, oh, it's a performance issue. We just throw more hardware at it. Let's get the next big version. And it's not always the case. I think inference is inefficient and low latency is difficult to obtain. I love that there's a statistic that I heard that one single GPT-4 prompt response, if you were just to use a single Intel processor, it could take 39 hours crunch through. And so it really tells you just the massive amount of compute and complexity that's happening kind of behind the curtain. And so making sure that when you're developing your app or designing how it's going to integrate into your product or services, you really need to make sure you realize that latency is never going to not be there or at least will be there for the immediate future.

27:01There's no magic bullet, but we do find ways that the customers can kind of lean in and work through it or around it. Happy to talk through those as well, if that's of interest. One thing that you said was interesting, you said that throwing more hardware at it wasn't always effective. Is it ever effective? Like, is that a thing that a customer can do for Azure OpenAI GPT model? Is there a knob to scale the inference hardware? Yeah. So let me kind of walk through for the audience the three ways that we could speed up performance. And I think the last one is absolutely going to hit on what you talked about.

27:40I think the first one and kind of the easiest one, in all honesty, is picking the right model. Jury's out. I haven't seen kind of the metrics yet on what got announced at OpenAI Dev Day earlier this week. But prior to that, 3.5 Turbo was the fastest you were going to get. I think that when looking at 3.5, your prompt response can be anywhere from three seconds to 15. GPT-4, it's like 15 to maybe upwards of a minute, but you get much more rich content back. So figuring out how to or which model to use, I think a lot of the kind of the leading startups are actually using like a pre-processing engine to determine where to funnel their workloads.