Building the Foundation Model Ops Platform — with Raza Habib of Humanloop

21:52They'll probably sign up first and have a play. But then they tend to book a demo because they want to discuss data privacy and how things will be rolled out and sort of going beyond just individual usage. But that's the usual flow is we see them sign up. Sometimes we reach out to them. Often they'll reach out to us. And then the conversation starts. Yeah. Yeah. Awesome. For people who want to get a better sense of Humaloop, the company, I think the website does a fantastic job of explaining it. Thank you. We're always working on it. Put in quite a lot of work. So it says here, Humaloop application platform includes a playground, monitoring, deployment, A-B testing, prompt manager, evaluation, data store, and fine-tuning.

22:30And based on our chat earlier, it seems like evaluation is kind of the more beta one that's in sort of like a private beta. That's correct, yeah. So we have evaluation and private. So there's always been some aspect of evaluation. It was actually the first problem that we were solving for customers. But evaluation in human loop early on was driven entirely by end user feedback. So if you're building an LLM app, there's probably three different places where evaluation matters a lot. There's the evaluation that you need when you're iterating in design and you haven't got something in production yet.

23:00But you just need feedback as you're making changes. Are you making things better? You're iterating on prompts. You're iterating on the context. Trying out different models. like how do you know that the changes are actually improving things? Then once you're in production, there's sort of a form of evaluation you need for monitoring. Is this, you know, it seemed to work when I was in development, but now I'm putting a whole bunch of different customer inputs through it. Is it still performing the way that I expect it? And then the last one is something like equivalent to integration tests or something like this.

23:25Every time you make a change, how do you know you're not making it worse on the things that are already there? And so I think we always had a really good version of the monitoring user feedback version. but what we were missing was support for offline evaluation and being able to do evaluation during development or regression testing. And we're going to be launching something for that very soon. Yeah. This is slightly unintuitive to me because I would typically just assume they're all three are the same evals. Yeah, so they can't necessarily be the same evals just because you don't have the user feedback at the time that you're in development.

23:59I'm not thinking about user feedback. I'm just thinking about validating the output that you get. Yeah, so you're validating in similar ways, But if you're doing a really subjective task, then I think the only real ground truth is what your customers say is the right answer. If you're building Copilot, do the customers accept the code suggestions or not? That is the thing that ultimately matters. And you can only have proxies for that in development. And so that's why those two things end up being different. Yeah. And in terms of the quality of feedback, so we did an episode with Amplitude, which is an analytics platform dedicated for collecting this kind of behavioral feedback.

24:33and you mentioned Copilot. There was a very famous post about reverse engineering Copilot that showed you the degree of feedback. I think typically when people implement these things, they implement it as a sort of thumbs up, thumbs down, sort of binary feedback until you find that nobody uses those, nobody does those feedback. Like I barely use the up down on ChatGPT. Yeah, so this was something we learned really early on in building HumanLoop. And, you know, the feedback aspects of HumanLoop were very customer driven. People were, the people who were getting, amongst our early users, the people who were getting traction and who had built something that was working well, had jerry-rigged some version of feedback collection and improvement themselves.

25:12And they were pushing for something better here. And they all were collecting usually three types of feedback. And HumanLoop supports all three of these. So you have the thumbs up, thumbs down type feedback that you just described. You don't get much of it. It's useful when you get it, but you don't get that much. And then the other form of feedback, so we call that votes. And then you have actions. and these are like the implicit signals of user feedback. So I can give a concrete example here. There's a company I really like called Pseudorite and Pseudorite founded by James Yu and they're building an editor experience for fiction writers that helps them.

25:43So as they're writing their stories or novels, there's a sidebar and you can highlight text and you can say like, help me come up with a different way of saying this or in a more evocative way. There's many different features built in and they had built in early on analytics around does the user accept a suggestion? Do they refresh and regenerate multiple times? How much do they edit the suggestion before including it in their story? Do they then share that? And all of those implicit signals correlate really well with the quality of the model or the prompt. And they were like running experiments all the time to make these better.

26:15And you could just see it in their traction figures. As they figured out the right prompts, the right features, the things that people were actually including, the product became much more loved by their users. Was there a third? You said there was two types of feedback. And a third one is corrections. So this helps particularly when you want to do fine-tuning later on. So anywhere you're exposing generated text to a user and they can edit it before using it, then that's worth logging. So a concrete example here is we have a couple of customers who do sales email generation, and they generate a draft, someone edits it, and then they send the draft.

26:46And so they capture the edited drafts. And I think a lot of this is sort of preemptive, right? They don't necessarily use that captured data immediately, but it's there if they want it for fine-tuning, for validating prompt changes and anything like that. Exactly. It's data that you want to have and you want to have in an accessible way such that you can improve things over time. Yeah. And you have a UI to expose it, but do you think that people use that UI or do they prefer to export it to, I don't know, Excel? Or how do people like to consume their data once they've captured it? Yeah, so we see a lot of people using it in the UI.

27:25And part of the reason for that is we have this bidirectional experience with an interactive playground. So we have the ability to take the data that was logged in production and open it back up in an environment where you can rerun the models when you make changes. And that ability has been really important for people to reason about counterfactuals. Oh, the model failed here. If the context retrieval had worked correctly, would the model have succeeded? And they can immediately run that counterfactual. Or is it a problem with GPT 3.5 versus 4? So they'll run it with four and see, you know, does that fix it?

27:56And that lets them build up an intuition about why things have worked or haven't worked. People do export data sometimes. So we allow people to format the data in the right way for fine-tuning and then export it. And that's something we see people do quite a lot if they want to fine-tune their own models. But we try to give fairly powerful data exploration tools within HumanLoop. Yeah. What about your integrations with the rest of the ecosystem? On your landing page, you have Langchain, AutoGPT is mentioned, Chroma, Pinecone, Snowflake, and obviously the LLM providers. Yeah, so the way we see human loop is sitting between the base LLM providers and an orchestration framework like code, you know, LangChain or LamaIndex might sit sort of separately to that.

28:36You know, you have this analogy, I think, of like LLM first or code first AI applications. And we're very strongly of the opinion that like most things should be happening in code, right? That developers want to write code. They want to be able to orchestrate these things in code. But for the pieces that require LLMs, you do need separate tooling. You need the right tools for prompt engineering. You need some way to evaluate that. And so we want HumanLoop to plug in very nicely into all of these orchestration frameworks that you might be using or your own code and let you collect the prompts, the evaluation data that you need to iterate quickly kind of in a nice UI.

29:09So here is where Lionchain collides with you. Has started to now. Yes, because they just released the prompt manager. Yeah. And they also have a dashboard to observe and track and store their prompts and data and the results. They don't have feedback collection yet, but they're going to build it. I'm sure they will. You know, it's a very vibrant ecosystem. There's lots of people running after similar problems and listening to developers and building what they need. So I'm not completely surprised that they've ended up building some of the features that we have because I think so much of what we need is really important for developers to achieve stuff.

29:46You know, I think one of the strongest parts of it is it's going to be very tightly integrated with Langchain, but a lot of people are not building on Langchain. And so for anyone for whom Langchain is not their production choice system, then I think actually it's going to be friction to work in that way. I think that there's going to be a plethora of different options for developers out there, and they'll find their own niches slightly. I think we're focused a little bit more, as I said, on companies where collaboration is very important, a little bit larger scale, and slightly less so far as individual developers in quite the same way that Langchain has been to date.

30:19That's a fair characterization, I think. It's funny because, yeah, you are more agnostic than Langchain is, and that is a strength of yours, but I've also worked for companies which have tried too hard to be Switzerland and to not be opinionated about anything, and it's bitten them in. You have to have opinions, right? Yes. You've got to bake into the... We learn a lot from our customers, and then we try to productize those learnings. So I gave you a concrete example earlier, right, on having good defaults for what types of feedback you can collect. And that's not an accident. Like, we're very opinionated about that because we've seen what's worked for the people who are getting to good results.

30:57And now if you sort of set up Humilub with that, you naturally end up with the correct defaults. And there's loads of examples of that throughout the product where we're feeding back learnings from having a very large range of customers in production to try and set up sensible defaults that you don't realize it, but we're nudging you towards doing the right thing. Yeah. Yeah. Excellent. So that's a really great overview of the product's surface area. I mean, I don't know if we left out anything that you want to highlight. No, I think that's great. And the focus for us, I think, being like a really excellent tool for prompt management, engineering, versioning, and also evaluation.

31:31So kind of combining those and making that easy for a team. Yeah. What's your estimate of the TAM now? Oh, God. I mean, eventually, at the current rate of growth, right, I think it's really difficult to— All known atoms in the universe. Yeah, it's difficult to put a size on it because how big it's going to be. Like, certainly, like, more than large enough for a venture-backable outcome. Today, I don't know, Datadog is something like a$35 billion company doing, like, web monitoring or whatever. I think LLMs and AI are going to be bigger than software. Yeah. And that market is going to be absolutely enormous.

32:04And so trying to put a size on the TAM feels a little silly almost. I know. You had to do it for your exercise, so I just figured I'd get an update. But it was a different world back then, right? And at the time that I was doing it, trying to get people to take the idea of putting GPT-3 in production seriously was work. And most people didn't believe it was the future. It was like it's difficult to believe this because it's only been a year. And I think everyone has kind of rewritten history. But I can tell you because I was trying to do it that a year ago it was still contrarian to say that large language models were going to be the default way that people were building things.

32:35Yeah. Well done for being early on it and convicted enough to build a leading company doing that. I think that's commendable. And I wish I was earlier. You've still been pretty early. You've done all right. I do think I do have this message because I think I talk to a lot of people who feel like they've missed it. But it's just beginning. It's still so early. What would you point to to encourage people who feel like they've missed the boom? I just think that I guess like a question to ask yourself if you miss ChatGPT. Yes. Was why did you miss it? And the people who didn't miss it, and I'm not necessarily including us in this.

33:13I think we were relatively late, even though we were earlier than most. Like, what did the people who get it right really grok, or what do they believe, right? What did Ilya Suskova or Shane Legg, the people who kind of saw this early? And I think it was a conviction about deep learning and scale and projecting forwards that, okay, if we just project forwards the current improvements from deep learning and assume they continue, like, what will the world look like? And if you do that today, and obviously it's extrapolating, right? That's not a theory-based prediction. It's just an extrapolation.

33:46But the extrapolation has been right for a really long time, so we should take it seriously. If you extrapolate that forward just a year or two, then you find that you would expect the models to be phenomenally better than they are today. And they're already at a scale where you expect large economic disruption, right? Even if GPT-4 doesn't get better, and if all we get is GPT-Vision plus the current model, we know that there's loads of useful applications to be built. People are doing it right now, but they're going to get better, right? This is the worst they're ever going to be. So if this is what's possible today, I think the hardest challenge actually is to take seriously the fact that in the not too distant future, you will have models even more capable than the ones we have now.

34:25How do you build for that world? I think it's a difficult thing to do, but it's certainly extremely early. Yeah, I think the quote that resonated with me this past week was Nat Friedman saying, imagine everything that we have now with six orders of magnitude, more compute by the end of the decade and plan for that. Yeah, and that seems to me like a... Six orders is a lot. Six orders seems optimistic, but I think it's a good mental exercise, right? Even if it turned out only to be, if it was only four orders or only three orders, right, it would still be transformative. If GPT-4, instead of costing$40 million or GPT, whatever it costs, tens of millions of dollars, became tens of thousands of dollars.

35:03I've heard a total all-in cost$500 million. So let's say it was$500 million today and it became$1 million or$2 million. That becomes accessible to even startups, let alone medium-sized companies. And I think we should assume something like that will happen. I would say even without significant research breakthroughs on the modeling side, I would just expect inference costs to become a lot cheaper. So training is difficult to optimize from a research perspective, but figuring out how to quantize models, how to make hardware more efficient, that to me feels like you chip away at it and it'll just happen naturally.

35:33And we're already seeing signs of that. So I would expect inference to get phenomenally cheaper, which is most of the cost. Yeah. And a previous guest that we had on by the time this comes out is Chris Latner, who is working on compilation for Python that's going to make inference a lot cheaper because it's going to fully saturate the actual compute that we already have. Yeah. So I think it's an easy prediction to make that inference costs come down phenomenally. Fantastic. In my mind, you went upmarket faster than most startups that I talked to. So you started selling to Enterprise NSE. You have Duolingo Max and Gusto AI as case studies.

36:08You have Trust Report. You don't need SOC 2. We're in the process of SOC 2. So we have SOC 2 Part 1, and we're currently being audited for SOC 2 Part 2. But you have the Vanta thing up. We have the Vanta thing up, and we have the part one. We have the trust report. We have regular pen tests. We have to do a lot of this stuff in order to get to procurement. To sell the enterprise. So, I mean, I love the Vanta story. It's not AI. But do you think that the Vanta trust report is going to work? In what sense? As a Sock2 replacement. A Sock2 proxy. I don't know, honestly. All I can say is that customers still care that we have Sock2, and we're still having to go through it.

36:46Vanta, even with Sok2, though, Vanta makes the process of doing it phenomenally easier. Okay, that's a big endorsement. So I would endorse the product. I've been less close to it than my co-founder, Peter, and a couple of others. Oh, yeah, there's always a Sok2 implementation person. Yeah. And that poor person is like, for a year, they're dealing with this. But it's certainly been a lot faster because of that. But just more broadly, like becoming an enterprise-oriented company, what if you had to change or learn? Yeah, so I would actually say that like we've only done it because we were feeling the pull, right?

37:22I wouldn't recommend doing it early if you can avoid it because you do have to do all these things, SOC 2 compliance. And I think Peter is filling out a very long InfoSec questionnaire today, right? And although you have most of the questions prepared, each one is just a little bit different. So there is just this overhead on each time. No comment. And, but the potential gain for some of these larger companies, right, if they can make efficiency improvements of one, two, four, 5 % is so much bigger. And the efficiency improvements probably aren't 5%. They're probably 20%, 30%. And so when the upside is so large, you know, if you are a, if you are a large company that's, you know, your costs are dominated, say, by customer support or something like this, then the idea that you might be able to dramatically improve that, or if you can make your developers much more efficient, There's just no shortage of things.

38:13And I think a lot of companies in the build versus buy decision, they want to do both because they want to have the capacity internally to be able to build AI features and services as part of their product as well. So they don't want to buy everything. Certain things, it makes a lot of sense. It's fully packaged. No one's building their own IDE. They're going to use Copilot or whatever is the equivalent. But they want to be able to add. I think the first AI feature that Gusto added was the ability within their application for people who are creating job ads could put in a very short description and it would auto-generate the first draft job ad and was smart enough to know that there are different legal requirements and what information has to be there for different states.

38:52So in certain states, you have to, for example, report the salary range. And in certain states, you don't. It's pretty easy to give that information to GPT-4 and have it generate you a sensible draft. But that was, I think, something that they got to production, you know, within weeks of starting. And just to see such a large company go from zero to having AI features in production, and now they're adding more and more, it's been quite phenomenal. Yeah, the speed of iteration is unlike enterprise, which is fantastic. I think a lot of people see a potential there. I think people's main concern with having someone like HumanLoop in the loop is the data and privacy element, right?

39:31Do people want on-prem human loop? So we do do VPC deployments where they're needed. We don't do full on-premise. So far, most people, we've been able to persuade that they don't need it. So whenever someone says we need VPC, the first question I always ask is why? And then we go through what are the real reasons, like what are they concerned about? And we see whether we can find ways either contractually or in our own cloud to satisfy those requirements. There are exceptions. Like we work now with some financially regulated companies. Amex GPT is one of our customers. Sorry, I should specify.

40:04I heard GPT. Yeah, no, Amex GPT is their global business travel arm. And they've got very sensitive information. And so they're particularly concerned about it. And there's more auditing. But for the people who are not financially regulated, usually we can persuade them that, look, we have SOC 2 or essentially there. We've got regular pen tests. We follow really high security standards. Most people so far have been accepting of that. Yeah. Have you ever attempted to classify the use cases that you're seeing? You see the whole universe, and you're not super opinionated about them. But there's summarization, there's classification.

40:42Okay, so interesting. I've certainly not tried to classify them at that granularity. Like, is it summarization or question answering? I often think more about the end use case. So, like, is this an ed tech use case? That's the vertical to me. I think a little bit more about it like that. in terms of use cases, it's really varied, right? There are people using models as completion, there's chat. Like it wouldn't be so obvious to know without doing some like GPT level analysis on it, like getting GPT to look at the outputs and inputs. Which you can do. Which we can do, whether they are doing summarization or something similar.

41:19But I would say, I feel like most use cases blend. Like that to me feels like an old school NLP way of viewing the world. Like an old school NLP, we used to break down these tasks into like summarization and NER and extraction and QA and then pipeline things together. And actually, I feel like that doesn't map very well onto how people are using GPP for today because they're using them as general purpose models. And so it is one model that's doing NER and it's doing extraction, it's doing summarization, it's doing classification, and it's often in one end-to-end sort of system. I think that's what people want to believe, that they're using them as general purpose models.

41:54But actually, when you open up the covers and look at the volume, 80 % of it is some really dumb use case that you could have. Like question answering our documents or something like that. Yeah. I'm trying to get some insight from there. I don't. Yeah. So I can tell you the trajectory we've seen, right? So really early on, the like killer use case was some form of writing assistant, whether it was like a marketing writing assistant. The Jaspers. Right. Jasper, Copy AI. We had like seven of them at one time, right? And then you had like specialist writing assistants. Some I think have gone on to be really successful products like Sudowrite or Type AI is another one.

42:27But they're still fundamentally like helping people write better. And then I think increasingly we've seen more diversification. There was a wave of chat to documents in one form or another. Chat PDF still doing well. Chat PDF doing super well. Once RAG started working, like retrieval augmented generation, there was that. But since then, as people are more problem driven and they're like trying to say, okay, how can we use this? We see a much broader range. So even within, like take Duolingo as an example. They've got Duolingo Max. So that's like a conversational experience. but they're also using large language models within the evaluation of that.

43:00They're also using it for content creation. And each of these companies sort of you start with one use case, and I feel like it expands because you just discover more and more things you can do with the models. Yeah, yeah, yeah. Do you see much code generation? Yes. So I would say that like developer-focused tools, I would say like EdTech and developer-focused tools are like probably two of the biggest areas that we see people working on. Yeah. I'm always wondering, because code generation is so structured, that you might have some special affordances for that. But again, that's anti the bitter lesson.

43:33I always wonder what we can optimize for. But that's my optimization brain when I should not. I should just scale things up. I think there's merit in both. Yes. Okay. So today, by the time we release this, you will have announced your new pricing. Yeah, that's right. So one thing that people have said to us a lot, actually, is that the barrier of entry to getting started with HumanLoop is just quite high. There isn't, you know, you can't just install an open source package and just get going or whatever it might be. And there's been quite a few small companies that have signed up and then sent us messages, you know, we're a not-for-profit or an early stage company.

44:07We really want to use HumanLoop, but it's just prohibitively expensive for now. We wouldn't mind paying in the future. And so we've thought really hard about how can we make it, like lower the barriers to entry for people to try it out and get started and get value and have the amount they have to pay scale much more with the value they get so that they're only paying for things when they've got value from Humanloop. And so we will be launching a new set of pricing. There'll be a free tier so you can sign up, you can get going on the website, you can start building projects and you won't have to pay anything.

44:33And only once you get to a certain scale, you've got more than three people on the platform, you're logging a certain amount of data to us, then pricing kicks in and it scales with you. So as your volumes go up, that's the time when you'll start paying us more. So much more gradual than it is now. And you're tying some features to the tiers? A little bit, but mostly we're trying to give you just a sort of most of the product experience. On the free tier, I think there's one or two things you don't have, but you have almost everything. And then once you're off the free tier, you have everything.

45:03But you pay, the amount you pay kind of scales slightly differently. So you get volume discounts at scale. Awesome. And so this is where one of the hard questions is, right? Like, is there a graduation risk as people get very serious about logging? You brought up Datadog earlier, and for sure Datadog is looking at your market as much as you're looking at theirs. So how do you think about that of, like, ultimately at scale becomes a commodity, right, the logging? So I think that actually this is really different to that. So the more people use it, we find actually the stickier it becomes. It's almost the opposite, that as they get to scale.

45:37So you're right that the millionth feedback data point is worth a lot less than the thousandth feedback data point. But what continues to be really valuable is this infrastructure around the workflow of prompt management, engineering, fixing things. So we see, you know, what happens over time is people put more and more evaluations onto HumanLoop. They've got more people on their team, the product manager, and also three linguists and someone else who are opening up the data that's being logged through HumanLoop back into that interactive environment. They're rerunning things. They're plugging in other data sources.

46:07And so over time, actually, the raw logs, I agree with you, kind of become commoditized. But the tooling that's needed to be able to not just kind of collect the data, but make it useful and do something with it to improve your model, that's the bit that becomes more valuable, right? Once you have something working at scale, then improving it by a few percentage points is like very, very impactful. So a lot of our customers early on would say exactly this to us. Like, oh, we can just dump our logs to like an S3 bucket or we can plug it, you know. And then like why do we need a special purpose tool?

46:38And most of them come back to us later because what they find is, oh, okay, I've logged something, but it's really difficult for me to match up the log to what model generated it and then quickly run that and try something else. Or I've logged something, and that log involved a retrieval, and I would like to know what went wrong with retrieval or which document the retrieval came from, and I didn't log that information correctly, et cetera, et cetera. And the complexity of setting this up well is quite high. So you can either spend a lot of time at that stage, two things happen. Either people roll their own solution.

47:10And early on, we saw a lot of people build their own solutions. Or they come and use something like us. And I think increasingly, because we've been working on this for now more than a year, the difference between something you would build yourself and sort of a bot solution is now quite enormous. And so I just wouldn't recommend it. And I guess the difference on the Datadog point or like other analytics tools, you mentioned Amplitude or Datadog. They're much more about passive monitoring. And I think one of the amazing things about AI is the interventions you can take can be very quick and very powerful.

47:39And so coupling very closely the ability to update a retrieval system or change a prompt to the analytics data and allowing you to run those experiments, I think is very powerful. Fantastic answer. It's almost like we prepped for this. It's also almost like I think about this a lot. If I didn't have an answer to that question, it would be difficult to justify spending all my time building this. But I do think it's very important. Company building, what have you changed your mind on as a founder? Ah, that's a great question. So one thing that comes to my mind as soon as you say company building is like a piece of advice that Michael Siebel has at YC, right?

48:10Which is like, don't do it, or at least don't do it pre-PMF, right? Like one of the biggest failure modes of early stage startups is, especially if they've raised investment from, you know, large investors, is that they persuade themselves that they have PMF too early and they go into sort of scaling mode and hiring people. And a lot of that stuff is important, but distracts from the most important thing that you have to do, which is understand the needs that are most pressing for your customer, figure out who the right customer is, and build what they really want. Or if they don't necessarily know what they want, build what they really need.

48:43So one thing that I believed and I still believe is that you want to do that at the right time, that company building too early is a distraction. And when was that for you? So for us, it was actually November, December last year. So November, December 2020. So we were a four-person company for almost two years. And it was only when everything was breaking, when all the charts were up and to the right, and we really could not service our customers anymore because the team was too small. That's when we started actively hiring people. And even then, we've been really slow and deliberate about it.

49:14Maybe a little bit too slow, given how much... There was a lot of suffering in being that slow. I wish we had a couple more people when things took off. There was a period of time, I'd say from November to March, where all of us were barely functioning because there was just so much to do. But we've continued to have the bar set really, really high and hire slowly and very deliberately. And I think we get more done with a smaller team of really, really excellent people than we would had we hired more people sooner. So that's something I kind of agreed on. The other thing that has maybe changed a little bit in my mind is related to how opinionated you should be.

49:52So I think you asked this question about opinionation in the product. And I think there's a risk of just listening to your customers and building what they want that can lead to hill climbing. And I think especially, and we were guilty of this, I think a little bit early on in the first year of Human Loop. Oh, you did it well, better than most. Thank you. But I think that, you know, where things started working for us was when we were, we had a lot more strength in our convictions, right? When we said, actually, you know, we believe GPT-3 is going to be the future of how people build this. And even if people don't believe that today.

50:24We're going to build for that future. That is hard to do. I still think we don't do it enough. Like I want us to do it even more. We have things we believe about the future that are somewhat contrarian and being able to plan for that and be opinionated and build for that future and also to be building the things that we believe our customers need, not exactly what they ask for. Because otherwise you end up, I think, with a lot of very undifferentiated products that are for everybody, so they're not for anyone, and they don't have a strong point of view. So I think, especially for building dev tools, I think you should have a point of view.

50:56Yes, I strongly agree with that. Hiring, what are you hiring for? And given that you're now hybrid, you're spending some time in SF, where are you hiring? Yeah, so we're hiring in both SF and London. The role that is most urgent for me right now, personally, is hiring for a developer relations engineer. So this is an engineer who loves community, loves documentation, loves doing talks, building demos. As part of launching this new pricing where we're going to have a free tier is also having a much bigger push towards helping individual developers and smaller teams succeed with human loop as well.

51:30And even developers in larger companies who just want to try it out before they're at scale. And I think to do that well requires a really good onboarding experience, really amazing documentation, and really good community building. And we need someone fully focused on that. I don't think it can be someone's part-time job. We want someone 100 % focused on building community. Ideally, we'd find someone as good as you, Swix, to do this job. So yeah, if you're a developer relations engineer, or even if you're just a product-focused engineer who is excited about AI and ML and has some track record of community building, then that's the role that I would love to hear about.

52:06And we'll be hiring for it primarily in San Francisco. Although if you are amazing elsewhere, we'll consider it. But SF being the focus. Yeah. Thanks for the compliment as well. But yes, I'd highly recommend people check out the job. It's already live on the website. A lot of people don't know. I have a third blog that is specifically for dev role advising because I do do some angel investing and people ask me for advice all the time. And I actually cache my frequently asked questions there. Anything else on the company side that I didn't touch on? If you're within YC, this will be boring. But if you're outside of YC, I think that you probably can't hear this enough times because I've seen so many people get this wrong, which is just like, before PMF, nothing other than PMF matters.

52:46And there's just, there's so many possible distractions as a startup founder or things you could be doing that sort of feel productive, but don't actually get you closer to your goal. Like trying to narrow focus to finding PMF. And what that means will be a little bit different for different startups and, you know, different experiences. I have friends who are doing deep tech, biotech startups or whatever, right? And so I don't think there's a one size fits all, but try not to do anything else. I think that advice has been really good for us. And it's often not intuitive. Does Humanloop have PMF right now?

53:19I think we have PMF within niches. So I think we definitely have like, especially for, I would say like, if you're a team building an LM application within a larger company, then like, yes, we see people sign up, they use the product, more people use it over time, usage goes up, they give us great feedback. There's always room for improvement. but we have a form of PMF. And I think there will be like multiple stages of it but we certainly found some PMF. What is the next tier of PMF that you're looking for? Well, I'm hoping it's on this evals project that we're launching, right? So we definitely have PMF on the current sort of prompt versioning management stuff.

53:55We've got about 10 companies currently in closed beta on evals giving us a lot of feedback on it. It's a real problem for them. We're seeing them get value from it but we haven't launched it publicly yet. I'm hoping that will be the next big one. Yeah. Just a technical question on the evals, which I don't know if it's too small, but typically evals involve writing code. Yeah. So it's like freeform, Python, JavaScript, something like that for you guys? Yeah. So it's the combination of, and again, we're iterating on this, but yeah, you can define them in Python and they can also call language models as well.

54:25And it's executed on your servers? Both are options. Okay. Interesting. So we have a protected environment. You can basically execute everything on our servers. Well done. which was not easy to build. And I'm not the right person to talk about it, but I think there's a really interesting engineering blog and how you can make it safe for other people to exec code on your service. But also it's going to be set up such that you can also run things on yours and just have the output logs still go to human loop and use for it. This is the promise of the edge clouds of the world. The Deno's, the Cloudflare workers, the modals.

54:56I don't know if you've explored any of those, but then you would not have to set it up yourself, essentially. I'm pretty sure they've all been explored by my team in recent months. Yeah, yeah. All right, excellent. Okay, brought it down to market takes. Yeah. Just, you know, brought it up for HumanLoop as a whole. How do you feel about LLM ops or prompt ops as a category term? LLM ops, I would drop one L firstly. I think we call them large language models today, but the goalpost of large is going to keep moving. So I think the point is sort of foundation models or... Oh, I have a proposal to deal with that.

55:24Oh, yeah? I have t-shirt sizing. So I've defined S, XS, and then M and L all the way to XXL. You're just going to have to keep updating that over time. But I think foundation model ops is maybe a better term because I also think that like within six months, we're going to have images and then people won't call them just language models anymore. Yeah. And is it worth a separate category than MLOps? But I do think it's worth a separate category. I think that the people from its four are different. We discussed this a little bit earlier, right? But a machine learning engineer and a traditional software engineer are very different people.

55:53They have different levels of knowledge and different goals. I also think that the generality of the models has changed what people are building. And so the problems they face are really different. It's, you know, like what you need for building a recommender, a small recommender system at enormous scale is very different from what you need to build a generative AI application that's very subjective. And so I do think that they have, I actually think like we've seen a lot of MLOps companies recently try to pivot into solving problems in this space. And I think it's going to be hard for them because they're changing who they're building for.

56:25So they now have to straddle two different sort of ideal customer profiles. and they also have a lot of legacy infrastructure focused around models whose output was like a measurable, quantifiable number. It was F1 or it was accuracy or something like this. And I think their lives are going to keep getting harder as the models go more general and go multimodal because what they've built so far just won't fit that world. So I think it probably can be done, but I think it's going to be very hard. You mentioned GPT-4 Vision and obviously there's more multimodal models coming along the way. How big does that factor into your planning because you're very language oriented right now?

56:59So it's increasingly like an internal conversation. Every time we have a product roadmap discussion, like planning for and starting to iterate on and when to build in support for vision has become very much front of mind. So I think like now, like we're working on it. Okay. One version of this, I posed this exact same question to Harrison, which is, let's say the GPT-4 Vision API drops tomorrow. What changes in human loop? Well, for one thing that you need just to be able, I mean, like very simple things, right? Like we need to be able to render and read in images and the playground environment that's interactive, right?

57:30So there's a bunch of just kind of follow your nose things that I think we'd have to figure out. But as I said, we've just started working on this. It's sort of become a product roadmap item. But not like we have to support it. Like it's very clear. This is not a question of if, it's a question of when. Okay. Yeah. Yeah. Excellent. Is prompt engineering dead? So we talked about this a little bit on the walk here. And I've never been a huge fan of the phrase prompt engineering because I think it simultaneously makes it not important enough and too important at the same time. I don't think it's a form of engineering in the way that software is a form of engineering, where it has this rich body of literature and theory and you have to learn about it and takes like very specialist skill.

58:09I think you can get good at it very quickly. But I do think that prompts are a very important part of LM or AI applications, right? Like natural language written instructions have become part of your source code and they have impacts on your product quality, they have impacts on the way your product behaves. So you should be treated with that level of seriousness as you would any other code artifact. So in that sense, I don't think it's dead. I think it's alive and well and becoming increasingly important. It's interesting. There's like, you know, Anthropic had that very well paid job prompt engineer.

58:39And I think they've hired a few prompt engineers now as well. And those people are leading on deployments in Anthropic and adding a lot of value. So there's clearly, it's clearly happening. But I think maybe it's slightly misnamed. I actually prefer your kind of AI engineer framing, where this is a different engineering skill set. You still need to be able to build product. You're still an engineer, but you have an intuition for how to get the best out of models, how to evaluate them. You understand the problems that come from stochasticity. And you also understand just the nuances. Like if you have a good mental model for how a large language model works, I think prompt engineering becomes a lot easier.

59:13And so having that skill set, I think is going to be important. But I doubt that five years from now, there will be like a separate job title of prompt engineer. Yeah. Yeah. I try to contrast it basically as prompt engineering is still 2022 and AI engineering is 2023. But yeah, the central thesis is you can't just get by with prompts. You have to write code to manage prompts, to generate prompts, and to generate code and to, for you to evaluate and run that code. Yeah, I think I'd agree with all of that. But to me, that doesn't diminish the importance of the prompts as an artifact. Yes, still important.

59:46Yes. I feel like when I saw Chain of Thought for the first time, I went from a world in which I was like, okay, models are not good at reasoning to models can do some reasoning. Yes. It was a sort of step change in my beliefs about the capabilities of these models. Yeah. And I still think that the LLM Cascades paper hasn't had the impact. Can you summarize that? So this was a paper from Google, and it's just sort of getting you to view LLMs as a way of doing inference in a probabilistic programming framework. So that's a lot of words. So let me try and sort of unpack that. And you have a PhD in this.

1:00:19But, you know, before AI was all LLMs, there was and there still is like a huge branch of research around probabilistic programs. So this is just ways of like writing code where probability and random variables are first class citizen. So you can have like random variables, and then there's lots of different operations you can do to condition and make predictions about them and do inferences around them. And this language modeling cascades paper basically said, hey, actually, like large language models are a really powerful inference engine that could be used as a composable piece inside something that looks like a probabilistic programming language.

1:00:55And we were chatting earlier today about the framework that will emerge for large language models. And I know you're working on small and you've given this a lot of thought. And, you know, Langchain and Llama Index and all these different groups, AutoGPT, are trying to circle around what's the right set of abstractions? How might we be able to compose LLMs in ways to write more complex programs? And I think that LLM Cascades paper was one of the first attempts to think about that in first principles and say, okay, what are the primitives you might want? And I think I'm surprised it hasn't been built on more.

1:01:25The very, very first AI grant from Nat Friedman mentioned that they were looking for a UI for Cascades and no one took them up on it. I don't think it needs a UI. I think it needs a... It's a framework. It's a framework. I think you want it in code. Yeah, yeah. And I would love to work on it if I had all the time in the world. It's sort of, you always have to choose your, you know, you can't do everything at once. Well, if someone is working on it, maybe reach out. I would love to chat to people about it who are working on it. Yeah. How many of your customers and users are actually worried about prompt injection and prompt security?

1:01:56Not enough. Really? So I would say almost zero. Yeah. And I think that's correct today because very few of our customers have action-taking LLMs. Yeah. And I think as long as your models are like read-only, prompt injection isn't that big a deal. Like it's not to me about like leaking your prompts or something because the prompts are only really valuable in the context of your code anyway. But I do think that once you get to the stage where you're letting the models have read-write access to any source of data, then prompt injection becomes a problem the same way any other form of code injection is a problem.

1:02:25But honestly, no one ever asked us about it. Right. Like almost never. Yeah. And I think that's because of the stage where people are at, right, which is that they're still trying to overcome hallucinations. and they're still trying to put guardrails in place around the behavior of the models. And very few people are using agents in production at meaningfully sized companies. But I think as soon as that becomes the case, if we do get to a stage where more people are allowing the models to read from a data source and write to a data source, then prompt injection will become something they care about.

1:02:53Yeah, and you guys will be well positioned to offer something. Absolutely. I think sort of being this layer between the raw model and the end application actually buys us a lot. in terms of what we can help with. Yeah. Well, you know, there are a bunch of security-minded people who are trying to offer that as a standalone thing, and it's a feature, not a product. I think I'd agree with that. OpenAI's fine-tuning rollout, which was last month, how does that affect HumanLoop? Yeah. So when we started the first version of HumanLoop, ChatGPT 3.5 wasn't out yet. It was all GPT-3, and we saw a lot of fine-tuning at the time.

1:03:27And post the release of 3.5 and 4, by virtue of the fact that it was impossible to fine-tune. Like we could just see in our analytics, the amount of fine-tuning just kind of fell off a cliff. Partly, I think, because the models were better. Yeah. But also just partly, like it wasn't an option. Yeah. And so I'm kind of interested to see now that 3.5 and 4 fine-tuning are back, whether that kind of fully recovers. 4 isn't back yet, but it's in a few. 3.5 fine-tuning being back. We've definitely seen a lot in the past, people generating outputs with GPT-4, filtering based off evaluation or feedback criteria, and then fine-tuning smaller, faster models.

1:04:04And so I think we likely see a lot of fine-tuning of GP3.5 on four-generated data, and that's a workflow that we natively support within HumanLoop now. So you can actually kind of do all of those things without having to leave it. If you have a bunch of generations, you can filter them on some criteria, click fine-tune, run ahead of evals, and then decide whether or not to deploy that model. But time will tell as to whether or not this is something that goes back up in importance the way it used to be. Yeah. The question that occurs to me, we talk about you being that layer that positions you very well.

1:04:37A lot of people are fighting to be that layer. And it occurs to me that as a user, potentially of Humanloop and your competitors, that I may not want to have to choose or be locked in. Is there room for an open standard that everyone agrees to that we all say like, okay, just adopt this one vendor neutral thing and then we all consume from it? Maybe. I think it could happen. We're not there yet. I think things are moving too fast for that to be the case, for people to have clarity on that. So maybe in the fullness of time there will be. My suspicion is that both will happen, right? That there will be some open standard that some people like to use.

1:05:23But once you come to working on serious production use cases, you often actually want the peace of mind of knowing that you're paying a real company that's going to be around to support you, that is focused on this, that has the knowledge and expertise. And so as we've seen in many other spaces, I suspect that there'll be a bit of both. A bit of both. Yeah, so the model I have in mind is Datadog versus the OpenTelemetry crew. And Datadog's doing fine and the OpenTelemetry crew is doing great as well. So last question on the market, did GPT-4 get dumber this year? I don't think so. We've seen a lot of conversation about this having happened.

1:06:00I think GPT-4 changed. I think that they are regularly updating it. And you certainly see that both in sort of people's attempts to, you know, papers being written about this and people were trying to do evaluations over time. I think that the main takeaway shouldn't be like, did GPT-4 get dumber, right? But the interesting question is like, did GPT-4 change? To which the answer I think is definitely yes. There's no question about that. And it's something that if you're a developer building products on top of GPT-4, it's something that you should think about a lot because you're building on a platform that will evolve and change over time.

1:06:31And you can pin the base model, but not forever. And so I think you need to, at the very least, have really good testing frameworks to be able to run regression tests and know, like, have things gotten worse over time, right? If you can't answer that for yourself, you're going to be scratching your head. Like, did we make the prompts worse? Did the retrieval system get worse? Did something else change? Did the user input distribution change? Or did the model get worse? And being able to disentangle those things easily, I think the importance of that's going to go up. But I also think that it should give us pause for thought about kind of the balance between what gets built on top of third-party providers and APIs in a closed world and what we might want to do more open source.

1:07:09And I suspect there'll be a mixture of both depending on the use case. But you are building on shifting sand whenever you're building on someone else's platform. Yeah, totally. And then one local specific question before we go to the takeaway questions. You went through IC. Yeah. And you are very, very familiar with the American tech scene. but also you built your company here in London. What should, and I'm very US focused, most of our audience is very US centric. What should people know about the European tech scene? Yes, I think that London's one of the best places in the world and Paris for AI focused folks.

1:07:45With the Hugging Face, I don't know. We've got Hugging Face in Paris. We're sitting right now. We're probably less than 200 meters from the offices of DeepMind. Facebook AI Research is here as well. UCL's AI Center is here, which is where, you know, Jeff Hinton was, and where a lot of great research is where DeepMind spun out of, actually. So Shane Legg and Demis met at UCL. So there's an amazing, and there's many more, you know, I can't list everything that's great, but there's many great AI institutions in the UK. What I would say is that I think that Europe has been amazing on research and continues to be like a fantastic place for researchers, but has been less good in my experience on productizing and trying to productize AI.

1:08:24And so the difference that I feel being here versus being in the US is just the number of, like if I go to San Francisco, the density of people who are trying to build useful things with large language models or with AI and butting their head up against it and discovering what works and what doesn't work and trying great ideas and trying stupid ideas and just learning together is much richer than what we have here. I think the pure like research labs, very competitive. Anthropics just opened an office here. When you're hiring for talent, you'll find as many or better people, like equal quality people in both places, but less so once you move towards productization.

1:09:04And I suspect it's also to do with the investor ecosystem. So we're sitting in the offices of LocalGlobe. LocalGlobe and Index were our first investors and they're both great. But the number of investors that you have of that quality in Europe is not the same as the US. And the type of people you interact with are very different. When I speak to VCs in the US, there's way more former founders. There's way more people who have done dev tools before. And there's way more support from the founders towards the ecosystem than there is in Europe. People are trying, but the culture is not quite the same.

1:09:33And that's why we're moving to SF, right? We want to be... Every time I've been to SF, good things have happened to people. Whether it's like bumping into you, or we get an introduction to an interesting investor or a customer, you know, or we just speak to someone who's been trying really hard to build something. And, you know, we share an office here in London with Bloop that does, you know, Bloop AI does sort of code search with LLMs. And we've tried our, you know, our very best to kind of aggregate a few other companies to us. And we're doing AI tinkerers, you know, tomorrow. So there is some of it here, but you have to work so much harder.

1:10:06Versus when an SF, you know, you can't move for hitting some AI thing. We had a Thursday recently with 10 AI meetups in one night. Yeah, it's almost too much. It is too much. I'll go there and say it is too much. You need some time to build things too. And there is, I would say, actually in the SF builder scene, privilege that comes out of just having so much opportunity thrown at you and that we have this arm's length, this taste for VC. And I'm like, no, they are partners in building your business. Oh, absolutely. So I think it's an interesting contrast. But as a person, I'm not American. I live most of my adult life in America.

1:10:41But I feel for non-US policymakers and VCs and people who care about their city who are like, okay, we're not SF. What do we do? I honestly think that it's – we think a lot about network effects and defensibility in startups. I think it's like the mother of all network effects. The reason I'm going is not because I love the city. I mean, SF is fine as a city. I like it. But I'm going because everyone else is going and everyone else is going because we're going. And once you've attracted a certain talent density, I think it's really hard to compete with that. Oh, boy. Okay. It is true. It's just the honest truth.

1:11:17Yeah. I do want to work on a path for non-tech hub cities because, I mean, that's where I'm from, right? Yeah, and me too as well, right? But I also think there's something to be said for the most driven, most ambitious people finding a way to get to where the center for their thing is. And like right now, today for like AI-focused products, I think it's San Francisco. But for different things, the center is, you know, different places. If you're a, you know, Hollywood is the place to go if you're an actor or whatever. And there are different hubs for different areas. It's a Paul Graham thing.

1:11:48Different cities breathe different ambitions into you. And in San Francisco, apparently it's power. It's not actually tech, it's power. Okay, interesting. And tech is a means to power. Interesting. There's a lesson in that for those of us who think about AGI safety. And also, you know, not anywhere in San Francisco, specific two square miles in San Francisco called the arena. You have to get in the arena and build. Okay, so broader takeaway questions. So we always ask three of all our guests. Acceleration, what has already happened in AI that you thought would take much longer? So this has been, since I started my PhD, like every year things have happened that I thought would take much longer.

1:12:25So when I started my PhD, it was at a time when like deep learning had just sort of started working. and transfer learning, even for like vision, hadn't been figured out yet. And people were talking about like, oh, it's going to, you know, how long before we can train models that don't need millions of annotated data examples? How long, you know, so AlphaGo was happening just at that point in time, the first version. I have made predictions and been wrong again and again and again. I've just been consistently too pessimistic. And I think I'm quite an optimistic person. You know, when would, you know, like Dota surprised me when it happened.

1:12:54The first like vision, first transfer learning, working in vision surprised me when it happened. the continued success of scale and deep learning. And then finally, although I believed that LMs were going to be enormous and I thought GPT-3 was going to be the future, just how good GPT-4 and ChatGPT turned out to be did surprise me. The first time I actually saw Claude before I saw ChatGPT, but the first time I saw Claude, I kept pushing the limits of it with tasks that I knew were kind of at the frontier of what was currently possible and just saw it blasting through these one after another, that was a mind-blowing moment for me.

1:13:28And I think it was for a lot of the rest of us. I think we're going to have a lot more of those. I think that's going to keep happening. Yeah, yeah. We are accelerating as we speak. Exploration. What do you think is the most interesting unsolved question in AI? I think there's actually some like obvious kind of elephant in the room unsolved problems that for some reason don't seem to get the amount of airtime that they kind of obviously should. So continual learning to me is one of these. Oh, God, yeah. Like we all walk around as if it's just completely normal that these models never learn anything new.

1:13:55Yeah, 2021 is when history ended. You just think, yeah, 2021 is when history ended. And you do retrieval augmentation with a vector database. And you're done, right? Why would the system keep learning after training? And I think everyone knows that this is a problem. But somehow it doesn't seem to me to get the amount of... I think this field in research is called continual learning or lifelong learning. And it doesn't seem to get the airtime that it used to. It seems to be like an obviously enormous problem. The other one that I think will happen naturally but just hasn't happened yet is just like more multimodality, right?

1:14:29Like it's kind of obvious that these models should be plugged in to vision, audio, speech, et cetera, and have shared representations because there's so much to be gained from that. And I think it's just like going to happen with time but hasn't happened yet. Yeah. Well, I think the cost is just token space, I guess. I don't know how much more you need to add every single modality. Although I think Facebook released like six modalities. We have some examples of this, right? So like Ghetto from DeepMind was a transformer model that they trained across. They just did policy distillation. So they trained a whole bunch of different RL agents.

1:15:02And they took the outputs of that, which is like observation, action, reward, triples, and trained a single transformer model on all of that. And then that one model could do any of those tasks. Actually, okay, whilst we're in exploration mode, there's a paper from DeepMind, came out at the same time as Ghetto, that I think is massively underrated. And I don't understand why it didn't get more attention. which it was at the same NeurIPS conference and I forget the exact title but I think it's called like In Context Reinforcement Learning or something like that. And they do something really similar to Gato.

1:15:31They take an RL agent, they train it and then they distill that into a transformer model. But what they do that's different is they don't take the trained RL agent. Instead, they take an untrained RL agent and they record the full trajectory of its learning. So early on in the data, the model's kind of crappy and by the end of the data, the model's been good at this task. and then they train a transformer model to predict that sequence. And in order to be good at predicting that sequence, you have to predict that the sub-agent, like the RL agent that generated the data, gets better at the task over time.

1:16:04And the only way that I can see to do that, and in fact this seems to be what the model is doing, is that you have to simulate a learning algorithm. The transformer has to simulate in-context reinforcement learning. And so they take all of these tasks, they train on the learning trajectories, and then they take a completely new task that that transformer model has never seen before. And it learns to do that task. And so it's learning from reward signals in context to achieve a new task. And to me, that's huge. It's a demonstration of inner optimization within a transformer model. And it's also a demonstration of in-context continuous learning that's limited only by the length of the context window.

1:16:41If the context window was really long, you could make this work practically. I don't really know why that wasn't a bigger deal. I don't know either. This sounds fantastic. Yeah, and Gato, I think the reason maybe it wasn't a bigger deal is it came out exactly the same time as Gato, and I think Gato just took all the attention. So we just got done talking a lot about focus, but given that you see potential in this, and this would be huge for literally training anything, would you be interested in exploring it at some point? As in trying to train it myself? Put this in production, some form of continuous learning.

1:17:12Obviously, that's on your radar, continuous learning. Yeah, I would love to, but I think you have to decide what kind of company you want to be. and this is something for OpenAI or Anthropic to focus on. I feel like you have to be thinking about the fundamentals of, this is the kind of research I used to do as a PhD student. So I'll put it this way, right? You have the research background to do this and you're choosing not to and you're building a company that doesn't use your research, specifically that part. I mean, you know. Reasonable question, but I think that I'm excited about getting things useful into people's hands very quickly.

1:17:52Like I like seeing, we talked about this earlier, right? We've moved from the research phase to the engineering phase of AI. It's the first time after having been in this field for maybe seven years where stuff goes beyond like just kind of a graph, right? Like the output of my work before would always be like, oh, look, there's a graph and like the number is better now. versus we actually get to see, you know, we have a customer between Duolingo and two or three of our other customers. We've got three or four customers working on like better versions of like teaching students, right? Tutors or language learning or whatever it might be.

1:18:24And to be able to make that incrementally better and accelerate the time it takes to get there, it just feels to be so much closer to it, to be on the engineering space right now. Whereas I think there's an alternative userverse in which I stayed in research and I went to an open AI or, you know, almost everyone from my research, PhD research group apart from Peter and now works at DeepMind. And I think I would have enjoyed that as well. But I really wanted to start a company that built something like useful and in production. And I don't even think those companies do that much right now, right?

1:18:53Like, it's only recently that OpenAI has sort of become a product company. They're more of a research company. They're building AGI. And I think that's true of the others. And I think that's amazing and fascinating. And if I had multiple lives, I would love to do that too. But at least right now, I want to be building products and putting them in people's hands. And it just feels a little bit far removed. Yeah, yeah, makes sense. And I think the world's better because you're actually coming at it with a full knowledge of what came before. Yeah, I do think it's a huge advantage. I do think like having a good conceptual understanding, like there's been a lot of people have pivoted into, as you said, LLM ops earlier.

1:19:28And I do think that actually knowing how it works, having a sense of what's going to come next and being able to project forwards and build for it is difficult to do if you don't have a good conceptual understanding of the machine learning. Yeah, yeah, yeah, agreed. Okay, well, I feel like this is a leading question, but what's one message you want everyone to take away today? Oh, wow, that's a great question. Really, if you're building, you know, if you're building a serious LMM application and you're trying to do, find the right prompts, optimize them, evaluate your models, then I really would encourage you to try out HumanLoop.

1:19:58Like, that's the use case that we really solve well for, especially if you're kind of having to collaborate with non-technical people, then Human Loop will probably solve a lot of pain for you. Yeah, excellent. Well, thanks so much for doing this. I had a real joy getting to know you and debugging real life issues with you. But that's the fun of Latent Space. So thank you so much. No, thanks for having me. It's been an absolute pleasure to get to spend some time with you, Sean.

1:20:22In this episode of the Latent Space podcast, we delved into the world of LLM ops and had a wide ranging conversation with Dr. Raza Habib, co-founder of Human Loop. We covered what is Human Loop? The three stages of prompt evals. The three types of human feedback. Human loop's new free tier and pricing. The competitive landscape and graduation risk of human loop. Prompt ops versus MLops. Prompt engineer versus AI engineer. Did GPT-4 get dumber? Europe's AI scene versus San Francisco. And don't sleep on Raza's in-depth explanations of LLM cascades and DeepMind's work on continuous learning. If you are interested in Human Loop, definitely check out their hiring page and new pricing and vote for them on the State of AI Engineering Survey.

1:21:07Thank you for tuning in to the Latent Space Podcast. Don't forget to like, subscribe, and tweet your takes at Latent Space Pod. Now go build.