AI Hype Cycle: How to Find Real and Practical vs. Shiny | OpenAI’s Louis Brandy

0:00We're always going to find ourselves attracted to some shiny thing at any given moment. Some of them are much more real than others. My big hot take on AI is that, like, it's going to get radically better at things you don't expect. Two years from now, it's going to be shockingly good at something you did not expect it to be that good at that quickly. But there will probably also always be ways that it's bad in ways that are surprising. Like, you'll look at it be amazing at this thing you thought it would do, but it's going to fail in some interesting way that's maybe unexpected. So it's always going to be this, like, lopsided creature that's getting better in various ways.

0:33I have a feeling there'll be a superhuman intelligence that can solve all our problems and we'll still be sitting around going, but yeah, but look how dumb it is at this thing. Are you looking to improve your engineering processes and align your efforts with business goals? Linear B has released the Essential Guide to Software Engineering Intelligence Platforms. This comprehensive guide will walk you through how SEI platforms provide visibility into your engineering operations, improve productivity, and forecast more accurately. Whether you're looking to adopt a new SEI platform or just want to enhance your current data practices, this guide covers everything you need, from evaluating platform capabilities to implementing solutions that drive continuous improvement.

1:13Head to the show notes to get your free copy of the Essential Guide to Software Engineering Intelligence Platforms today and take the first steps towards smarter, data-driven engineering. A quick note from the Dev Interrupted team. Since we recorded this episode, Rockset has been acquired by OpenAI, and Lewis is now member of OpenAI's technical staff. Congratulations to the whole team at Rockset on the acquisition. Hey, everyone. Welcome to Dev Interrupted. I'm your host, Dan Lyons, LinearBCOO. And today, I'm joined by Louis Brandy, VP of Engineering at Rockset. Welcome to Dev Interrupted, Louis.

1:50Thank you for having me. It's super exciting to be here. Yeah, really. I think we're going to have a great conversation today. I love the topics that we have in place. We have like AI hype cycle, things around AI, a little bit about your background and that type of stuff. What I see here is you joined Rockset. You spent more than 10 years at Meta. So before coming to Rockset, you were 10 years at Meta. And one of the things that you were working on and built, and I think even leading an engineering team, was around Meta's AI initiatives. Could you tell us a little bit about your background in Meta and then getting to Ruxet?

2:36Yeah, so I want to be careful here because AI initiative at Meta is a very loaded term. But I was doing AI and ML stuff early on and before it was cool. So we're going to talk about hype cycles. So this is kind of cool because it'll be, this is in the before time. So specifically for people who know a lot about like how AI has developed in the last 10 years, this is like the pre-deep learning time. There's still a lot of that stuff. Like there's still a lot of like classic machine learning and there's still a lot of use cases for that kind of stuff. But this is, I actually worked at most, mostly on this stuff doing a lot of like proto, I don't want to call it proto AI.

3:14I mean, it was still AI, but it was different than it is now. And we built a lot of really important systems. I mostly worked in the spam fighting and image classification kind of world. So this is actually quite relevant to what's happening right now, like right as we speak. But this is before everything broke loose, so to speak. So this is before Facebook had what is known as FAIR and a lot of the other extensive and major investments into AI. But yes, I was building all of that for a long time. We built a lot of cool systems in that era, many of which got replaced later by much bigger, more sophisticated AI as that research kept going.

3:48Yeah, that's really cool. I mean, tell us about the different stages, right? Because you were there kind of like, I don't know, you said like pre-stage a little bit early, early on. Then something shifted and now you see AI everywhere and everyone's talking about it. Can you tell us the difference between these two stages that we're in? And so like my history, I even got back before Facebook. I worked in face recognition and face detection. But I've always been on the implementer infra side more so than the research. So like my thing was always like how fast, how many frames a second can we make this thing run?

4:21How many dollars or watts does it need to run? Like that's kind of always been my home turf, like the infra side of it. But what's interesting in like how it's developed was like I kind of watched it all happen. A lot of the stuff that we were doing in that first few years at Facebook as well as before that, all that stuff sort of vanished in favor of this deep learning type of stuff where the idea was like you have these things that just scale really far with the amount of data. And a lot of the research before that, as I understand it, was like of the form you built better models to make use of the data you had.

5:02It turns out that's actually the wrong approach. There's so many situations where that's the right approach, but there's at least this one category of model that it's actually no, it's actually just more and more data. How far can this model model a huge enough data set, and then can you show enough data at it? That's roughly what happened. These really big places were able to do lots of training on lots of data, and they started to build things that were far more powerful than a really sophisticated model that was trained on less data could ever hope to accomplish. And that's really cool. I mean, I think that's what you're saying where we're seeing in like consumer application, chat GPT, everyone and their mother knows about AI now in some way, whether it's like a buzzword or maybe it's not a buzzword, but everyone is experiencing it.

5:49And I think that's kind of what you're saying. I mean, that was kind of like the big change with the big data sets and Google and all of that. What is your stance on the current AI hype cycle? Is it really hype? Is it not hype here to stay? What's your viewpoint? So I was doing, we did a conference not very long ago. And one of the questions I got asked was, is it all hype? And I love that question because it's easy. The answer is, it's definitely not all hype. And I was really thankful they used the word all. Because you asked the harder question, right? which is like, what do you think about it?

6:26I think, look, I think that there are always going to be hype cycles. We're always going to find ourselves attracted to some shiny thing at any given moment. Some of them are much more real than others. My big hot take on AI is that it's going to get radically better at things you don't expect. Two years from now, it's going to be shockingly good at something you did not expect it to be that good at that quickly. But there will probably also always be ways that it's bad in ways that are surprising. Like you'll look at it be amazing at this thing you thought it would do, but it's going to fail in some interesting way that's maybe unexpected.

6:58So it's always going to be this like lopsided creature that's getting better in various ways. I absolutely don't think it's all hype. I absolutely think there's an insane amount of hype too. And there's a lot of froth that you're going to have to like sort through and figure out what's real and what sticks. So in my current role, we're building infra for AI. So in some sense, I have a good horizontal view of all the different kinds of things people are trying. But even from where I sit, it's hard to tell what's real. You know, in other words, someone's spending a lot of money to build a thing.

7:29I don't know if they're making money yet. Like it's hard. That's going to take years to kind of figure out if. Yeah, monetize. Yeah, like so how much of this is just like fueled speculation versus like a real value, so to speak. but but then again on the flip side like i also get to see lots of really clever things that people are doing that like at least on the surface like seem promising and in some ways that's seductive like it's dangerous like the fact that it seems promising but maybe it isn't is like sort of the way you that's the that's a good way to lose like a lot of time and money to chase something um but so so no what's my current take on it i think it's going to change a lot i think it's going to also there's going to be false promises along the way there's gonna be things we think it can do that it won't and that it's going to this is like a classic moment of disruption where we're going to have to figure this out and we have to be careful to some degree yeah i really love how you summed it up the thing that you said that i like most is well a few things actually one thing is ai is maybe right now good at certain things that are a little unexpected or could be unexpected and other things that you may think AI like should just be able to do and it's like not so good at it and with your background you're kind of sitting I think at the infrastructure type area so you get to see all of these like trials and errors there's a lot of money going into it I don't know if it's like the dot-com boost because I didn't run a company uh in that like era, but maybe it kind of feels that way in the same thing of like, okay, let's try AI for everything and see what sticks.

9:09Let's put a bunch of money into it. The ones that are hit are going to be like gajillion dollar companies. And the ones that don't, it's like, whatever, I'm going to invest in like 20 of these and one of them will hit. But what I would ask you then is where do you think AI is doing well, like useful right now? And do you also have an example on the other side of like something that you think like oh it should be good at but it's actually like stinks at this and it's not gonna do well i mean we could do it in reverse i mean i yeah sorry answering in the reverse order because i actually think this was like we could like have a whole philosophical discussion here because like every single ai breakthrough that's ever happened it's really trivial for a human to sit down and be like look at this absolutely ridiculous thing that it's failing at like like you can do that with chat gbt right now you can go in there and actually did this recently where I was like, hey, give me the three biggest, most expensive software bugs in history.

10:03And it lists all three of them, like the three most expensive. And I was like, put them in order. And it didn't, it didn't put them in order. Like it had the right numbers. It just couldn't order them. And it's like, this is a really trivial mistake to make. I think it'll always be like that. It'll always be like that. Like it's making a mistake in some surprisingly trivial way because it doesn't think like we do. So something that's trivial to me and you is not going to be trivial to it. And I have a feeling that's going to, like, there'll be a superhuman intelligence that can solve all our problems and we'll still be sitting around going, but yeah, but look how dumb it is at this thing, right?

10:39Yeah. Something that is like so easy for all of us. Right. And you would say, oh, this like advanced intelligence, like AI, that should be nothing for it. Yeah. And it's tripping up. Exactly. And like, what I think that means practically is that you can't really take its expertise for granted. Like, like, like you could imagine a world in which a self-driving car is like provably better than humans or a, or a doctor is provably better at diagnosis than a doctor. However, it will make mistakes that any human will be like, no, whoa, whoa, whoa, whoa, whoa, whoa, whoa. That's crazy. And so it, what it means, I don't like, so I guess my, like the idea that like, in some sense, the combo maybe is it will always, at least for the foreseeable future is going to be very powerful.

11:25But I think that's going to happen everywhere. So in some sense, it's like horizontal. In other words, in every question, the AI is going to get really good, but also be really bad at some other way. So it's not like it's going to be good at these jobs and terrible at these jobs. It's like it's going to be really good at significant aspects of everything and really bad at significant aspects of everything. And it's actually hard to predict ahead of time how that's going to feel as things go. So you're describing it more, you said horizontal, right? And when you said that, what I started to think about it, like when I think about horizontal, does it mean kind of AI could be now useful for maybe every profession in some way, but not necessarily, but still make like the dumb mistakes also in every profession.

12:22so for example if i'm i mean the obvious things i think that ai is good as you could say something like write me an opening like speech or something or like do a template or like write my book report like all that that kind of stuff yeah i can collect all information do do really well but do you see it the same way that i'm seeing in that sense of like horizontal could kind of help everybody but kind of miss also for everybody? I think the most likely medium term, if it's going to end up being majorly disruptive, I think the most likely medium term version is like it disrupts a significant fraction of most of our jobs as opposed to it takes, it is the one reason why this is a whole job.

13:08Yeah. Yeah. It's one of the reasons why this is different maybe in a material sense from like previous automation kind of disruptions. Whereas like, hey, we built, I mean, we'll use a classic example. Like, hey, we built mechanical looms. The weaver profession is in big trouble. This is a little different than that because it's like in the sense of the disruption to everyone is actually hard to predict exactly how much of your job can become automated. And like certain parts of your job can just become a lot easier. And that gives you more time to do the other parts of your job. So exactly how that impacts things, it's super hard to predict.

13:42but just to be clear like I think all this is still a little bit this is like medium term I don't think in the short term we're like in we're in super grave danger of a lot of any of these like major disruptions but I think it's worth thinking about so that you whether you're depending on where you're at in this in this equation at least thinking about like if it comes to be right maybe maybe it won't maybe it will but if it does come to be to at least to put some intellectual effort into figuring out like how that's going to impact the different parts of our of our for jobs yeah well let's say i mean it's a great intro and an opening let's turn it into i mean you're an engineering leader right you're a vp of engineering that's been your i think your trade throughout your career that's what you've kind of like studied specialized in that type of thing but also you have a lot of background on infrastructure like you said for ai and is even like when you were at in the meta days like you said hey it's like wasn't ai it was like the precursor, but how should we think about this as engineering leaders for those listening on this pod where it's like AI is, it's almost like there's a pressure of like, hey, how are you using AI?

14:50What are you doing with it? How should we think about it as engineering leaders? I love this question. I have a question first you have to answer before I have an opinion, which is where are you in this space, right? So there's like, I think there's like a decision node right at the start, which is either I am in AI, like I'm betting some significant fraction of my business on this idea. And how did I get here? Like, should I even do that? Then there's like, you're not there. You're on the outside. As you said, there was going to be like a pressure, like a thing has just happened. Like I'm an insurance company.

15:23AI has just happened. Does that matter to me? Like, is there, am I missing something? Am I about to get sideswiped in a way I don't appreciate or understand? Like what's just happened? And I think these two situations are different, but there are some things I would say is the same. So it's funny because this whole intro, I gave you the pro, like, hey, I think there's something very real about this hype cycle you should think about. Then there's like the exact opposite. So like if you ask me the VP of engineering, I actually have a completely different opinion, which is like, I have like a small, we have a whole bunch of software engineers.

15:52I can assure you that if we said, hey, AI is super cool, go work on that. they would all like run full speed to go work on that like that's a super that's fun i'm an engineer i want to do fun shit yeah exactly so like and in that situation honestly the fun part like i don't care honestly if it works out well or not it's like i just want to be involved exactly exactly like so i have this like there's okay this is a this is a much more boring podcast but like in management 101 one of the things i always say is like there's like a two like any project you have like lives in this two-dimensional space like one one dimension is how shiny it is and one dimension is like how impactful it is and so like the the truth is shiny and impactful things are almost always never real like those are because if they were if there was something that was truly shiny and impactful we would have done it last half like we wouldn't have waited till now to do it it's already done in other words now it's not always true of course but typically that's true so that means that any given and by the way if something is not shiny and not impactful we're definitely not going to work on it like if nobody wants to do it and it and it has no impact.

16:54Like that's not going to exist. So everything kind of exists on this like horizontal front. And so like as an average rule, as an average rule, this is an average, the less shiny it is, the more impactful it is. That is the mathematical thing I just worked out. So like as a manager, you should be deeply distrustful of shiny things. Like, oh, no, no, shiny, shiny is scary because like you don't need me to pump us full of like enthusiasm for shiny. Like that's not like, right? Like that's not what, that's the easy part. right the hard part is to say like no no actually like the unit tests are actually that's the thing we should we should care about right now not like not not go solve all the ai problems in the world let's get our test coverage in order so we can right right and so that's so this is like the counterpoint from before which is like when you're in these kind of like frothy moments you don't want to miss it but you also don't want to get sucked away by it like you don't want to get like you don't want to like lose a half a team for half a year in some in the froth right of the cycle.

17:55Okay, so I think that gives us kind of maybe like a mental model of how to think about it. And then for you, I'll put you on the spot, for you personally, as an engineering leader, so you gave us a good mental model. Are there aspects with AI or maybe it's not even fully AI, but like things like co-pilot, Are there things that you're saying, you know what, I will spend effort to look in an experiment? Or no, I'm like, I'm not going to do it. I think step one is you have to make a value judgment on this cycle. You have to predict the future. We can choose if you'd like a previous hype cycle, which was blockchain.

18:46I think that was the most hype-y thing. Oh, I remember that one. Yeah, before AI. It may still be a thing. it's good to have two in your brain because even if you're very pro-AI, it's fairly easy to be skeptical of blockchain or maybe vice versa, depending on your environment. What's the one before that? I don't know. I'm old enough to remember like Web 2.0. Like the internet? Like the first one? Yeah, like the internet. There's probably, if we put some time in, we could probably think of some other. Like the computer? The personal? The PC? The PC. I mean, I guess before that was mobile. No one will ever use a PC.

19:20That's a business machine. Mobile was the hype cycle before that. Everything had to be mobile, right? Everything had to move onto the phones was maybe the hype cycle before that. You have to kind of make a value judgment. And for example, AI, I think it's very easy for me in my position to not be hyped up about blockchain. I build a database. That's my day job. So blockchain is not something that's going to be super impactful for me. I can make that call ahead of time. AI is quite different though because it actually has two aspects that are interesting because one isn't just my product. Can my product be made better?

19:51but also my developers, right? So you mentioned co-pub. So that's like a whole extra layer to this. So my rule, there's two ways to think about this, and this is the way I always do this. One is outside in, like market in. What does the market want from this product and is there a way that AI can help? And then one is bottoms up, right? And this is like, can we come up with cool ideas for the product using AI? And or can we use it internally to make ourselves more productive? And we actually, for the second one, we did both of these things. I mean, we did it in a fairly straightforward way, which was like time boxed sort of hack style things.

20:24Like we did a week and we said, everyone go nuts. We didn't actually limit it to AI. We didn't say, hey, do an AI hack for whatever. We just said, do it. But like AI was super frothy. So it was like, everyone go experiment and play with this. You can work whatever you want. But we did basically like a one week go nuts. And, you know, demo day is on Friday. So build something this week that you can show off by Friday. And what's cool about something like that is even if nothing comes to be, by the way, sometimes it does. Like there are very famous stories of stuff like that turning into entire billion-dollar ideas.

21:01Yeah, like hackathon into billion-dollar idea. But even if it doesn't, even if it doesn't, something very important happens, which is like you no longer are operating at this first level of AI understanding. You're like two or three levels deep. Like you'll see a demo and you'll say something like, man, that isn't going to work. But if it was better at A, B, and C, this suddenly gets interesting. And now you have a new understanding and now you can pay attention to A, B, and C and pay a lot more attention. And you've also, by the way, the manager brain, you've time boxed this. Like I'm not – like I dedicated a week or a month or whatever, right?

21:35But I'm not going to like – I'm not going to just like flush huge amounts of time. Yeah, you've capped your initial investment. Exactly. So you did – did you do it for like a week? We did. So this is part of our – we do this anyway. We do this periodically. We just have a week and we just do this. This is already a thing we've built into the way we work at Rockset. Lots of good things have come out of this. So again, we're building infra. So for us, it's a little bit easier when you're sort of building shovels rather than digging for gold. And a lot of our existing infrastructure did come from some of these, from early prototypes in this space.

22:12So, for example, the first version of vector search within Rockset was built during that week. And it was, again, vector search is a cool, it's its own super exciting, shiny problem. People love that problem. And so it was pretty easy to get people to go implement cool vector search and get it working properly on our infrastructure. And then all of a sudden it's like, well, this actually fits. This is actually a really good, you know, so that eventually became a full work stream. And it was originally birthed in, yeah, one of these hack weeks. I mean, even one takeaway is like Hack Week can produce something of value that you can, you know, so it's and then cap your investment.

22:54Learn. I like what you said, like, OK, now you're a few levels deeper and maybe you can have like a more educated opinion of how much more or not to invest like that, I think is a good takeaway. There's a lot of reasons to do this kind of a Hack Week thing. like it's just fun for morale like as a team building thing it's fun it's just fun in general but this one ulterior motive i think is actually really valuable which is like it's a really great way it's a really effective way to kind of do a bunch of things fail very safely and learn a lot from failure so to speak in a way that is like feels awesome it's a totally different thing to like set up a project in three months and kill it because it's not going well that doesn't feel good But having a hack week where you just build something crazy and you're like, oh, this is terrible.

23:42And then you erase it at the end. It feels great. And you learn a lot about the limitations. And so this is something I've always kind of, yeah, I mean, for AI in particular, I was actually quite interested in like, if I take our docs and I throw it into an LLM, like how good is that robot at answering questions about our docs? It's surprising. Going back to the very beginning, it's shockingly good but also weirdly bad in certain ways. But it's really visceral to experience that directly in a quick way. Are you using AI in any of your software development practices that are like baked in? Like into your workflows?

24:20Are you invested in it? We turned on Copilot, like in Copilot-like things. We have people who are using it and trying it. As of this moment right now, I cannot say that it has been wildly successful. yeah okay interesting you have to define wild success i mean i don't you know but i still so i don't think i would at the current moment i don't think i would um call it necessary like i wouldn't say you need to go do this right now or you're or you're behind um however i don't know how much longer that might be true like i think today it's not necessarily earth shattering you have to go do it right now, it's unclear to me how much longer that can be true, especially as we get into like these AI systems that start to use tools.

25:13Like they, in other words, they can write tests and run the tests. And like this, you get into this, like, I don't know how much, how deep we want to get to be getting these kind of like tool using agent type AI workflows. All of a sudden, it may not be insane to like have it say, write a test suite for something. I, you know, again, I don't really know. I would say as of today, we do not use, at least not like as a first-class citizen in our development environment today, no. Okay. Yeah, I mean, so that's awesome. Like, thank you for sharing because I think when I was saying like the pressure, I was saying, you know, some of the engineering leaders I speak with, there's almost like, hey, what are you doing with AI?

25:55What are you doing with Copilot? Is it everywhere? And a lot of what we, so at Linear B for our customers, we measure the impact. Did Copilot actually improve the developer experience like you thought it might or did it not? Or is it like somewhere, you know, in the middle? This stuff isn't free, so it costs money, right? So it's like, is your investment worth it? So we're doing a lot, like there's a lot of companies there, I would say, experimenting with the use of it and want to know the impact. And then the other thing that you said that is interesting is you do see some of this like you said it with tooling, right?

26:37I think that's the terminology. It was like different tooling. Like, okay, maybe like a PR, pull request review type AI or like I can analyze the code and say like here's like the summary of what this is or the description or what you should look out for. And I wouldn't say, and I think that is actually interesting. There are some companies using that. And the other thing I would say is like, I don't even know if I would call it AI, but bots. Let's say bots. Yeah. And the way that I define at least a bot in like the SDLC, like the workflow is, I see a lot of companies where 50 % of the PRs are not made by humans.

27:23Or like 25 % or 70%. Like it ranges. And when you start getting into that situation, then I've seen the situation where it's like the PR was not created by a human. And the first review was not done by a human. Okay, now we're into this like, that's interesting. How do we orchestrate this together? So let's go to your point. All of these, and tests are maybe lower impact, but the tests were not created by a human. And then like the review of the test was not created by a human. What do you do to orchestrate that? And I see the industry experimenting with that, let's say. I mean, so you get – again, it's hard to have this conversation without things rapidly getting philosophical.

28:12But like there is a spectrum, right? Like I mean at some point like having a linter that complains about your code having clear bugs in it is like a proto form of AI. Like it's not a huge jump, right? Like, and like, we use a lot of that stuff, right? Like I, you know, I've used Clang Tidy and stuff for years, right? To like clean up codes. And to be clear, when you work at the giant places like Facebook, like, there's robots everywhere. They're making changes. They're commenting on change sets. They get in fights sometimes. Like, you know, like one will yell at the other one for doing something.