Automating code optimization with LLMs

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:43Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm the founder of Prediction Guard. I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing well today, Daniel. There's so much going on these days in this industry in terms of AI that just constantly learning new stuff and finding out who's doing what. Yeah, it's almost like you need to optimize some things about your life to keep up. Would you say that's accurate? Yeah. Speaking of optimization, I think that's a thread to pull right there. Yeah, yeah. So speaking of today, we have with us Mike Basios, who is the CTO and co-founder at Turin Tech AI.

1:32Welcome, Mike. Hello, guys. Nice to meet you. Yeah, well, we alluded to optimization. And I know one of the things that Turin Tech is working on is code optimization with AI. And maybe some people, actually, probably a lot of people listening to this podcast are familiar with certain developer tools that are AI flavored, maybe GitHub Copilot or something like that for generation. I'm wondering if you could take a moment before we dive into AI-driven code optimization, if you could just help set the stage. For those that aren't aware, what do you mean when you say code optimization? Why is it useful?

2:14How has code optimization, quote unquote, been part of the developer lifecycle for some time? We are in an area nowadays that we see more and more applications consuming a lot of cloud resources and a lot of resources, and everybody is trying to optimize the performance of their code. Now, when we're talking about code optimization and in particular performance, typically people would like to optimize things like application being faster, maybe memory consumption. We know everybody complaining about Chrome using too much memory, for example, in the past. Or CPU usage, which is connected very much with the energy that different software is using.

3:03If we talk about mobile phones, application there, we would all like to be more efficient and consume less energy. And practically, that's the area that we have been focusing in my research and our group in our research and in the company. Maybe you could talk a little bit about some of the history of that research. I'm sure that that has, just like everything else, been impacted by this kind of latest wave of AI technologies and generative AI. But I know that the company and yourself have been involved in research prior to and all during the development of these things. So could you give us a little bit of background on how you first started thinking about these problems and how it kind of developed over time?

3:51Yeah, code optimization is not a new thing. If you read the research papers 20, 30 years ago, everybody would like to optimize and make the code efficient. There are a lot of tools like profilers that help developers find hotspots. The biggest problem in this area is, okay, we profile our code, but how we can automatically improve it so we make it faster. Typically, this is a very, very manual process. And majority of the people that work on this area are super specialized. And it's more and more difficult nowadays to find people that know how to optimize the performance of their code because the programming languages are becoming higher and higher level.

4:39Like now people write more in language like Python, JavaScript, TypeScript. So nowadays, the real companies that deal with code optimization are companies in the lower from the hardware space, like big Intel, NVIDIA, and those specialized software that take advantage of their hardware, whether they show the hardware outperforms, or technology companies that need the scalability. But majority of developers will not necessarily bother about the performance of the code as an immediate first thing that they need to optimize. And we have tried and we have a platform that we helped practical engineers to automate this kind of process to make it easier for developers to identify places in their code that are slow.

5:30and then to optimize it without necessarily having the knowledge that they need, but also do it automatically. Now, the history of code optimization, as I said, it was a very, very manual process at the very beginning. Very few people know how to optimize code for specific hardware. In the past, I guess, they had to read books and compiler options. Eventually, then you had better and better compilers. So you can use compilation option to optimize your code and tune the options of the compilers, et cetera. And then you have a lot of profiling tools that help developers also optimize. But still, all these processes, most of the time has been manual or semi-manual, right?

6:17And that's where we see the advances of AI helping this process. And to give you a bit of context, how we started our startup, this started after we published a paper in 2018, I think, that was in the Foundation of Software Engineering Conference, where we showed that we could automatically help developers choose better data structures by taking a code, looking at data structures, and optimize it by giving variations. For example, sometimes in languages like Java, for example, you don't need to use an array list if you potentially can use a link list in a scenario that may be better. So we try to do these small changes and showed that we have good performance impact.

7:03But majority of people at that point were manually translating code, like rule-based transformation, like regular expressions, if I see this pattern, convert it to this pattern. So that is how people have been doing those things with code refactoring tools until the LLM came into the discussion. Speaking of that, kind of getting into this latest wave of maybe what I'll just refer to as developers tools that are AI related. Again, you know, people might be familiar with code generation type of tools or explainers or something like that. There've also been, I've seen kind of agentic type of tools that will like write up PR for you to do like you say, hey, I want to do this thing.

7:53And there's a PR generated here, given the focus on code optimization, could Could you draw out some of, just for people that are maybe kind of getting into this, could you draw out some of when they would want to use this kind of tool versus some of these other maybe generative applications or applications of generative AI to code, I guess? Yeah. How does it fit into that ecosystem? So the way we present the code optimization tool currently is part of your CI-CD process. So you make a pull request, then you will run some unit tests, you will have your integration testing, potentially you'll have a security scanning tool like SNCC or check marks, etc.

8:37And then the next step is, depending on where your application is deployed, we analyze your code and we tell you make those changes because you can have this 20 % improvement in CPU and execution time, etc. That is the way we present it currently as a CI, CD tool in the tool chain of the developer tools. However, if you think about the technology underneath and all dev tools, the way I see it, are going to be using AI and they will take advantage of LLM-based solutions. From the moment you are using LLM, if you generate code or you translate code, it's a kind of the same kind of approach. It depends on the data that you will apply the LLM.

9:24So code generation tools, practically they see when you say for the PR example, the way those LLMs have been trained is that they see that there is some comments below the code. So you say, if I give you those comments, can you predict the code? So in the code translation, where you don't know about the speed of the code, you say, I have seen this C++ code and the equivalent of Python code. And LLMs do it. Like, for example, Copilot or ChatGPT, they can translate code. Like, if it's perfect, probably not yet. But that is a fundamental technology. So code optimization is on the same set of tools, but it says this is a slow code that I have seen.

10:08Now I have seen a variation of faster code. So I can recommend you this faster code. But eventually you can expose the LLM like any other tool that is built on VS Code or LLM based. You can expose it in an editor, and then the developers from our LLM will get suggestions for faster code. That doesn't mean that will be beautiful code necessarily, but it will be faster for the hardware that you need to run it. I'm kind of curious, as we're talking about speed, are there any other dimensions that are relevant in there that you guys are interested in that are either adjacent to speed or contribute to speed or any other characteristics that may not be directly speed specific, but are things that you're starting to target or expect to target?

10:59We apply multi-objective optimization. So when you do the translation, you can have different objectives that you try to optimize. Like speed is one factor. Memory usage is another. CPU usage. Typically, there is a trade-off between speed and memory usage. If you have more memory, you would like to use it, it increases speed. So our tool allows you and gives you different suggestions for what you need. But we see users, for example, we see this paradigm to be used in other use cases. Like users have told us, I can improve the readability of my code as far as I know that I don't impact performance, for example.

11:41because there are managers in teams that they have five projects and they would like, let's say, five projects with five teams with developers and they would like to guarantee that the quality across is good. So this AI approach, these tools, Artemis, our tool, or other tools in the LLM space will be able to help with those. But the biggest problem of any LLM-based tool currently, like Chris, you mentioned you are working for Lockheed Marketing, etc., is that the code generated by any LLM, it's not guaranteed that it will work 100%. Also, you need existing tools to check if this code is secure.

12:28And also, you do not know if you will break your code. So still in the early stages of incorporating those LLMs, that's why it's also easy for a developer to give it as a suggestion right now. But I'm pretty sure like companies like that working bug fixing, code security checks, they are using and they will be using more and more LLMs because they can train on the data that they have access so they can have competitive advantage. Would it be a good parallel just to kind of draw to people's mind, like maybe something they've seen before in other domains? It almost sounds like a parallel to kind of a rephrasing type of prompt in an LLM where you might say, like, I do this a lot with, you know, emails or other things like here's my really bad email, you know, make it flow better and sound better.

13:24Or here's my goofy email, make it business professional or something like that. you drew the comparison to like maybe machine translation or something like that. Is rephrasing kind of a good way to think about this? Let me give you a very simple example. Let's say you have an essay that you need to write and you have 10 paragraphs in that essay. And you would like to have a version of that essay that is much better and you can get a better grade. right now what we do is we will look all the paragraphs and we will provide you better variations so we will get you version one with three changes that are applied by different llms then you would need somebody to grade that essay so in this case we measure how fast the code is so somebody says okay you have a 70 percent then we take that output and then we provide and give it back to the LLM and give you another version.

14:25So practically, you start from version zero of your essay, you apply the different LLMs, you get feedback. So it's like reinforcement learning, live learning, and get different variations. And that score will be increasing eventually. And at the end, you have like a translated version of your original version that the LLMs and all those refactoring did, which is better on the metric that you have. But we in the platform test that the code passes, we compile it, and also we run the performance. In this scenario, we'd have a teacher that would grade. And if you think how OpenAI and everybody has been doing their training, they usually get LLMs and then they use reinforcement learning and HLF and all those techniques.

15:11So we have done that in the code optimization setting. That's why we have had some impressive results in taking an open source library and we just put in a tool and then suddenly optimize by 30 % execution time without us doing anything. The models learn themselves.

15:33This is a changelog news break. You can add Meta's CodeLlama to the ever-expanding list of code-generating LLMs. Based on LLAMA 2, CodeLlama comes in three sizes, 7 billion parameters, 13 billion, and 34 billion. And it comes in three different varieties, a general model, one tuned for NLP instructions and one fine tuned for Python. How does it stack up? Well, Meta claims it outperforms other publicly available LLMs and it shares the same open-ish license as Llama itself, which is free for research and commercial use unless you compete with Meta. You just heard one of our five top stories from Monday's Changelog News.

16:18Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com slash news.

16:39so mike you you got into something that i'm super interested in in terms of how you're going about this problem which you alluded to the fact that you're using this sort of reinforcement learning loop or feedback loop to improve the performance of your tools given that you and your team have worked with sort of code generation or code specific models for some time now. Before we get into kind of some of the cool stuff that you've done specifically, could you just comment on kind of the state of code generation models that are out there on maybe the open source side specifically, but if you want to highlight any, you know, closed source ones, that's perfectly fine too.

17:26But from your perspective, how is that the ecosystem of code generation models changing and advancing? And what is the state of it sort of these days, I guess? So I was one of the very first believers of LLM assistive code generation tools. Like I tried to get bad access to GitHub Copilot, even tried to use GPT-3 to see all this kind of technology. I'm a very big believer because I have seen members of our team using LLM to build things much, much faster to code. For example, we had the backend engineer. We needed a prototype for a frontend. He just used one of the closed source LLMs. And in one day, he did a new UI that he didn't even know the language.

18:18He didn't know TypeScript. So I see that they are very, very promising. I see more usage right now for good developers that already know the basic of computer science. I'm not a believer of, hey, you don't need to code. You will use this LLM for you. That are some videos, potentially they are overpromoting that. But if you are a good enough developer, you know what you need and you use, they can practically help you dramatically to build easy applications, to generate tests, to generate comments about your code. And of course, the performance of, from our experience, I believe GitHub, Copilot, ChatGPT are still outperforming the other models.

19:03But we see more and more open source models starting to become very, very good into the different languages. Like we have been trying Lama 2, we have been trying CodeGen, we have been trying all of this. and we even expose them to our platform so people can compare the results of those. Those tools will need to become a bit easier for developers and VS Code tools to use because by default, people, this API exposure that some of the closed source models are giving is solving a lot of headaches for a lot of developers. That's why a lot of developers still have preference for this. But definitely open source models, they are very good and I see them becoming even better if they are fine-tuned a specific language or a specific context.

19:56I'll give you an example if you want if we want to do translation from SQL to an SQL let's say we want to optimize SQL queries that people are doing in their databases you can take one of the open source models and fine-tune it on SQL and you probably will outperform GPT 3.5 or four on the context that you have. But I'm truly a very big believer. It's a bit of also a bit of psychology between developers that, hey, but I am a believer that people that will use, they have advantage over people that don't use those tools currently. So that raises an interesting point. It's a little bit of a tangent, but you've kind of inferred it.

20:36This is kind of changing the way we humans are coding. I know you also talked about whether LLMs would just write the code for us and the overhype, you know, certainly today about that. But it's kind of changing the way that we code as humans and it's extending our capabilities dramatically in terms of being able to reach beyond what we might have been able to do two years ago, for instance. Do you see that accelerating when you're looking at the fact that if you looked at a traditional coding team a few years ago as you're producing your product for folks and you're starting to recognize that individual coders are starting to elbow their way out of their traditional swim lanes with these new tooling capabilities that you're providing, how does that change things?

21:23How does the market change that you're looking at going forward? I believe dramatically and that we have tested this with our internal developers. I have seen dramatic improvement in productivity of developers without access to LLMs and developers that have, like, there are two things. One is you can make more efficient developers that everybody would like to hear. But from a more senior level, manager level, you do not need, unfortunately, as many developers as you would need before. You cannot avoid this. And developers should think for me, like, okay, I am competing with some other developer in my team to produce, let's say, an API or produce a UI.

22:08If the guy has access to Copilot or ChatGPT, the other person doesn't have, I guarantee you, like, with very, very big chances, the guy that has access to ChatGPT and those will outperform and have faster results. So it is like you have a very, very good assistant next to you that you should use. Otherwise, you're losing. That's how I see it. Yeah, as a two-second follow-up to that point you just made, when we start pervasively, because I think I've seen some stuff recently over the last month or two that the majority of active developers out there are now using LLMs. So there's been really rapid adoption here within the developer community.

22:47so we've kind of moved very quickly from those who had versus those who didn't have into a world where everybody has they may not all be using exactly the same models as things progress but everyone has it any thoughts about kind of what that means it's like you're democratizing llms across the population of developers and now they're competing so it's kind of like me and my llms are competing against you and your llms as a developer just any thoughts waxing poetic a little bit about what the implications there are? Yeah, I mean, it's a weird world that I don't know anymore. Like, I code less and less, but I now can code again.

23:27Because, for example, we have a data scientist in the team, and he says, I don't feel like I'm a coder anymore. I'm just a manager of, you know, a user of this LLM that I validate the output. And yeah, it's okay. And it's a bit ridiculous, he says, because even simple things like copy-paste, I will not bother. I'll just say, okay, can you refactor this? It definitely has changed. And okay, there may be implications about creativity of people. And if you go into the AI, LLM space, like for example, for images, when those models generate images, somebody, a painter may say to you, hey, you may lose creativity because you generate always the same thing.

24:09I don't have answers for those things. that I don't think a lot of people have answers. We'll just see, right? Like how things go sincerely. But yeah, it's interesting, right? Yeah, this idea of being a manager of your assistants, I think is really helpful. I forget who it was we had on the show, Chris, but they were saying, hey, if you think about this thing, just like a high school intern or something, you know, like is a high school intern going to solve all of your problems? No, but if they work all day on your problem or, you know, let's say you have infinite number of those those high school interns that just can do work all the time.

24:51Is that useful? Certainly. There's a management aspect to that. Right. Probably more so with high school interns than with LLMs. I'm not sure. But but yeah, I love that metaphor. That's really good. I am also wondering, so this particular application of AI within someone's code base, right? I think similar to what we've seen in other cases with Copilot and some of the things that have happened there, it's a very sensitive area, particularly for like enterprise business users. I think if you're like an indie hacker and like you say, you are wanting to create a TypeScript UI and you don't know TypeScript, like boom, you can get, you know, some really cool results really quickly.

25:39Of course, enterprise code is part of the IP of a company. There's, you know, two aspects of that, one of which is the fact that companies have been hesitant or even sued others over usage of their code or data in ways that they didn't expect. I think, though, the other aspect of this that you alluded to is it really is powerful when you start to bring your own data to the table, especially with these open models, both because they have kind of privacy conserving deployments. And there's also like code preference things and other things that your company might have. So I'm wondering if you could speak to that a little bit and how you envision you're helping build a product that's doing code optimizations for people.

26:27So how do you think about, you know, people creating customized models for their code bases and the sort of proliferation of these customer specific models and the hosting of those? What goes through your mind when you're thinking about those things? That's a very, very important topic. And we have quite a lot of experience with this. And I will mention an example where we went to a client, very big technology firm, super big, one of the best. And we said, hey, this is our platform. We have these LLMs. You can use any LLM of your choice, like GPT-4 or open source LLMs, like LAMA-2, et cetera. In the beginning, they said we don't have any approval for OpenAI because, first, they don't know the IP issue.

27:13Second, they don't want the code to go outside. We are talking about proprietary code at a lot of such companies. So the solution there was, okay, you can use the custom open source LLM on your data. We do not see anything. So it's a custom solution on-premise. And while they're using a product, then practically our platform allows them to generate their own training data set. So, for example, they use Artemis. They optimize code. They see sometimes the code is optimized. It's not optimized. but they generate the data and those data we cannot see and the client should have those data for fine-tuning their own model then through the platform they can say fine further fine-tune their own model that is how the industry will go especially in financial sectors and technology sector or you know defense companies they will never give their code outside on an open source level.

28:11But LLMs are super powerful on that. Like with one client that they said, I'm not sure if I can get a recommendation. If you give me a recommendation from an LLM, who is liable if that code doesn't work, right? What is the IP issue? I said, how do you solve it now? They have the IP checkers, et cetera. I say, you can just use the IP checker for this moment, the same code, the same process that you would do, right? But also LLMs, and we have added this functionality, can do very good similarity search. So if you have other code bases and similar functions in your code base, you can fine-tune nicely, or you can look like the same way people are building chatbots on your documents.

28:56You can build your own chatbot on your code, so it's practically similarity search. And then we could recommend, We even identified that three teams had implemented the same functionality a bit different, right? So you even save time, right? So still, you know, this technology, the underneath technology, if you know how to use it properly, you can take advantage of it. And the biggest example is Databricks about Mosaic ML, because Mosaic ML, like they acquired Mosaic ML, that helps organizations fine-tune their own LLMs of their data. Because no organization will give their data to OpenAI or any other company to fine-tune on top of them, especially big organizations that there is value on their data.

29:42So the same thing applies for coding. That's how we see, and that's why we adapted the product accordingly. My next question is, I guess, sort of selfish. And I like to ask this of people that have, you know, really built impressive things with this kind of new reasoning layer of LLMs. I'm wondering, as you look back on building this product for code optimization with LLMs, are there any challenges that were unexpected that you had to overcome? And are there any sort of takeaways that you would give to practitioners that are maybe working on their own products or integrations with LLMs? What has been important for you to stress, especially as a CTO and bringing new people into the team, as you're working with these types of models, what's important in your mind and any of those challenges that had come up, anything you'd like to highlight?