GPT Engineer Generates An Entire Codebase Based On A Prompt

0:00Today on the AI Breakdown, we're looking at GPT-Engineer, a new project that promises to generate entire codebases from just a prompt. Before that on the brief, the Grammys announces AI rules, LLMs outperform humans on data labeling, and much more. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Like, subscribe, and share, and go to breakdown.network for more information. Welcome back to the AI Breakdown Brief, all the AI headline news that you need in five minutes or less. We start today with a study that I think will surprise no one, but is still hugely important.

0:35Refuel is an AI company that just raised around$5 million, whose focus is on cleaning and labeling data for all sorts of different types of business applications. Now, they recently did a study across a wide variety of data sets in which they found that LLMs are able to label data as well as humans in terms of accuracy, but for much cheaper and much faster. In fact, they say LLMs can label text data sets 20 times faster and 7 times cheaper than human counterparts. Now, when it comes to accuracy, GPT-4 performed best out of the box, with 88.4 % agreement with ground truth as compared to 86 % for skilled human annotators, so actually slightly better than human.

1:11The types of data sets that were part of this study included online banking queries, USSEC filings, toxicity detection in public user comments, product data from Walmart and Amazon, company descriptions from Wikipedia, science exam questions, and more. So again, a really wide-ranging set of data. OpenAI, Anthropic, Hugging Face, and Google all had LLMs that were considered as part of the study. Now, while GPT-4 was the most performant of these models, numerous other models also achieved strong performance of above 80 % agreement with ground truth at one-tenth the cost of the GPT-4 API. I don't think this is all that surprising, but it is a great example of the type of rote work that is likely to be automated away.

1:50For people who are paying attention to this, some will see in this the destruction of an entire category of jobs, but others will see a freeing up of human capacity for other types of higher-order thinking. Next up in news that surprises no one, 92 % of developers, according to a new GitHub survey, are using AI in the workplace. Not only does this cohort represent early adopters in general, it's also one of the areas where there has been the fastest development of new tooling to open up these possibilities. Remember, before Windows started rolling out Copilot across the entire platform, Microsoft had Copilot in GitHub.

2:21As Chris Cashtanova puts it, AI isn't programming's future, it's its present. Speaking of AI and programming, our main AI breakdown today is about something called GPT Engineer. This is exploding on GitHub right now with tons of developers piling on, and it's basically an AI agent that can write an entire code base with a prompt. We'll discuss where it came from, what people are using it for, and what the implications are a little later. For now, let's move over to the policy side of AI. Last week, of course, the European Parliament passed a draft of the AI Act. Now, there are still a number of bureaucratic processes through which the AI Act could change in some details, but by and large, it represents the most comprehensive legislation we have on AI from a global power to date.

3:03As we discussed last week, a lot of the EU AI Act was written in a previous time, a pre-ChatGPT era. And so much of it has to do with how government, such as law enforcement, uses AI as opposed to regulating generative AI specifically. But given the rise of generative AI, they did graft on a few different provisions, and those are some of the most controversial. so far. One of them is a rule that foundational models, whether it's closed source or open source, will have to create a comprehensive accounting of the data that they're trained on. Some developers say that this isn't practical or is unworkable and would bring existing LLMs into non-compliance almost immediately.

3:36In response, Stanford University researchers have looked into whether the existing foundation models from companies like OpenAI and Google would actually pass muster. They organized the AI requirements into a number of categories, including data sources, data governance, copyrighted data, compute, energy, capabilities and limitations, risks and mitigation, evaluations, testing, machine-generated content, member states, and downstream documentation, and then gave each of the models a score out of four for each of those different categories. Hugging Face's Bloom model would be the most compliant right now with 36 out of 48 possible points with this Stanford scoring system.

4:10And of the other big guys, Google scored a 27 out of 48, OpenAI had a 25 out of 48. Meta had a 21 out of 48. Stability AI had a 22 out of 48. And pulling up the rear was Anthropics Claude with just a 7 out of 48. Now the researchers note that a lot of places where companies lost points wasn't necessarily because they were out of compliance with the EU regulations, but because there wasn't documentation available to show that they actually were in compliance. They write, Our work indicates where each foundation model provider can improve. Our work indicates where each foundation model provider can improve.

4:41We highlight many steps that are low-hanging fruit, such as improving the documentation made available to downstream developers that build on foundation models. Overall, they say, we find that foundation model providers unevenly comply with the stated requirements of the draft EUAI Act. We believe that all providers can feasibly improve their conduct independent of where they fall on this spectrum. Meanwhile, over in the UK, Rishi Sunak's government is taking its own steps towards better AI alignment and AI policy, allocating£100 million for a new UK AI task force. Over the weekend, the UK announced that tech entrepreneur Ian Hogarth, who is also a prolific AI investor and frequent commenter on AI risk and AI safety issues, would be leading up that task force.

5:21Ian co-founded Songkick and sold that company to Warner Music in 2017, and for the last five years has authored an annual State of AI report. Ian concludes his announcement thread by saying, I am fundamentally optimistic about the potential for science and technology to transform our lives for the better. The opportunities for AI to be a force for good are truly remarkable, but we need to do it safely. Now, staying on the theme of how AI might change things, the Grammys have come forward with their own policies around AI use in music. On Friday, the Recording Academy announced that artists who use AI in their songs can still submit them for awards.

5:52There just has to be what they call, quote, meaningful human authorship. However, songs that are fully generated by AI are not eligible and will be banned for the purposes of the Grammys. Between the Grammys then soft approving AI and Paul McCartney using AI to help the Beatles release their, quote unquote, last record, it seems like the trajectory for AI music is firmly focused in one and one direction only. Lastly, this week, the annual Cannes Lions Advertising Festival happens in France, and so it's only appropriate that we close on an advertising campaign for the real world. Camera maker Nikon, who of course faces some pretty significant headwinds when it comes to their core business model, has just released a new campaign called Don't Give Up on the Real World.

6:30It's actually a really cool campaign. They're taking what is clearly meant to be a mid-journey prompt and overlaying it on a photo that was shot on a Nikon camera in the real world, but that looks like it was generated by AI. The tagline of the campaign is don't give up on the real world. Now, outside just a really beautiful reminder of how amazing the world that we live in is, it's notable to me that we're already at the point where companies and industries are now feeling like they have to actively lobby against artificial intelligence in order to secure their future destiny. Fascinating times we live in for sure.

7:01That's it for today's AI Breakdown Brief. If you're enjoying the AI breakdown, please like, subscribe, and share. And I'll be back soon with the main AI breakdown. GPT Engineer is a new white-hot AI agent that can create entire code bases and which has the attention of basically everyone on GitHub right now. You may remember Emmett Hom from my show that I did with him about the Andrew Tate chatbot that he had created. Emmett had actually gone from non-coding to building this bot inside the scope of just a couple months, and we talked about that journey. Now, along that journey, Emmett had come to some conclusions about where one can learn about the latest advances in AI.

7:38On May 5th, he tweeted,

7:43Now,

7:53given that Emmett had said that, I was particularly intrigued to notice another tweet from him on Friday where he wrote,

8:07Now to me, GPT Engineer represents two converging trends. The first has to do with developers being on the front lines of how workflows are being reimagined and reorganized on the basis of new AI tools. Just this morning on the brief, I talked about a new survey from GitHub that suggested that 9 out of 10 developers, actually 92 % to be precise, were already using AI coding tools at work. This matches anecdotal evidence and basically the presumption that anyone at this point is making. Part of the strategy of companies like Google to catch up to competitors like OpenAI has been to jump out ahead of them in terms of tools to help people code.

8:46And in general, there is just a massive discussion about the increases in productivity that AI supported coding could unleash. So this is trend number one, AI as a tool for coding. Trend number two is autonomous AI agents. At the beginning of April, AutoGPT and Baby AGI really started to smash their way into public consciousness. These were tools that before ChatGPT and BARD had access to the internet could go out and search and gather information. They had long and short-term memory management. And the goal was for them to be able to, when given a task, figure out the set of steps that were necessary to complete that task and then go out and actually do it.

9:23Now, that included the potential of spinning up other AI agents to accomplish specific tasks. And that's really what got people excited. It was the idea that rather than just using ChatGPT to help them plan out the set of steps that they themselves needed to take, all they needed to do was prompt using natural language. and some autonomous AI agent would be able to figure out how to accomplish a task and then go do it. To understand just how excited people were about this, all one needs to do is look at their chart of GitHub stars. Stars are simply a way that developers can flag that they want to be able to get back to a repository later to interact with it or learn from it in some way.

9:59And by the beginning of May, just one month after this project launched, AutoGPT had more than 100 ,000 stars. Today, that number is over 140 ,000. What's more, lots of people tried no-code implementations of AutoGPT-like tools. There was God Mode, for example, as well as Agent GPT. But it wasn't long before the hype wore off, and some people started to think that maybe AutoGPT wasn't all it was cracked up to be, at least not yet. Now, I've talked before about why I think this has more to do with our expectations and the speed with which we assume that things are going to happen, as opposed to any failure of AutoGPT.

10:32But what makes Anton Oseka's GPT engineer interesting to me is that it takes that same impulse to try to get autonomous AI agents to do more, but puts it within a specific domain, in this case coding. It felt to me for some time that the way that we're going to see AI agents actually come to practice and be useful is by having them focused in on specific types of tasks. The GPT-4 engineer readme says, specify what you want to build, the AI asks for clarification, and then builds it. GPT Engineer is made to be easy to adapt, extend, and make your agent learn how you want your code to look. It generates an entire code base based on a prompt.

11:11Anton, who's the founder and CTO at Depict, wrote on Twitter, Introducing GPT Engineer. One prompt generates a code base, asks clarifying questions, generates technical spec, writes all necessary code, easy to add your own reasoning steps, modify, and experiment. Let's you finish a coding project in minutes. The example in the demo that he gave on Twitter is multiplayer snake in the browser use a python backend with mvc components the view needs to stream the state to all connected players please implement also the html and javascript necessary to run the game with only the code you generate now the next step and this is part of what makes gpt engineer interesting is that there's a process where it asks for needed clarification so in this case one game rules and mechanics how exactly does the snake move grow and interact with other players are there any power-ups or special game elements two player connections how many players can join a game?

11:58Is there a lobby system or matchmaking? And then there's a number of other questions, including game state updates, user interface, game controls, game end conditions, code structures, etc. Anton then answers this set of questions to give GPT Engineer the information it needs. And then it's off to the races. What Anton's left with is a complete code base, ready to go. Now, there are two very different categories of responses so far. The first, and definitely the most common that I've seen, is excitement. This is another one of those projects that to people really shows the possibilities of what generative AI could be.

12:28But in this particular domain, there is also some amount of skepticism. Mr. Catted writes, This is great. I think people are going to be skeptical about it working due to the failure of previous projects like this. But if you look at it as a time saver to producing projects that require multiple files, it makes more sense. Benjamin DeKracco writes, Interesting, but very skeptical. Even directly supervised GPT-4 cannot successfully build things currently, like dynamic websites, interactive apps, and games. It can build components which sometimes work and sometimes don't, but it goes off the rails fast.

12:58We'd like to see tests. Tom Gensler wrote a LinkedIn post about this called Conversational Code, an Exploration of GPT Engineer. Tom writes, Imagine a future where creating a software project is as easy as a friendly chat. Envision sharing your needs and watching them transform into a well-crafted software project without writing a line of code. GPT Engineer, Tom says, is more than just a project. It's a glimpse into a future where large language models like OpenAI's GPT play a pivotal role in shaping requirements in orchestrating software development. Though not yet fully featured, it foreshadows a time when software creation is a dynamic dialogue involving human creativity and machine intelligence.

13:34Tom clarifies the process of interaction between GPT Engineer and the user at the beginning. First, the user supplies a text file with the software requirements. GPT Engineer places an initial message to OpenAI's GPT to identify clarifying questions. The GPT Engineer system responds, prompting the user with those clarifying questions. And then this loops until all of the relevant questions are clarified to GPT's satisfaction. From there, the refined requirements are packaged up as system prompts. And also, GPT Engineer adds an additional set of instructions around what it wants to see as an output.

14:05From there, the GPT Engineer system receives a response from GPT-4 and then creates the source code files for the software project that the user provided instructions for. Tom shares a number of potential improvements for how this could go to the next level. One he suggests is iterative development. Quote, Any project like GPT Engineer that relies on LLMs is still subject to misinterpretation of a user's requirements and intent. This would be mitigated if there was the ability to iterate after the initial generation of code. GPT Engineer currently does not have the capability to iterate development, but it's easily imagined how it could.

14:35Tom also recommends some changes around the workspace project structure to allow for a multi-level file directory organization, and points out that GPT Engineer is currently limited by the context size limits of the input for the LLM it's working with. Now, Matthew Berman, who's another great AI YouTuber and who focuses often on technical implementations of new tools, got GPT Engineer to work and was really impressed by the first outputs. A new AI coding partner on the block, and it is absolutely incredible. Let me show you what I've been able to do the first time. No edits, no bug fixes, know nothing.

15:09This is the game Snake. I have not been able to get any other large language model to create this on the first pass. And this new project called GPT Engineer is absolutely able to do this the first time. I'm going to show you how to install this. I'm going to show you how to use it. And you're going to have your mind blown as well. Let's go. Now, this is still an incredibly nascent project with a ton of work to be done. And every day we're getting updates from Anton about new improvements. Anton shared a chart of GitHub stars yesterday afternoon, showing that over the last three days, he had gone from around 500 to 12 ,000, which is already a remarkable jump up.

15:43But as of recording this afternoon on Monday, that number was already up to nearly 17 ,000 stars. As people share what they've actually been able to build with it, I will be sure to share that here as well. All right, guys, that is it for today's AI Breakdown. If you are enjoying this, please like, subscribe, and share. Check out the podcast and the newsletter version. And until next time, peace.

16:08Thank you.