Is Google's New Bard Really the ChatGPT Killer?

0:00On today's AI Breakdown, we test GoogleBard versus ChatGPT. Before that, on The Brief, we cover Microsoft Prompting News, Unreal Engine 5.2, and new AI regulation coming out of the EU. This is the AI Breakdown, your daily AI news analysis show. Please like, subscribe, rate, review, and share with your friends. Welcome back to the AI Breakdown Brief, all the AI headline news you need in five minutes or less. We start today with something that's become a bit of a debate over the last few months, which is how much people should invest in this new skill set of prompt engineering. Here you have Logan, a developer relations person from OpenAI, saying, Hot take.

0:38You should not become a prompt engineer even if someone paid you to be one. So by Logan's reasoning, he thinks that the genesis of people being interested is correct. He writes, Many people are looking at AI, thinking about how it will disrupt the jobs market, and trying to position themselves well for that future. This is 100 % the right approach. However, he says, just because there has been a lot of media narrative around prompt engineering, that doesn't mean that will actually happen in practice. He says, The problem is that more and more, prompt engineering will be done by AI systems themselves.

1:08I've already seen a bunch of great examples of this in production today, and it's only going to get better. I imagine that this will be a skill that is learned as part of people's standard education path, not some special talent that only a few people have like it is today. Dr. Ethan Mollick says, When the folks at OpenAI are telling you that prompt engineering is not going to be the job of the future, because AI will be able to figure out what you need, believe it. Every trend is pointing that way as well. AI systems today are 20x easier to use than even four months ago. Now, for any of you who have watched any of my videos around AutoGPT, you'll know that part of what makes those systems so interesting is that they are doing self-prompting.

1:42And while they aren't always perfect, they certainly show that future in which AI is doing a lot of the prompting itself. Well, today we got new research from Microsoft that's all about APO or Automatic Prompt Optimization. They call it a simple and general purpose framework for the automatic optimization of LLM prompts. As part of their abstract, they write, According to Google Trends data, prompt engineering has seen a steep rise in popularity over the past six months. Several guides and templates are available on social media networks for creating persuasive prompts. However, developing prompts entirely by trial and error methods might not be the most effective strategy.

2:14To solve this problem, Microsoft researchers have developed a new prompt optimization method called Automatic Prompt Optimization, APO, to solve this problem. And so if you were someone who was wandering down the path of prompt engineering, what should you do instead? Well, Jason Gu here has an interesting answer. He says,

3:07So the logic here, I believe, is that instead of viewing generative AI as a way to shortcut your way into a field, it's a way to go deeper in a field and do more things than you would be otherwise able to while still having a grounding in that field. Next up, everyone who had a Jetson's fantasy is getting a little bit excited today because leaked documents from Amazon show that they're working on a new robot that has a generative AI technology underneath it called Burnham that could make its Astro home robot a whole lot smarter. So one part of that means that they're adding a conversational spoken interface to the home robot.

3:38You'd probably expect that given what else they're doing in the AI space. But it sounds like they're trying to use Burnham to give Astro a better contextual understanding of what's around. For example, in these leaked documents, they describe this Astro product as being able to use Burnham to find a stove that's left on or a faucet that's left running or identify an owner that has fallen down and needs 911 to be called. And in addition to that very basic sort of help, they're also seeing if it can initiate more complex tasks as well. An example given was a robot that sees broken glass on the floor, knows that it presents a hazard, and prioritizes sweeping it up before someone steps on it.

4:10Essentially, spot problems and solve them. Now still, it sounds from these documents that this is not an immediately to be available product. It is something still for the future. Next on the brief, a lot of people are getting excited about Unreal Engine 5.2. We've had a ton of games that are starting to leak using the new Unreal Engine. And over the weekend, we got this demo of its deformer tools, which allow for simulation of muscle, flesh, and cloth. Finally on the brief today, we told you last week that the European Parliament had agreed to increase the severity of regulation proposed in its AI Act.

4:42And we're getting some more details of what that meant. Marco Mascoro tweets, a new European AI regulation proposal would make any American open source developer that hosts an unlicensed LLM on GitHub and available in Europe liable for 20 million euro or 4 % of worldwide revenue. Basically, if an American open source developer places a model or code using an API on GitHub and that code becomes available in the EU, the developer is then liable for releasing an unlicensed model and GitHub would be liable for hosting an unlicensed model. Now, that is not the only red flag in this AI act. It's 144 pages long, and people are saying that it has one, very broad jurisdiction.

5:17Two, it requires registration of, quote, high-risk AI projects or foundational models with the government. It requires expensive risk testing. It defines the risks very vaguely. It says open-source LLMs are not exempt, and APIs are essentially banned. There's much more in here as well. Those are just some of the highlights. Zandersteinbrug writes, the EU is about to pass legislation that'll make it impossible for regenerative AI startups to compete with large tech. Knowing how transformative this technology will be, I'm furious these decisions are made by legislators who simply don't fully understand what they are doing.

5:46Y Combinator's Paul Graham wrote, I knew EU regulators would be freaking out about AI. I didn't anticipate that this freaking out would take the form of unbelievably stupid draft regulations, though in retrospect, it's obvious. Regulators gonna regulate. At this point, if I were a European founder planning to do an AI startup, I might just preemptively move elsewhere. The chance that the EU will botch regulation is just too high. Even if they noticed and corrected the error, it would take years. Now I think about it, this could be a huge opportunity for the UK. If the UK avoided making the same mistake, they could be a haven from EU AI regulations that was just a short flight away.

6:17It would be fascinating if the most important thing about Brexit, historically, turned out to be its interaction with the AI revolution. But history often surprises you like that. Now the problem, of course, for folks even who are sympathetic to the idea of wanting better regulatory guardrails is that compliance usually rewards incumbents. The more burdensome the compliance regime, the more it costs to actually comply, which means that it favors companies that have a lot of money already. That tends to be contra to the spirit of innovation and contra to the spirit of decentralization that regulations sometimes are going for, which leaves them in something of an ironic place.

6:46Anyway, I think many of us watching in the US aren't necessarily that much more optimistic, particularly seeing regulators dealing with other areas like crypto, so we'll have to just wait and see what happens. That's it for today's AI Breakdown Brief. If you're enjoying this, please like this, subscribe to the channel, and share it with your friends, and I'll see you back here soon for the main AI breakdown. Ever since Google IOS conference last week, there have been so many threads claiming that BARD is now better than ChatGPT. Is that really true? And if it is, in what? Today, we're testing BARD versus ChatGPT across a number of different areas from research to travel planning to see which actually outperforms.

7:24Welcome back to the AI Breakdown. Well, as I said in the intro, last week, Google had its IOS conference with some major upgrades for BARD and the LLM that underlies it. And since then, we've had a ton of tweets and threads, all claiming that BARD is somehow now better than ChatGPT, or at least on that trajectory. Now, part of what's making this interesting to do right now is that obviously BARD being connected to the Internet has some serious advantages over ChatGPT, which is only trained on data up until 2021. However, OpenAI has said that they're now rolling out web browsing across all ChatGPT Plus users.

7:58we're all supposed to be able to have access to it this week. And so that is going to get one-to-one parity for that access to the internet, which could make a huge difference. So what are we going to test? Well, obviously this is totally subjective, but I decided to test five categories. The first is creative or storytelling. The second is coding for non-coders, basically asking these services to help me build a website when I don't know how to build it. Number three is business strategy or entrepreneurial design. Number four is travel planning. And number five is research. And for that research, I'm going to choose it in an area where I have some existing knowledge so I can actually grade it a little bit more confidently.

8:31Let's start with creative storytelling, because I think that part of the magic for a lot of folks around ChatGPT is that it doesn't just seem like an encyclopedia talking to you. It actually feels like there's interest and verve and personality and joie de vivre. So part of that comes out in its creativity. So I asked both Bard and ChatGPT, can you help me write a story? The story is about a nine-year-old boy and a dragon who happens to be his best friend. I'd like a story about them in the style of classic children's books by authors like Roald Dahl. Now, the simplest way to put what came out of Bard was that instead of writing a story about a nine-year-old boy, it gave me a story that sounds like it was written by a nine-year-old boy.

9:09Of course, I don't want to be overly critical. These are technologies that didn't exist just a couple years ago, but it's still not particularly good. It does, in fact, follow the prompt roughly. It has a character named Billy who's the nine-year-old boy, and it has a dragon that he meets. It doesn't really have much of a narrative through line, and it certainly doesn't have anything in the way of poetry, but it exists. Chad G.P.T., on the other hand, actually put something together that's pretty quality. The title of this one is The Adventures of Leo and Ember, the Unlikely Duo. Once upon a time in the small and quaint town of Dunhaven, there lived a nine-year-old boy named Leo.

9:38Leo wasn't like other children. He didn't have time for schoolyard games or bicycle races, for he had a most unusual best friend, a dragon named Ember. Now one thing to note here is that technically my prompt did say that the dragon was already his best friend, and in Google Bard it actually was an origin story that's neither here nor there, maybe quibbling, but ChatGPT seems to have gotten that little detail better. The story that it puts forward actually has some interesting details, like Ember was a miniature dragon the size of a plump Labrador. The other thing that ChatGPT does much better than Bard, which is something that I've seen before, is that it actually has a narrative arc.

10:10In this case, the pivotal moment is that Leo and Ember discover a luminescent flower blooming by the water's edge, known in dragon lore as the moon petal. Adventures transpire, and then, quote, in the end, Leo and Ember discover the greatest treasure of all, the value of true friendship. Comfortable points on this one to ChatGPT. Next up, we have coding for a non-coder. This is one of those use cases that people are most excited about. Not just that they can code things that they didn't have any business coding before, but that they can actually learn how to code alongside these tools. So I gave it a pretty simple request.

10:40Can you help me create a website that is like news.ycombinator.com, but for AI stories? Now, for those of you who don't know Hacker News, it's an extremely simple website that has been very popular for going on a decade and a half at this point. All you can do is upload an article, vote it up or down, and comment on it. And this creates a really interesting list of what stories people are most interested in, particularly from the developer and entrepreneurial community. So you can imagine why it would be valuable to have this type of thing, but customized for whatever particular niche you happen to be interested in.

11:09So here's what Bard came back with. Yes, I can help you create a website that is like Hacker News, but for AI stories? The steps involved are one, choose a domain name. Two, install a content management system such as WordPress or Drupal. Three, choose a theme or template. Four, create pages and posts. Five, add content to your pages and posts. Six, customize the look and feel for your website. Seven, promote your website. This, I have to say, was pretty disappointing. And of course, I think I could push BARD to get a lot more out of this. I'm not trying to hone it in and narrow it and focus it.

11:35But what I will note is that part of what is supposed to be better about BARD is that it is natively connected to the internet. So it should be able to go check out news.ycombinator.com. I gave it the URL after all and see what the website actually is. And if it did that, it would know that it's not an interface that you can just replicate with something like WordPress unless you found a theme that was exactly that theme. Compare this to ChatGPT, which says, let's assume you want a simple version, keeping the key features such as user registration, posting stories, voting and commenting. Those are, in fact, as I just said, the key features.

12:07ChatGPT says, here's a basic plan of actions. One, define your requirements. two choose your tech stack three design the database four develop your back end five develop your front end six testing seven deploy eight iterate and for those of you who are listening not watching in each case there is a set of steps underneath this that it's suggesting for example under the choose your tech stack it has recommendations for front end back end and database under develop your back end it gives the basic endpoints we'll need so for user register login logout view profile edit profile as just an example and so again just as a learner who's trying to figure out what to do next.

12:40This is a much more comprehensive, clearer picture of what I actually have to do that would give me a better jumping off point to go in from there. So again, point in ChatGPT's favor. Next up, business strategy or entrepreneurial design. This is the type of thing that's been getting a lot more headlines, I think, and video views than it is actually useful in the real world. But because of its prevalence, I decided it was a good fit as well. So I said, I want to create an e-commerce business using generative AI. Imagine you're a great entrepreneur and have$1 ,000 to start the business. What business would you start and how would you do it?

13:13Bard comes back and says, I would start a business that creates custom design clothing and accessories. I would use generative AI to create a variety of designs that are unique and stylish, and I would sell them online through my own website. I would also use generative AI to help me with marketing and advertising so that I could reach a wider audience and generate more sales. It says, here are the steps I would take to start my business. Research and identify a target audience, develop a business plan, choose a platform for my e-commerce store, design and develop my website, market and advertise my business, fulfill orders and provide customer service.

13:39ChatGPT's response on this one looks quite similar. They say, one possible idea could be a personalized product design business where the AI generates unique designs based on customer preferences. Let's break this idea down into actionable steps. Business concept. The business could focus on creating personalized digital assets like posters, logos, apparel designs and more. The generative AI would use customers' inputs, e.g. color preferences, style themes to create unique designs. Then it says steps to start and this is very similar to what we saw from Bard. One area where it was perhaps slightly ahead of Bard in terms of the specificity was its budget allocation.

14:10With that$1 ,000, it suggests$300 to$500 for AI model development, which is a little bit more complex than is probably necessary,$29 a month for website development. It's suggesting print-on-demand services, so there shouldn't be upfront costs for that, and it says$300 for marketing for running a few targeted ads. Now, in this one, I was pretty inclined to give Bard and ChatGPT a tie, and I saw a few other people who had done something similar. Florian Kamiade, for example, did a Twitter thread about how he worked with Bard to figure out how to start a business that could bring up to$15 ,000 a month in revenue.

14:41And one of the things that he does that I think is very important if you are actually trying to do this is work with the AI to hone it in and focus it. He says Bard is very powerful in generating ideas, but this time will challenge him on something more specific. When the results are too vague, he challenges it to go a little bit more niche. He asks for an assessment of the competition, and even does a little bit of strategy design, asking for a buyer awareness matrix. I think this is a good example of how you would actually advance this ball down the field if you wanted to use BARD to build a business.

15:08And the key here is continued specialization and refinement. Min Choi also wrote that one of the things that makes BARD better potentially than ChatGPT around this sort of business ideation is that because it has access to the internet, it can leverage the latest information. He says, while ChatGPT can generate good ideas, it tends to be verbose and occasionally out of touch. Now again, ChatGPT with Browse potentially changes this dynamic, but for now, maybe there's a slight edge for Bard. Next, we go to the travel or personal assistant use case. The prompt is, I have three nights and two days in Paris with my wife at the end of June.

15:42We don't like lines. We have a medium to high budget. We like La Marée and Montmartre, sorry about the pronunciation, best, but like all of the city. Love classic cuisine and are fans of the city's literary and artistic past. Can you create a recommended itinerary. Bard's answer is, in a word, generic. Day one, it says, start your day with a leisurely breakfast at a charming cafe in Le Marais. Yeah, but which one? Then it does give a specific museum suggestion and a specific afternoon attraction. But in the evening, again, it says, enjoy a romantic dinner at a Michelin-starred restaurant in Montmartre.

16:11But which one? Meanwhile, ChatGPT just knocked this one out of the park. Day one, morning, begin your day with breakfast at one of the charming cafes in the Le Marais neighborhood. I recommend Caret, known for its excellent pastries and coffee. Basically, it goes through and gives all that sort of detail over and over. For lunch, try Chez Genoux. It's a Provencal bistro that serves classic French dishes and has a charming terrace. ChatGPT also really nailed the literary history part of the prompt. Day three, it suggests start your day with breakfast at La Procope in the Latin Quarter. It's the oldest cafe in Paris and was frequented by Voltaire, Rousseau, and other literary figures.

16:43Visit the Shakespeare and Company bookstore, a historic gathering place for writers like Hemingway and Fitzgerald. Now, neither of these are super unknown or anything like that, but they were both features of the last trip that we took to Paris, which was specifically designed to have these literary touchpoints. Anyways, it is entirely possible to me that Bard could end up producing similar results if I used more specific prompts, but the ChatGPT got a lot closer right out of the gate. The final area I wanted to test was research, and as I said, I wanted to do something where I had good information, so I actually had the basis to know which one was performing better.

17:13I asked both Bard and ChatGPT about the Nigeria-Biafra Civil War in the late 60s and early 1970s, which is what I happened to write my undergraduate history thesis on. I asked what were the main causes of the conflict, how did it come to an end, and how was the international community involved? What is the conflict remembered for? What are the resources would you point to to learn more? Now the biggest thing for you guys to note for the sake of this video is that this is basically the conflict where we got that famous idea of having to finish your food because there are kids starving in Africa.

17:38The Biafran War was really the first place that international humanitarian aid was born as we know it. It was the first time that there were televised images of starvation, and there was mass humanitarian involvement in a way that simply hadn't been the case for previous conflicts. Google Bard missed a lot of that. It did recognize that it was remembered for the use of starvation as a weapon and the international outcry being condemned by the international community, but it didn't really pick up on how integral it was to the development of the humanitarian industry. It was also a little reductive in terms of causes, but hey, this is a summary I wasn't expecting much more than that.

18:09Chad CPT just did a much better job. It pointed to a number of specific incidents rather than just sort of vaguely articulated grievances, did a better job explaining how the war came to its end through a blockade that led to starvation, and it nailed that those images of starvation were one of the most important legacies of the conflict. ChatGPT writes, this helps spur the development of international humanitarian law and the principle of responsibility to protect. So summing up on four out of my five categories, creative and storytelling, coding for non-coders, travel planning and research, ChatGPT was just very clearly ahead of Bard.

18:41On business strategy, they were roughly equivalent, although I'm willing to give that one to Bard based on what other people had said. Now, I looked for a couple of other comparisons as well to see, one, if people were having similar results to me, and two, whether there were categories that I had missed. Brian Kent on the Apricot blog did a similar comparison and found notably that ChatGPT did a better job of summarizing long-form content, and ChatGPT did a better job of writing a Python function. And then Tech.co did a long comparison, which I'll link to, that had self-awareness, ethical reasons, small talk and conversation skills, retrieving facts, generating formulas, creative flair, idea generation, linear planning, ability to summarize small extracts, ability to summarize broad topics, ability to simplify text, and ability to paraphrase text.

19:22I'm not going to go through all of them, but there were six that they declared BARD the winner, five that they declared ChatGPT the winner, and two that they declared were a tie. So really close overall, but the edge goes to BARD in their accounting if you weighted these all equally. Now, there are other feature reasons why you might want to use something like BARD over ChatGPT. For example, some have pointed out that BARD can process images as prompts, which could be a useful tool. And I think more than anything else, the thing that BARD has going for it is that it's integrated with the Google suite of tools.

19:50You can export BARD answers directly into Gmail or directly into Google Docs, and you can extend prompts with search by pressing Google it. So where I want to close it is back with Nate Chan, whose tweet kicked off this whole conversation. And I think he makes a really salient point in his second tweet. He writes, the arguments that GPT-4 is better than BARD missed the point that these LLMs will eventually converge to quote-unquote good enough at everything LLMs can do. If BARD came first and devs were building on its API first and the prompt community was exploring BARD first, considering all of Google's inevitable BARD integrations into all of its ubiquitous services, there would have been little to no chance that the entire community would have switched to a little-known startup named OpenAI to use a slightly better LLM.

20:30Google missed the first mover boat, but they still have every opportunity to retake the lead. That might be true, but OpenAI has a ton of energy around it. And for now, ChatGPT4 is absolutely, no doubt about it, better than Bard. How long that will last is another question. That's it for the AI breakdown. Hope this was helpful. Let me know what you found. Are there areas where Bard clearly beats ChatGPT? I'm super interested to know. Hit me up in the comments. And if you're enjoying the AI breakdown, please like, subscribe, and share it. Until next time, guys. Peace.

21:06Thank you.