AI that Can See the World? Meet MiniGPT-4 an Open Source Image-to-Text Model

0:00The episode you're about to hear was first released as a YouTube video on Wednesday, April 19th. This AI breakdown is all about Mini-GPT4, a new open source software just debuted by researchers that can take an image and actually describe it in words. And more than just describe it, it can infer things from it, turn it into working code, or even write a poem about it.

0:28Over the last six months, we've had our minds blown about AI's ability to take words and turn them into images. But what if it could go the reverse order, where AI could take images and turn them into words? What if AI could look at a food and turn it into a recipe? What if it could look at an image of a whiteboard and turn it into working code? That's been on the table for companies and projects like OpenAI for a while. But this week we got an open source project called Mini GPT, just research at this stage, which shows exactly that happening in practice. So in this video, we're going to talk about Mini GPT-4, where it comes from, what it does differently, and then look at some demos to see if it actually works.

1:17Mini GPT-4 is a research project that was just announced a couple of days ago. The researchers call it enhancing vision language understanding with advanced large language models. And effectively, as I mentioned in the intro, what they're trying to do is to allow AI to work in the opposite direction of tools like MidJourney, to be able to see images and then describe them, interact with them, generate information on the basis of them. Now, they've done a few things differently, which are a little bit more technical than I want to get into in the context of this video. But suffice it to say that they've trained a smaller data set, but more.

1:54And that's producing some really interesting results. For those of you who are interested in the technicals, let me know, and maybe we'll do a deeper video on some of that aspect of the research. But for now, I want to focus on what the use cases are and what's coming out of this. Nate Chan here really sums it up. Ask questions about pictures, he says. What is this? Tell me what's wrong with my plant. Give me a recipe for this dish. Write the code for this rough mock-up. Now, these all come from the mini GPT-4 demos that they show, and so I'll walk you through these kind of quickly. They show identifying problems from photos, providing solutions.

2:29So this is an image of a plant that looks sick. The person says, what is the issue with my plant? What should I do now? Mini GPT-4 says the image shows a leaf with brown spots on it. They're likely caused by a fungal infection, which can be treated with fungicide. It then gives a set of steps to follow up to treat the plant. The next example they show is a product advertisement. The prompter says, we design new mugs as shown in the image. Can you help us write an ad? Mini GPT says, sure. Introducing our new line of mugs featuring adorable cat designs. These mugs are perfect for coffee lovers and cat enthusiasts alike.

3:06Another use case which I think many of us Pinteresters will be very excited about is food recipe generation. The prompter drops a set of beautiful looking lobster tails that are stuffed and says, how should I make something like this? MiniGPT figures out what it is and suggests both ingredients as well as the steps to the recipe. This one could change how people build websites, right? Generating website code from handwritten text and then the rendered website. So this is a whiteboard that says a joke website, content of joke one, punchline one, and the prompter asks to turn this into HTML and JavaScript, which MiniGPT then does.

3:45There's even more creative use cases that the researchers have stumbled upon. So this is an example where the prompter says, can you write me a beautiful poem about this image? It's a dog sitting with presumably its owner on a hill during a beautiful sunset. Many GPT said, sure, here's a poem about the image you provided. A man sits on a hill with his dog by his side. The sun sets behind them, casting a golden light. the city below is bustling but they are at peace enjoying the moment in each other's embrace now we can get into whether it's a good poem or not but it's a poem and that's pretty remarkable then they can also and this is the last example they use discover unusual content so the prompter posts an image of a cactus sitting on ice in a lake it seems clearly unreal and says describe the image mini gpt says the image shows a cactus plant standing in the middle of a frozen lake and then goes on to describe other aspects of it.

4:41When the prompter says, can this happen in the real world? Mini GPT says, no, this image is not common in the real world. While it is possible for cacti to grow in cold climates, it is rare to see one standing in the middle of a frozen lake. So these are the demos they presented. But really, the question, of course, is what can people repeat or do? And this isn't a full-fledged release. This is just a demo release. So that should be kept in mind. But the initial responses that people have are pretty impressed. Simon Willison says, Mini GPT-4 is pretty astonishing. An AI chatbot you can use to ask questions about an image, a feature that's been promised but not yet shipped by GPT-4.

5:18So in his post, he shows an image of an ice cream that's in a heart shape with sprinkles on top, sitting on a black napkin on some sort of table. He says to Mini GPT-4, what is this thing? Mini GPT-4 responds, This is a pink ice cream cone with sprinkles on top. It is sitting on a black napkin on a table. Another person who tried this out was Broski on Twitter. They say, so I tried out mini GPT-4 and used an image of Lionel Messi and asked the LLM which soccer player it is on the image and for what soccer club he plays for on the image because Messi right now plays for PSG. The results are astonishing.

5:54Basically, the mini GPT-4 was able to correctly identify Messi and what team he was playing for in that image based on his jersey. Broski also had another test, saying, So I tried out mini GPT-4 again and asked it to describe this mid-journey image I made a while back. This might be the perfect way to do reverse image prompting for mid-journey. Now, of course, mid-journey actually just released a similar feature, which they called describe. So using the describe command on mid-journey, you can now ask mid-journey to figure out how it would prompt to get the result of a particular image. It's a really great way to reverse engineer prompts that you like or images that you're impressed with.

6:32It's a great learning tool, and I think it's probably going to build into mid-journey workflows going forward. But this is another example of a similar type of technology. Now, from a technical perspective, Shiva Kintali here says, forget about LLMs. I'm suggesting a new name, MMLMs, Mighty Might Language Models. Remember the new mantra, train a smaller model for a longer time. The latest breakthrough is mini GPT-4. So Shiva here is saying that this is a different category. It's a new approach to training that involves a smaller model for a longer time. But all of this, of course, made me want to try it out.

7:07And like I said, this is just a demo. So we have no idea if it will be as performant as the tests we saw or that the tests that they suggested in their demo. But let's try it out. I've got here a picture of the David and we're going to upload it and start the chat. So let's ask it, what is this and where is it? Let's see how long it takes to figure it out. This is a statue of David, a biblical figure known for his victory over Goliath. The statue is located in the Academia Gallery in Florence, Italy. So obviously got it right. Now let's try one of those creative responses. I have no idea if we can get it to write a poem, but let's see.

7:44What type of emotions does this artwork inspire in visitors who come to see it? I have no idea how it'll handle that question because it's so subjective. The statue of David in the Academia Gallery in Florence is likely to inspire feelings of awe and admiration in visitors. The statue is a masterpiece of Renaissance art and is considered one of the greatest achievements of the humanist movement. The depiction of David as a strong, confident, and heroic figure is likely to evoke feelings of pride and inspiration in visitors. The statue is also likely to inspire feelings of wonder and amazement at the skill and artistry of the sculptor, Michelangelo.

8:21Pretty impressive. Now let's do one more. Can you write a poem about this sculpture that captures some of those emotions? In the halls of fame, a statue stands tall. The David of Michelangelo, a work of art. The marble form, so lifelike and grand. Inspires awe and admiration, a sense of pride. A symbol of strength and courage, a heroic guide. The mastery of the sculptor, a work of art. Skill and craftsmanship, a sight to behold. In the academy, a gallery, a place of fame. The David statue stands a source of inspiration and awe. A symbol of humanism, a work of Renaissance art. A testament to the power of the human spirit.

9:03In the statue of David, we see the strength and courage, a source of inspiration. a symbol of Michelangelo's enduring power. Pretty cool stuff, I have to say, but let's see if we can replicate one of the other demos where we post a picture of food and see if it can come up with a recipe. So let's upload and write, how would I make what's in this photo? This is a Greek salad with tomatoes, cucumbers, red onion, feta cheese, and olives in a white bowl. It is topped with a vinaigrette dressing. So it's a pretty good summation of what we saw in that demo. Anyways, I think we could get a lot more into these, but let's do a quick summary.

9:38I think why people are so excited about this is a couple parts. First of all, it works. You can see from that demo, even though it was very nascent and simple, and this isn't really a full production model or anything like that, that we were able to replicate some of the demos that the researchers had presented, which is really cool. Second, I think people are really excited about the fact that this is a feature that companies like OpenAI have talked about for a while, but it's being shown in practice by researchers doing it in a slightly different way and in an open source way. Yesterday, we talked about whether open source can actually compete with closed source projects, and when we see things like this, it makes it seem a heck of a lot more viable.

10:18Anyways, mini-GPT is definitely the type of thing that feels like it will be commonplace going into the future and something that we just expect to be a part of what we can do, even though it was unimaginable so recently. Anyways, that's my sum up of Mini GPT-4. I will certainly keep you guys posted with any new use cases or experiments that I see, but for now, thanks for watching. Until next time, peace.

11:01Thank you.