Advent of GenAI Hackathon recap

0:28Welcome to Practical AI. near your users. Learn more at fly.io.

0:37If you're listening, you know software is built from thousands of small technical choices, and some of these seemingly inconsequential choices can have a profound impact on the economics of internet services, who gets to participate in them, build them, and profit from them. This is especially true for artificial intelligence, where the decisions we make today can determine who can have access to world-changing technologies and who can decide their future. Read, Write, Own, Building the Next Era of the Internet is a new book from startup investor Chris Dixon that explores the decisions that took us from open networks governed by communities to massive social networks run by internet giants.

1:16This book, Read, Write, Own, is a call to action for building a new era of the internet that puts people in charge. From AI projects that compensate creators for their work to protocols that fund open source contributions, this is our chance to build the internet we want, not the one we inherited. Order your copy of Read, Write, Own today or go to readwriteown.com to learn more.

1:55Welcome to a very special fireside chat, which corresponds with the ongoing advent of Gen.AI Hackathon that will also be reposted on the Practical AI podcast. I'm very pleased to have been participating in this hackathon as one of the organizers but I'm also joined here in the fireside chat by an amazing team from Intel's liftoff program for startups who helped organize this hackathon that we'll be talking about throughout the day so I'd like to kick it over maybe to Rahul to describe a little bit about what Intel Liftoff is. Hey, hey, all. Thank you, Ryan. This has been an incredible experience.

2:48Let me, before that, even talk about the advent of Gen.AI two sentences. This was probably the biggest generative AI hackathon that our team has organized. And the submissions, all the different chats and things that we have seen, And this has been really awesome. We've got a lot of positive feedback and a lot of things that we need to improve for the next time. So thank you for participating for the hackathon. And we will be announcing the winners of the final product development in a couple of hours. Before that, let me talk about Liftoff, right? So Liftoff is an accelerator program, specifically a technical accelerator program for early stage startups.

3:28So if you have an idea, you're a seed startup or till series B, you want to scale, you want to build some cool things in AI or machine learning, please join the program. It's free. I would categorize the benefits basically into three different pillars, world-class technical support and technical expertise. That's me and Ryan here. We lead the engineering side of things and we have an incredible engineering scale team. Some of the folks are here like Jyotius and Varad. then you get access to technology both intel software and intel developer cloud intel developer cloud is a production ready cloud specifically designed for ai workloads prediction guard is one of our the startups that came to our program earlier this year and they are running on intel developer cloud now using a gaudi to accelerator and i'm sure that many of you folks who have participated in the hack app would have used prediction guards llm apis the third one is co-marketing and bringing your once you have built that product and deployed it uh the next thing is to make some money right so we uh co-market your um startup your idea through all of intel channels and also our we have a network of accelerators a network of folks that's beyond intel that we take the product you have built the company i built uh and basically market uh it all over the world we also connect you with um our sales teams to see if there's a potential for selling the things that we built through Intel channels.

4:58It could be a service on IDC, or it could be separate that one of our customers of Intel is looking for. I would urge anyone who is looking to really bootstrap and accelerate your startup journey to join Intel liftoff. Yeah, that's great. That's great, Rahul. Could you describe a little bit? So I remember initial discussions between a few of us, you had this idea for the advent of Gen AI hackathon. Now, a lot of people in the audience out there might be familiar with advent of code. So how did you start thinking about this advent of Gen AI hackathon? And what was your initial vision for it? A lot of the geeks in the call, which most of us are, would know Advent of Code.

5:48It's a set of programming challenges. You can take any sort of programming language you want to learn or to attempt to solve these algorithmic questions. I've been doing that for many years. And I thought, you know, Gen AI is something that's new and something a lot of people are talking about. But there are no fun set of exercises that you can use to learn and also build cool things. with the technology that's existing today. So I was thinking, why not just create something, a set of challenges that's tailored from a person who might not know how to code, but have seen prompt engineering and creating cool prompts to create images, all the way to people building with LLM APIs.

6:31So we designed a set of challenges to bring in as much as audience as possible to introduce them to Gen.AI and also learn through that process. Many folks I've seen in the chats where they came with just prompt engineering knowledge and have built really cool things, graduating from one challenge to the other. And there has been a really good community help also, like people talking to each other and trying to help out how to run, how to build these things. This has been a really, really good exercise. Even some of the challenges when I was building it, I was like, oh, whoa, I would like to do that because we always wanted to add a fun element to it.

7:07so no challenge is dry just an algorithmic question but there is something fun element to that that's where the whole advent of Gen.ai came about and we want to do this yearly so next year it might be a new technology it could be a multi-modality hackathon some other things we already saw some cool engineering and stuff folks are building this challenge but it would be something but I'd like to do in December advent or something else that lift up dust for the community yeah that's awesome So if you're listening to this after the holiday season, or as this will be coming out on the podcast later, you didn't miss out on participating in 2023.

7:49But there's going to be more opportunities to participate in things like this that Intel is going to put on later on. So I definitely recommend people keep an eye out on the Liftoff program and social media to hear about things. Could you speak a little bit to the response to this first advent of Gen.AI and the participation that happened? I think we were all a bit surprised at how many people joined in to this. So could you speak to that a little bit and anyone else from the Liftoff program, any of your observations for the type of people that joined in and the range of experiences and all of that?

8:32This has been fantastic. We had grad students, we had even students who are in school who have taken prompt engineering courses who just wanted to have fun working on some of the earlier challenges. There has been experts also, some of the experts in LLM's and Gen AIS, I worked on it. And some of the products that they've built, right, or some of the challenge solution is, it's like an MVP, a startup will build. We have many startups who are building a similar solution who are taking like six months to a year to build a full solid solution. But some of the challenge answers, especially using the RAG example, Dan, or the Python code explainer, things like that are difficult.

9:12And they have even gone further. When we asked about how to, can you create a storyteller chatbot? People have created a story plus image chatbot, a multimodality chatbot. So these are many levels of experience and that's even some of the folks from Intel participated. That's also a very positive thing where we see you can, in a level playing field, work with folks from Intel and solve the challenges together. We had folks from Berkeley, folks from many different enterprises participating. So this has been a mix. And I was amazed at the level of participation. I didn't expect this many people would participate.

9:52And we had to stop registrations after 2000. people registering for the event and uh this has been just great ryan do you want to add something yeah i just remember this is raul's idea when he first called me it's like cleaning up after thanksgiving in my basement or something he's like you know i have in a code yeah of course of course i want to do advent of gen ai like all right what do you want to do we worked it out and it was like set the goals like let's get like it's like yeah we can get big like you know a couple hundred people at least would be a success but like stretch it like hey i mean we could get maybe a thousand even and do the biggest event of the year well we end up cutting it off at twice that at twice the stretch goal so then of course the entire time we're like oh man did we do a good job are people gonna like this are we gonna get submissions every time knock it out of the park way more submissions than we were hoping for and the quality was excellent and the amount of people we saw in the chat helping each other where it's like somebody you know we're all trying it's 24 hours a day so we're you know our team is trying to stand and answer questions as much as possible we set that as a goal but what we saw like from go like from when it started was when people would ask questions other people would jump in and like link them to the explanation or the documentation or whatever these were all the dreams for the event so it's been incredible and i want to thank every single person who was involved yeah and maybe it would be good for those maybe some people jumped into certain challenges and not other challenges, or they might be hopping in at one point or the other, or they're learning about advent of Gen.ai as they're listening to this.

11:27So what were some of the challenges that were presented to the participants? And how would you consider them in terms of like relative challenge level or skill required to complete them? We designed this challenge in a progressing level of, I wouldn't say difficulty, but I mean the ability to code, I would say. It's not difficulty exactly and creativity also because a lot of Gen AI, what we see at least in Liftoff is that AI has become truly commoditized. And you don't really need to go through a couple of courses on neural networks to build an application right now. You have the amazing transformer ecosystem.

12:11It's really easy to integrate some of these things, the AI applications to AI superpowers to the applications you build. I would take the first challenge. For example, the first challenge was to create a narrative-based set of images. And if you look at that challenge the first time, it looks like it's very easy. You just create a couple of images using stable diffusion. It's all about prompt engineering. You don't need to know a single line of code. All the notebooks and the models, Everything is available on Intel Developer Cloud. You just create a standard account, log in there, get JupyterHub open and just play with it.

12:45But the thing is, creating a transition from an image to another and creating a whole story with set up by images, it's not easy. It's really difficult. And we even saw some folks creating a comic book generator using this challenge. That sort of imagination, right? That's what I really wanted people to do. But I didn't want to say that, OK, please create a comic book generator as a challenge because it's really difficult. some of the folks even without knowing that built it. You take the final challenge, right? That was Python code explainer where you give in a Python code, you use LLM model to understand the code and give an explanation of it.

13:20We had additional challenge, additional sub-challenge there, show the source of documentation or stack overflow questions where I can go into and learn about it more. These kinds of additions makes it very interesting and a little bit more complicated where you have to use a vector database you have to use prediction guts, LLM APIs to get the right model. And you had to design a UI for it, all in the constraints of a Jupyter notebook. So I would say that there was a progression of difficulty. Difficulty is a very relative word, but yeah, there's a progression of difficulty if you're just coming to Gen.AI.

13:54And the whole idea is that it's a single package. So even if you're not participating in Gen.AI, we have released all the resources that we have built for this. You can take this to basically get you an idea of what Gen AI can actually help you to build and infuse to your applications. And you can just go through the different challenges. Or if you're an expert in LLM rank-based application, go to that particular challenge and take a look at it. So it's now become a learning resource also, not just a hackathon. If I could just add on to that real quick, it did on purpose go up in difficulty. The level of each challenge was supposed to get harder and more advanced, let's say, in terms of coding ability as we went on.

14:35But the real focus was about skills and understanding the tools that are being used. Within the industry, there's been a huge focus on creating new levels of abstraction to make neural networks easier to use and build with. It used to be very challenging. Now, not nearly as much. You don't even need to stand up your own neural network anymore, right? You can grab an API. So if you look at the challenges, each one is kind of focused on a different skill. And if you go through all five, you cover prompting, specifically for images, text to image. You cover using an LLM API and the different things you can do with it.

15:14You cover image to image, so image editing with AI, then the third one. And then RAG-based applications with LLM APIs. And finally, we thought the fifth was the most advanced, like the code explainer one. There are companies that are basically, that are big companies that are working on that exact problem that are betting that it's going to unlock, that a good solution for code explanation improvement and generation is going to unlock many billions of dollars of value. So all of these things are focused on these different skills. And our hope was that for people who are maybe software engineers looking to move over to AI or to students, like whatever, anybody who is interested in learning AI skills, that by going through the ones they chose to or all of them, by the end of them, they'd have kind of a portfolio of knowledge in their head about the different skills.

16:03both on understanding how to use them and how to do a good job with prompting, but also by going through the code and understanding all of the code and all the notebooks and idea of what else they could build outside of the narrow set of applications that we asked for over the course of five days. I think one thing that impressed me about the set of challenges that you all came up with was that it focused really around image generation, coherent image generation. and on the LLM side, sort of retrieval-based methods along with chat. So I think all of these things are the things that people are finding most utility out of when they're first implementing AI solutions within their actual enterprise or industry or startup or whatever environment they're working in.

16:52In particular, retrieval-based methods and RAG systems, at least for our clients, we're seeing, that's like the first thing that everybody is building, right? You have your own company's set of data. In the case of the challenges that you all put together, maybe that's external Python documentation for the code explainer, or maybe that's just some external PDFs or YouTube videos or whatever it is for the RAG-based solution. But lots of companies have this data that has this sort of unlocked potential and is unlocked via these retrieval based methods, which is a lot of times what people are building first when they adopt this technology.

17:32So I think it was great that you all tied that together with the participants to give them practical skills in that area and kind of help them learn, you know, what is a vector database? What is a rag system? How do you implement this with custom data rather than kind of immediately hopping to fine tune a model, which I know, of course, you can do more easily than ever as well. But there's a lot you can do even just by integrating your own data with retrieval or other sorts of methods. I do want to ask here in a second, some of the solutions that you saw and what stood out to you just to highlight some of those really cool things that we saw.

18:11Before we do that, so you can have that in the back of your mind and think through some things you'd want to highlight. I'm wondering if you all could speak to the Intel developer cloud specifically, which is something, of course, I've found utility out of, but it was something that was kind of unique about this hackathon. And it might be something that the participants here are a little bit like this was their first time using it, but also there's a whole lot available there in terms of different ways to run AI models that are maybe some people are less familiar with. So could you describe a little bit the Intel developer cloud and maybe also highlight some of those like different unique ways that people were running AI models outside of just like throwing it on a GPU.

18:58There are actually some interesting kind of other either tooling or hardware software available for people. Could you highlight a little bit of that and the unique ways that people were specifically running AI models throughout the hackathon? Sure. So Intel Developer Cloud is Intel's production-ready cloud specifically for AI and machine learning workloads. And of course, when we say AI and machine learning, it's matrix math. So many other compute-heavy workloads can run really well on IDC. So for this particular hack, we provided anyone logging into Intel Developer Cloud registering on IDC as a standard or free tier user, you get a shared JupyterHub instance where you get access to Intel's data center GPUs, Intel Xeon processors.

19:46And I would say this system, for a free tier user, I don't think any other service provide. I mean, there are many services with the JupyterHub front end, but the amount of compute and amount of memory and RAM and even file storage that you get in the systems. I haven't seen a single cloud service provider providing that. And we have seen a lot of people really using it and giving us feedback on how we could even improve it. Today on IDC, we have a lot of models or LLMs already. There are tens of, even hundreds of local models that we are planning to add further to boost this. There are stable diffusion models, LLMA models, and things like that.

20:26Beyond that, for productionizing the workload, right? Dan, in your case, you are using the Gaudi 2 accelerators. Those are specifically designed for workloads that request high bandwidth, like LLM and Gen AI workloads. And I like this.

20:44I lost my train of thought. But yeah, we have Gaudi 2 accelerators, which we are seeing incredibly competitive and sometimes outclassing the best out there for your particular workloads. Along with Gaudi accelerators, those are specifically designed for Gen AI and AI workloads. We have general purpose GPUs, the data center max series GPUs, both with 48 gigs and 128 gig versions. So the folks in the hackathon, they actually used both our fourth generation Xeon, that is the latest Xeon that we have, which what it particularly does is that it accelerates your machine learning workloads. We have dedicated instructions in the CPU to sometimes even take your workload to 2x, the performance that you got in an earlier generation.

21:29It's all about making the CPUs. How we see it is that making it as efficient as possible and making it as fast as possible, just still maintaining the general purpose utility of a CPU. Then, like I said, the data center Max series GPUs, a little bit more generic solution where you can run your AI workload, HPC workloads. each of these machines when you're productionizing you get a vm you get an eight node eight card system there are also clustered systems available then comes the gaudi accelerators both there are single node machines and also clustered machines if you want to do pre-training or big fine-tuning all those cool things and soon we'll have kubernetes surveys object store file store all those things coming up so it's going to be great what i see is that if you are building a startup, it would be very difficult to find a performance and accelerator cloud like IDC out there.

22:26I'm sure that there are different hyperscalers, but this uniquely for startups, from my personal experience, is a really, really awesome solution. I'd like to know more from you, Dan. You are one of the first customers of IDC, right? What are the things that you thought that really made you decide to choose IDC from the performance and also the team side also, right? Appreciate that and appreciate the support that you all have given. I think it's interesting maybe for people out there that are less familiar with the various options for model deployment to understand that there is really good tooling.

23:02like you say whether it's optimizing a model and deploying it on a cpu or an edge environment or or just a cheaper inference solution or it's like all the way to these gaudi 2 processors that we've been experimenting with i think there's a lot of interesting and approachable tooling for that so i first came across some of the tooling around gaudi 2 by actually seeing blogs on the hugging face blog about Gaudi 2 and I think at the time like the Bloom model which is a very large model and running it on either a single accelerator or spread across eight accelerators with really high throughput on the inference side and doing that with tools like Optimum Habana.

23:50So for those of you that are out there and wanting to explore things actually if you look up the Hugging Face Optimum library there's a lot of great tooling that you can play around with there even not for for Gaudi, but for other processors too. So whether that be CPUs, GPUs, the Gaudi 2, HPUs, the Data Center Max GPUs, Optimum kind of provides you a way, if some of you can visualize, maybe you're writing in your code and you're importing a model from Hugging Face, it's just like auto tokenizer or auto causal LM or whatever it is. with Optimum, a lot of times either you can just do like a one, a couple line replacement and just replace that with the Optimum version of those classes or do some wrapping of the various models with optimizers.

24:43And this allows you to run your model very fast on a wide range of architectures. So I think to your point, Rahul, I think one of the things that we found really useful is the actual ease of use in coming in and saying, okay, well, we have this stuff running on a GPU. Let's try it on this various other architectures. I remember even maybe two or three years ago trying to do some of this model optimization things for edge deployments. It's very, very challenging. So a lot of times I would try to optimize a model at the time working on speech models and other things, and it just wouldn't work because operations wouldn't be supported or something like that.

25:27But this tooling, which is cool because Intel is working directly with Hugging Face on this tooling. And of course, the ease of use has just been ramped up drastically. And we've been applying that with really good results, particularly for inference for LLMs. So that's been a key feature to that change happened. That's really awesome to hear, Dan. and particularly also the thing you mentioned, right? So you can think of Intel in two ways. Probably the biggest semiconductor manufacturer, the coolest chips. The other is Intel is an open-source software company also. We are contributing to almost all big open-source projects, Linux kernel, most of all things.

26:10If you see, we would be anywhere in the top three, PyTorch, TensorFlow, HuggingPace, any sort of open-source solutions out there. we work really hard across to make sure that your adoption of a technology is as easy as possible and try to upstream as much as possible to the core PyTorch library or TensorFlow library and things like that. In cases where we feel that there are further optimizations that could be done and these things cannot be upstreamed in a couple of months to the mainline repositories, we release extensions also. So for example, if you take the Outerbox PyTorch and run it on a CPU, you already get a lot of performance because Intel's neural network accelerator library, 1DNN, that's powering a lot of these operations when you're running on a machine like an Intel Xeon, for example.

27:00But if you want to go a little bit further, we have things like Intel extensions for PyTorch that with one line of code, essentially it's Intel extensions for PyTorch as IPEX and IPEX.optimize and pass in the model. We add further optimizations to run it as fast as possible. We are also working on even upstreaming whatever possible to PyTorch mainline. So that thing you mentioned, right, it's very important to work with the community and enable the software that the community uses rather than having a completely different architecture and something that's sometimes is closed source and working on it.

27:39that's not the way Intel things. Even the whole concept of OneAPI, heterogeneous programming, everything open about it where other vendors can come in and add their accelerators to the OneAPI and use the OneAPI standard where if you're writing code for a CPU, there should be minimal to no changes that's required to run that on another accelerator. That's the philosophy that we are working with overall in the OneAPI architecture that works underneath all these acceleration libraries. Optimo Habana, we've been working very closely with the HackingPace team. Almost all LLM models work out of the box.

28:16There are models that we have tested and benchmarked that's available on GitHub. And things like VLLM, right? Our info support. All those things are enabled through Intel libraries. For example, BigDL and things like that. Giving a higher level abstraction beyond PyTorch. Because when we talk to startups these days, we feel that PyTorch is considered as a low-level library right now. And that's a little bit funny for folks who have worked in Tiano or, you know, even beyond that in 2016 and 17. And coming from the early days of TensorFlow to see fighters going low level and there's this higher abstraction libraries to work on top of that.

28:55It's really an exciting time to be in and work with you all. Yeah, for sure. And I also want to highlight in addition to like open source code, it's been cool to see Intel recently released Neural Chat, which was a fine tune on the Mistral model, which is openly accessible on Hugging Face and permissively licensed. So we've been experimenting with that and we saw usage of that in the hackathon. So it's cool to see people like a couple of these models, Neural Chat, which is a fine tune of Mistral, came out like, I don't know, maybe a week before the hack. And Notice, which is another fine tune on Mistral, came out like a few days before the hack.