Supercharging Developer Productivity with ChatGPT and Claude with Simon Willison - #701

31:47I will copy and paste in the bits that worked from the previous chat, and then I'll start things running away. That, I feel, is really important to spot when your previous context in the conversation has got it into a weird state and throw the whole thing away and start again. The one thing that I use for coding all the time, which is amazingly effective, is just copy and paste in a bunch of examples from different projects that illustrate that that feed into what you're trying to do i think my favorite example of that is um so i built a tool that lets you run ocr against tech against pdfs in your browser so it's a web page you go to and you literally open a pdf file and you say ocr this and it runs through and it gives you the text from every page of that pdf no matter even if it's like scanned handwriting or something it'll it'll still do a reasonably good job the way that works is it It uses a JavaScript library called tesserect.js, which is the Tesseract OCR library and web assembly running in your browser.

32:45And it uses a JavaScript library called pdf.js that Mozilla built, which can read PDF files and turn them into images. And so when I built it, I literally, I already had examples of those two different things that I'd use in other projects. And I literally pasted in a chunk of JavaScript and said, here's how to use tesserect.js. And the chunk of JavaScript says, here's how to use pdf.js. And then I described my project. I said, okay, I want it to take the PDF, turn each page into an image, run Tesseract against the images, show them on the page. And it just worked. It got me exactly what I wanted right out of the gate because I'd fed it the exact, it's exactly what I'd do with an intern.

33:23If I had an intern, I'd say, look, here's the thing I wrote with Tesseract. Here's the thing I wrote with PDF.js. Here's what I want. And that's so effective. Like often you always have to consider the training cutoffs for these models. Like you mentioned fast HTML. They probably haven't seen that because it came out in the past two months. But you can copy and paste in. With cursor, you can at mention the docs and it'll pull in the docs and kind of do like a rag like thing. And it's actually pretty effective at incorporating knowledge like that. That's exactly. Yeah. My version of that is I copy and paste the docs into Claude.

33:58But yeah, cursor being able to do that is it's the same mechanism, just with an even nicer UI for doing it. It's brilliant. That works so, so well. And the context windows on these models keep on getting longer. Like Claude does 200 ,000 tokens. They have a version of Claude that's 500 ,000 tokens if you pay them a lot of money. The Google Gemini models are at 2 million tokens. 2 million now. It's, yeah, mind blown. Absolutely astonishing. So, yeah, you can just, you can chuck a lot of additional stuff into the sort of short-term memory of these models to solve problems. we briefly touched on vision models but we didn't dig into that that's an area that you're excited about as well I remember you were one of the first people I saw that dug into the Gemini vision model you did this thing where you like took a video of I think it was like your bookshelf and you asked it to to list out the books and it did an incredible job of listing out the books on this like fast scan of a bookshelf um you know talk a little bit more about what you've done since with vision models are you still excited about them have you incorporated them into your workflow or practical you know problems or things you're building yes i mean the vision models so vision models are quite new like the first llm with vision that was any good was gpt4 vision and that was november last year like they're not even not even a year old at this point and since then um the open area ones have got better the claude 3 family came out and those are all vision models and 3.5 and then there's the gemini ones as well that um the demo i did with the bookshelf got me in the google um in the keynotes at google io um they they had me coming in and shoot a shoot a bit of like um face to camera for them for that as well and that one what was funny about that is that that's just a trick right if you if gemini can't actually see video if you upload a video to it cuts into one frame per i think it's 10 frames a second and it just runs against 10 frames a second of images but the but the context length is so long you can fit an hour of video in like 2 million tokens it's astounding you know um so yeah the gemini stuff um the fact that you can do this with with like videos and images is super super exciting the um And recently, I noticed that Gemini now has bounding box support trained into it.

36:25So you can say to the model, give me the bounding boxes of the goats in this photograph. And it does it scale zero to a thousand on the X and Y axes, but it works. And so you can actually do something that's quite a sophisticated computer vision problem. Now, Gemini just does it out of the box as just one of the many things that it's able to do. The other models can't do this. Is that with Flash as well as Pro? Yes. Yeah. Both Flash and Pro have it. Yeah. And those models, like Flash is a very inexpensive model and it's a very high quality model as well. So I built a few things on top of the image models.

37:03I have this. So my dataset software is all about working with data in a SQLite database. One of the interesting questions is how do you get data into that database? So I built this plugin for it called Dataset Extract, where the idea is that you define a table. So you say, you know, I want the name and the address and the description and the star rating or whatever. And then you give it a bunch of unstructured text. You just copy and paste a bunch of stuff from like a web page and click a button. And it uses the sort of structured text extraction function calling mechanisms of the models to pull out those, to populate your table with everything that's in there.

37:39And so I got that working. And then I tried it against GPT-4 Vision. And it turns out that works really well too. So one thing I built with that was a thing for taking a photograph of an event flyer in a shop window. So you just take a photo of a flyer and you stick it in this thing. And it pulls out the title and the start date, the end date, the price, the location. And it actually can output that as an ICS file that you can then put in your calendar. So it's like a take a photo of a flyer, have it show up in your calendar. Apple had that exact demo in their thing yesterday. It was one of their Apple intelligence demos.

38:14I'm like, ah, I should have made a bigger deal of this before Apple stole my thunder on it. Because it's such an obviously good idea, right? Take a photo of a flyer, shows up in your calendar. But that was really easy to get working. And then I've also been looking at this. So the software I'm building, my primary audience is meant to be journalists. Like I want to build software to help journalists analyze data and tell stories. Turns out anything I build for journalists is useful for everyone else, right? There are no features that people outside of journalism won't find interesting. And I've been giving demos at journalism conferences.

38:48And so one conference, somebody gave me a page full of like some campaign finance documents, like a paper form handwritten filled in with who donated what money to what. And so I was on stage at this journalism conference doing live demos. And I fed that one into Claude III Opus and said, hey, extract the details of this campaign finance memo. And Claude III Opus said in front of a room of 100 journalists, I don't feel comfortable translating this document because it contains personal information. Why don't we have a debate about campaign finance reform instead? So it said no. It refused to do the useful thing that I asked it to do, which is very funny.

39:29But also, it's a real problem. In journalism, a lot of the source documents you work with are nasty stuff. And you kind of want these tools to help extract data from nasty stuff and not complain about it and sort of trigger their safety filters. But yeah, it's still really, really interesting. Another fun thing is if you upload a PDF into ChatGPT, if the PDF has selectable text in it, it can use it. If the PDF is just scanned notes, it can't do anything with it at all. if you take screenshots of the pages of the pdf and upload the screenshots it works because their vision model can work on images but it can't work on pdfs because they haven't wired that together yet which is frustrating because now you have to explain that to people like okay if you've got a pdf it might work it might not work depending on these arcane situations you haven't tried getting it to do it in code interpreter for you that might be possible right no that works well The problem is it's got Python libraries for dealing with PDF files, and sometimes it'll use those.

40:32But those are even worse because those can't do OCR either. So there are so many ways that you can mess up just with a PDF file. But yeah, so generally, I think the vision models, I don't feel like people have really even scratched the surface of what they're capable of. Those keep on getting better as well. I don't see many people. People talk about prompting strategies for text models all the time. where's the conversation about prompting strategies for image models like it's it's it's very thin on the ground at the moment there is one fun thing i was playing with yesterday um if you give these models a photograph that you've taken just outdoors somewhere and ask them to try and guess the exact location of this photo they're shockingly good like i took a photo of my garden i took a photo of my garden and it said well it looks like that's coastal california because of the the the the the moss hanging off the tree in this corner and the and the like and the style of the house in the background is characteristic of californian houses and then i gave it a photo of like a graveyard in england and it's like well that's an english graveyard you can tell because the style of the tombstones is that that kind of thing you get in rural england really fun and i mean it's creepy it's super creepy but this it's a great way again of just exploring trying to get a feel for what these models have been trained on and what kind of things they're capable of doing yeah yeah yeah you you mentioned a second ago the you suggested the development of visual prompting as a um you know kind of a field in the sense of like prompt engineering for text models like do you have a sense for you know what that might look like do you have like tips and tricks do you like What are your first entries in that?

42:17So a month ago, I was bugging this chap, Alex Albert. He's Anthropix's developer relations person. He's fantastic. And I was bugging him on Twitter about this. And he said, yeah, vision prompting has been a tough nut to crack. We found it very challenging to improve Claude's actual vision through just text prompts. But we can improve its reasoning and thought process. In general, I think vision is still in its early days. So basically you ask the experts at Anthropic for prompting tips for vision models and they're like, yeah, we haven't figured that out yet. And Anthropic have the best documentation for text prompting of anyone right now.

42:50Like the Anthropic prompting guides are absolutely superb. They still have nothing on vision because they haven't figured it out themselves. Like it's so early with the way the vision things work. So far, we've been talking about traditional prompt and response and running that loop with you as the human operator in the middle. What do you think about the agentic concept? And in the case of software development, like these tools, AI engineer and others, Devin, that aim to pull the human out of that loop to some degree or another? so i'm very keen on the the the llm in a loop mechanism right code interpreter the reason code interpreter is so good is that it can write code run the code see if it worked or not iterate again like that i'm totally sold on i hate the word agent because i think everyone you talk to about agents has a different idea of what it means and they don't necessarily acknowledge that other people have different meanings so if somebody says hey i'm building agents it gives me no information at all i'm like well i i don't know what your version of agents is so so i personally tend to avoid agent and agentic but if one of the definitions is tool use in a loop i mean really into tool use in a loop i think it's it's it's proven to work really really well the um the one that i've got the most experience with actually is the is github's copilot workspace which is i think still in a like in it's a beta feature that's where you can be on a github repository and you can basically prompt it and say, add this feature or resolve this issue.

44:27And it does pretty much the Devon style of thing where it says, okay, well, what's the current state? This function doesn't exist. There's no test for this. My plan is I will create this in this code. I'll create this in this code. And then it pauses and it lets you look at the plan. So you look at it and says, well, I'm going to modify these three files and add this. And you can edit that plan right there and say, actually, don't modify this file add a new file called this and then you click a button and it does the work for you right in front of you and you can see it like literally generating like diffs you can see what changed that ends up as pull quest and i really like that as a mechanism like all of these things you have to review them github pull requests are a great code review mechanism already so so that i'm sold on i like the idea of um i i feel like there is so much you can do with the prompt engineering around come up with a plan first, verify the plan, execute the plan, all of that sort of thing.

45:26So I think there's a huge amount of potential there. And we're beginning to see working versions of this. I've not played with Devin myself. GitHub, the Copilot Workspaces thing, I have shipped code that it wrote for me a few times now. And it's good. And often I'll get it to do me a pull request and then I will manually take over the pull request and I'll tidy things up and I'll change a few things and but again it's it's that weird intern again right so i now have a weird intern that can generate a multi-file pull request for me to then take over and and iterate on yeah i think cursor also has a beta feature i forget what it's called but it's also kind of their multi-file editor i think it's like compositions or something like that i've not played with it at all um so the other one the one i'm more excited about that i still haven't seen done perfectly is the research assistant thing.

46:20Like, um, if I want to, let's say I want to buy a home backup battery and to do, to do that, I basically have to go to 20 different websites and on each website need to click around and find the pricing and the comparisons and all that kind of thing. And so every now and then like chat GPT browsing mode can almost do this perplexity can almost do this to date. None of them have quite nailed that enough that I don't feel like I basically have to then go and repeat the work myself. But that's the dream, right? The absolute dream is that you can come up with quite a complex multi-step research plan that involves visiting dozens of different websites and figuring out data from different tables and then pulling it all together in one place.

46:59And they can so nearly do it. Like I try this every month or so, I try out that kind of prompt. And sometimes you're like, oh, I think they've nailed it this time. And then you look into it and one of the details was hallucinated or they didn't properly follow the links to the things that you need at all and that's frustrating because that would be such a huge productivity boost if we can get to a point where we can reliably have these models go away and do that kind of research for yeah and there have been a number of uh papers on the academic side we've talked about some of them in our community uh like gen ai meetup um that propose like iterative loops uh and i think even some of the like lang chain and um some of the agent agentic tools like these are some of their canned demos as well um but that's the problem is i've everything i've played with has felt like a really good demo of something that isn't quite good enough yet like in all of generative ai get it like getting rag i love rag because building an initial version of rag can take a couple of hours getting to a version of rag that you could deploy takes another six months because all of the it's such a there are so many tiny little details and ways it can go wrong that you have to work around right and on that rag topic in particular i usually talk about the that gap that you're referring to um arises often arises because people kind of over index on the g the generation and not the r the retrieval and like search and where that data is coming from.

48:32Do you spend a lot of time exploring, incorporating search and relevance and retrieval into your projects? So I love search engineering. I've built lots and lots of search engines in my time. I've used, you know, solar and elastic search. And these days I'm using SQLite itself because the SQLite database has quite good full text search built in. Like every Python developer has a full text search engine in the Python standard library that they probably don't know about because they haven't looked into SQLite's FTS extension. So I do a lot of work with that. And then over the past year, I've been playing a lot with this sort of the vector embedding stuff as well.

49:11Like I wrote a bunch of, I've got a bunch of open source libraries I built around like working with vector embeddings. What I find interesting is, and so search retrieval is an information retrieval is an academic discipline going back what 40 years. And we know how to build a really good search engine and then i feel like a lot of people who get into rag they go straight to the vector stuff and they miss all of the full-text search bm25 like tfidf all of the boring old search thing which is still massively relevant like um everyone is trying to figure out how to write evals for like evals for llms search engine people have been figuring out how to do automated evaluation of search quality for decades like there's so much from those two worlds that can be brought together.

49:57So yeah, the retrieval side of things really matters. Also, I feel like the problem with RAG, though, is it isn't actually a relevance problem. It's the problem that you build up a whole RAG system, and then somebody says to it, what did Simon write about last year? And there is no vector search or full-text search in the world that can answer that question, because you need to know, oh, in that case, I need to constrain it to where year equals 2023, and then you'll get back too much information. And that's the problem. The problem with RAG is there are so many questions that a user might come up with that the system just won't be engineered to be able to handle at all.

50:33How do you deal with that? There's a fun problem with the vector search thing as well is vector search, because it works on distance to your search terms, if you ask for 10 results, you will always get 10 results, even if there was nothing relevant in the document corpus at all. like that that's at least with full text search if you ask for 10 results and there are no matches you'll get zero matches somebody did a wonderful demo um against one of my projects a little while ago they were building a rag system a question answering system they loaded in the documentation from one of my projects my llm project and then they asked it what is the meaning of life and the answer they got was as a as a sentient hamster i really care about snacks so the meaning and it's like what the hell happened it turns out in the documentation for my tool i had this one example where i said hey tool pretend to be a hamster and talk about snacks and it had and if you look at if all of the chunks in my documents if you say what's closest to the meaning of life the hamster who loves eating snacks is the closest to the meaning of life so that's the one that came back.

51:41Just hilarious, but also a great illustration of why this stuff is really not. Because it's so open-ended, because you're saying, we'll answer any question in the world, well, good luck even testing that. It's very, very challenging engineering. Does SQLite, like all of the rest of the databases, have vector extensions and that kind of thing? Does it have its own ecosystem around vector? It does. And that's a friend of mine. A friend of mine, Alex Garcia, has been building the SQLite vector extensions. There's a couple of other ones out there now. They're very good. I'm not convinced by vector databases as a product because I think vector lookups is an index type in a normal database.

52:23Postgres has PG vector. SQLite has SQLite Vec. Now, I believe Elasticsearch has added vector indexing and so forth. That makes sense to me. I like relational databases where I can do nice, reliable filters and then add a and order by distance from this embedding thing. And yeah, that ecosystem, it's quite new. Like most of the tools I just mentioned are less than a year old, but they're getting pretty good. Like the computer science find is actually pretty well established for fast cosine lookups and so forth. And is there a standard way for doing a hybrid search that incorporates full text and vector?

53:06Or is it kind of getting two result sets and trying to integrate them some way? That's a great question. I don't have an answer for that yet. I've been pondering that myself. Like, clearly, people out there have figured this out. I don't think there's a... I've not seen the definitive answer where if you ask me that question, go, yeah, go and read this blog post and it describes how to work it. But you can absolutely do it. Like I've built some very basic versions just with the SQL union where you do the FDS search and then you union against the vector search. But again, this is really where we're getting into the dense field of information retrieval.

53:37If you found a search engine expert and asked them that question, they'd just tell you. Rattle off 10 different ways to do it. Yep. These problems are solved in communities that are out there. We just need to get them to tell us what to do. And one thing we haven't dug into yet that I wanted to chat about is like the ecosystem of open source models and LAMA in particular. You've explored that a bit. What is your kind of current thinking about how that ecosystem is evolving and what are you excited about? So I'm very conflicted on this one. On the one hand, I love local models. Like I've been one of my open source tool, LLM, this command line tool for talking to models.

54:21One of its features is it has plugin support and you can add plugins to it that let you run local models. So out of the box, you can use it to talk to OpenAI over their API. You can install the LLM GGUF plugin and download a Lama model and you can just run it locally on your laptop. And that works astonishingly well. Like it keeps on getting easier to run these things. the reason i'm conflicted is that they're they're just not very good compared to the big hosted models like i use clawed 3.5 sonnet on a daily basis there is nothing i can run on my laptop that even comes close to the quality that i get from that so despite having the side project when i'm building like software for these things i'm not actually using them for the most part because the because the hosted ones are very inexpensive and really really good now that's changed slightly like the llama 3.1 release from a couple a few months ago those models are getting really good like um llama 70b runs on my mac it takes up almost all of my system my system memory so i can't it crashes other applications which is why i don't run how much memory do you have um i've got i only got 64 gigabytes and i'm regretting it i should have got i should have gone for way more um so llama llama so the 70b model feels equivalent to or maybe even better than gpt 3.5 did um the 405b llama model is a gpt4 class model so nowhere i could run it on a laptop like that you see i think you need to spend fifteen thousand dollars on hardware to run that on your own equipment which is amazing that that's now feasible so you know it's not really hobbyist levels but if you've got a lot more organizations can afford to run that now um so that's exciting Because one of the things, like a year ago, actually six months ago, GPT-4 was completely uncontested, right?

56:14There was no model out there that was even remotely close to GPT-4 in capability. And it held that position for like 12 months. And that was kind of upsetting. You know, you don't want this incredibly sort of transformative field to be controlled by a single organization. That is no longer a fear at all. Claude III Opus was the first model that came along that felt like competitive GPT-4. Now we've got Gemini 1.5, we've got Llama 403b. I'm no longer worried that there's just one company that's making the best models. But also, you kind of want this stuff to, you want to be able to run this stuff yourself.

56:49The 8b models, like there are models which are just a four gigabyte file download. And those are actually very, very good. Especially when you think that there are sort of two different ways to think about language models. There's the thing where they know everything, right? like claude 3.5 sonnet and gpt4 know so much stuff about the world i can ask them for restaurants in my local town and they spit out a list of restaurants which is what's that even doing in there why why would that have made it into the training data um then the other side of them is when you're using them as language manipulation engines like summarization or churning out code to a certain extent or extracting facts and for those i don't care if my model knows the um knows who the king of France was 400 years ago, I want it to be able to translate English into Spanish or whatever.

57:37So it needs a bunch of knowledge, but it doesn't need that sort of Wikipedia-level encyclopedic knowledge. The small models that run on your laptop, that's what you're getting from those. It can summarize things. It can extract facts. It can maybe generate basic code for doing mathematics and things. So if you combine that, the ability to do language with the tool usage, if you can have a small model that can run a Python interpreter, or open your garage door or whatever it is, that's super interesting. And that's where we are now. Like if you want to do a home automation project that's driven by an LLM, the models that will run on a Raspberry Pi are almost good enough to do that.

58:18And they also get better. Like I've got the same laptop now as I did a year ago. The models I'm running on it are much more capable because the model, they keep on finding new tricks to make them smaller, to make them faster, to sort of specialize them for those kinds of tasks. So it's all really promising. I'm just frustrated that despite all of those advances, on a day-to-day basis, I'm using the hosted models because they're so, so cheap and so good. The other thing to consider is the pricing. Lama 3.1 is now, there are dozens of companies that provide APIs for that, and they're all competing on price, and the prices are dropping through the floor.

58:54It is astounding how inexpensive... Meaning for hosted access to Lama 3.1. The hosted access. There's also, there are companies that are like making custom chips to speed this stuff up. There's a, I think it's Cerebrus AI. Cerebrus. The physical size of their chip. It's like, I couldn't believe the photograph. I didn't know you could make a chip that was like larger than a dinner plate. And they offer like a thousand token the seconds of inference. And I don't know if they've announced their pricing yet, but that's going to be pretty inexpensive as well. That's amazing. I love that because Llama is an openly licensed model, the competition from different vendors is just driving that price into the floor, which is great news for those of us who are building on these models, especially since there's no risk in building on Llama because if the vendor you're using shuts down, it's the same model.

59:46Switch to somebody else. Run it on your own machine if you have to. I just had a thought based on that last comment about the extent to which you change your prompting approach explicitly from one model to the next. Do you think about, this works well in Claude, but if I'm prompting OpenAI, I do it slightly differently? Or do you just kind of talk to it naturally and iterate to get to where you need to get? So surprisingly, I use the same prompting style with all of the models. which i was not expecting i thought that i'd be building up that sort of intuition oh this works better with this one this one works with that but what's actually been happening i feel like my prompting style keeps on getting more and it gets shorter and more succinct like i'm not really into all of those pretend that you're an expert accountant and all of that kind of stuff i find that i i my ideal prompt is just like two or three words if i can get away with it like summarize this colon paste in some text that kind of thing um and a lot of it also comes down to this idea of filling in the context with examples i think examples are the most powerful prompting technique that there is so rather than trying to construct several paragraphs of instructions just throw in a couple of examples of what you want and you'll get get great results that's also the way to build software against the models like when i'm building features that are powered by models the prompt often shrinks down to get to something pretty small.

1:01:13And I throw in three or four examples instead, and that gets me the results that I need. At the same time, every now and then, one of the big AI labs will release some of their prompts, either accidentally, it'll leak out, or Anthropic actually do publish their prompts now. That's in their documentation. And those are quite long. And those are really interesting to read because clearly they've got the best prompt engineers in the world, and they are producing like five or six paragraphs of prompting. It's also fun reading those. And for each sentence in it, you think, oh, they have to work around a problem there.

1:01:46Like when it says, output markdown, we really mean it, definitely output markdown, all of that kind of stuff. um but yeah so so weirdly prompting is weirdly transferable like i've seen i've heard people who work in ai like research saying it's interesting how this stuff all seems to like seems to come together like the different models being trained on different training sets and so forth their behavior still ends up more or less the same maybe because they're all just scraping the same web data but increasingly these days they're doing licensing deals there's all sorts of like the secret source of an ai lab is the training data associated press licensed 100 years of and so open ai licensed 100 years worth of associated press journalism is that in their training data i'm pretty sure it is um all of that kind of stuff so yeah it's um it's weird that the prompting is not as different between the models as i expected at the same time if i'm building a feature that's what i'm iterating really heavily against an individual individual model like i was building against claude 3 haiku a lot because it was the cheapest these i'm considering a gpt-40 mini is even cheaper as of a few weeks ago so i've been using gpt-40 mini for for stuff too and yeah it's it's it's it's it's it's weird it's it's definitely a craft there's also the whole issue of automated evals like to make sure that you you really are being scientific about how you do this i'm not using those properly yet.

1:03:17And that's my greatest regret as an engineer is that I haven't got a really good pattern for eval setup. And I'm working towards it. At some point, I'd like to be able to ship features that I know have been evaluated against the latest version of the models. And if the new model comes out, I'll see if it works better, but I'm not quite there yet. Another random question. You mentioned scraping. Do you have a preferred scraping stack or setup is it the usual kind of beautiful soup for you know public and selenium kind of thing for private or like how do you do a lot of that i do i do a lot of scraping um i actually skip i go straight to playwright these days which is more so selenium was first then there was puppeteer which was the team at google chrome playwright is basically puppeteer two in that a bunch of the engineers who worked on Puppeteer switched from Google to Microsoft and rebuilt Puppeteer.

1:04:12So Playwright is sort of the accumulation of like a whole generation of expertise in this. I love Playwright. It's like Selenium, but they ironed out all of the frustrating pieces. And it's also, Playwright works really well. You can call it from JavaScript, you can call it from Python. I built my own little tool on top of Playwright called ShotScraper, which is a command line tool for taking screenshots of web pages, but also for running JavaScript against them. So you can say shop scraper URL to a page and then give it a string of JavaScript and it will run that in the browser and output the JSON directly to your terminal.

1:04:46So it's basically a command line scraping tool that you can put into a pipeline with a bunch of other things. Love using that. A lot of my scraping as well, I do just in the developer tools console in my browser because you can open that up on a web page that you're already logged into run a bunch of document.query selector things to pull data out and then just copy and paste it out of there so that's then the other trick i use is um gina.ai have a thing called gina reader which where you can say r.jina.ai slash url and it will take whatever url you give it load it up in i think they're using puppeteer pull out just the the content and give it back to his markdown so it's a url that turns anything into markdown a markdown is still the best format for feeding into language models so you can take any document get the markdown out paste the markdown into a language model and work with it that way but yeah it's kind of fun how scraping is having a little bit of a resurgence now because anyone who's working with language models wants to scrape data and if you feed in just the html you get like two megabytes of stuff that you don't need um it's a fun time also language models are really good at writing scrapers for you you can paste in that html and say write me javascript that pulls out the titles and the descriptions of all the news articles and it'll spit out four lines of javascript to do what you need so i guess as we kind of uh start to wind down i i'd love to have you riff on how you keep up with everything um you know the space moves so fast are there yeah i talked to some folks who like you know build their own tools you clearly are doing a little bit of this you know it sounds like um you know have you built like you know twitter summarizers llm you know rss feed summarizers that kind of thing or is twitter you know what's your go-to news source like how do you think about you know staying at the at the edge so the best tip for staying at the edge is to write about this stuff because i've got so i update my blog every day mostly with ai llm related stuff as a result people ping me people send me information all the time which is great um i do use like twitter i i'm on i'm on mastodon and um threads and twitter twitter is where the ai community are mostly hanging out still so i follow a bunch of people there but also um i turn on notifications for specific accounts.

1:07:19So like the OpenAI developers account, the various developer advocates for different models, those I get push notifications on my phone whenever they tweet, which is such a cheap trick. And it works so well. Like if a big new announcement comes out from Google Gemini or Anthropic, I hear about it within three seconds of the announcement going out. So that helps a lot. I'm in a bunch of like private discords that have various people looking at this from various different angles so those are great as well like um that the sort of the private discord there's a whatsapp group i'm in but mainly it's um mainly it's it's mainly twitter and people knowing that i'm interested in things which means that they ping me about stuff that's interesting like that's that's a superpower honestly so few people are doing long form blogging about this stuff right now that you can have a major impact on ai just by writing about it somewhere.

1:08:14Like just by being one of the few people that will write four paragraphs of text about the latest model release, as opposed to just, just like tweeting about it somewhere. What are you looking forward to? What are you, what are you, um, I I'm curious, like your take on the future, it doesn't necessarily need to be crystal ball, but like, you know, what, where you think things are going, like, how do you think about the trajectory? so i'm very i try to avoid the futuristic thinking because it's just so much noise about it like any ai discussion where people like yeah well we'll have agi next year and none of this will matter it's like well i mean maybe but there is so much interest to be had in what's available right now like that we've got models from like gpt4 is a year and a half old we still don't know what it can do and how to get the most out of that so my writing and most of my thinking has been focused on just razor focused on what's what's available right now what can we start working with that said i am very excited about the claude 3.5 haiku and opus models because sonnet was so good like sonic sonnet 3.5 came along and it was such a note like it was the first time since gpt4 that i felt like there was a really visible difference in quality opus is going to be a is should be a much better model than that.

1:09:33Haiku should be a much cheaper model. So the idea that we could get that sort of like 3.5 sonnet style thing at Haiku prices, that's really exciting to me. OpenAI are clearly going to launch something in a couple of weeks. They've got their Dev Day event coming up. I imagine that will be quite a substantial release of some sort. I've been avoiding all of the rumors because the rumors don't really tell you anything. So I'm professionally interested to see what happens when that comes out. The thing I most want, I don't know if I'm ever going to get, is I need a way to turn the safety filters off on some of these models for journalism.

1:10:09I need to be able to take video footage of a police brutality incident and feed that into a model for analysis and not have it refuse to answer questions about it. That's particularly difficult because on the one hand, AI labs love the idea of people doing investigative reporting with their models. They're always looking for positive stories about AI usage. Investive reporting is a great one. On the flip side, try telling OpenAI that they should give the unfiltered model to journalists who write stories about AI. The last thing they want is people getting it to spit out a recipe for a Molotov cocktail and then publishing it on the front page of a newspaper.

1:10:49So it's tricky, right? And I'm hoping that that's where the openly licensed models get really interesting. A fine tune of Lama with all of the safety filters turned off would be a genuinely useful thing for people who are trying to report on toxic content on fascist message boards and things like that. So that's something I'm constantly holding out for. And then there's also the most interesting problem in vision for me is you've got a crap table of data from like a PDF and you want to turn that into CSV. And the models are getting closer and closer to being able to do that. Like every time a new vision model comes out, I'll feed it a big complex scanned document, a table and see if it can get me the right results.

1:11:33I need to update my knowledge more recently. They're getting so close and that's going to be transformative. The amount of data about the world that's locked up in crack tables in a PDF file if we can suck that out and turn that into a sqlite database that's amazing and all of my other tooling comes into play what's your crap table is it something like from a financial document or like exactly it's whatever whatever whatever i've seen most recently actually the campaign finance reports are an interesting example of that well again again some models ethically refuse to deal with those because they have personally identifiable information in them which is frustrating because that's that's what the story is the story is who donated what money to who um so yeah there's there's lots of things like that generally though i'm just really enjoying watching i love that i feel like that we're at a point now where the new models that come out are incrementally better than the previous ones and i love that you know i don't have to rethink everything about how i use these tools it's just like okay well the things that it couldn't quite handle last week, now it can handle.

1:12:36Ethan Mollick suggests anytime an AI fails to do something for you, save that thing in like a notes document somewhere. And then in six, when new model comes out, try it again and just use that as your personal benchmark of how these things are improving. I haven't started habitually doing that. It's a great idea. I need to do that. Well, Simon, thanks so much. It's been a great pleasure. We touched on, I don't know, maybe 10 % I know the overall things I've tried to keep it like related to AI. I'd love to dig into Django and of and all these other things that I see you writing about that I'm curious about.

1:13:14But hopefully we can do it again sometime and dig into another corner of mutual interest. That would be awesome. This was really fun. Thanks a lot for having me. Thanks so much. Take care.

1:13:32Thank you.