Autoresearch clearly explained (why it matters)

0:00Andre Karpthy:Andre Karpthy, I mean one of the godfathers of AI has just launched something called auto research and auto research is a huge deal and it's going viral on Twitter. And I just wanted to do an episode where I can explain to you in the clearest way possible what it is, what are the use cases, how to make money from it, how to be more productive with it, how to create impact with it. And by the end of this episode, I'm going to give you a bunch of different ideas, use cases for how to use auto research. I'm going to explain it to you in the most clear way possible. And at the end, I'm going to tell you how you can actually get started with it.

0:35Andre Karpthy:So let's go right into it.

0:44Andre Karpthy:So what is auto research? Well, it's like having a super nerd robot intern that runs science experiments on AI models for you all night without you doing the boring stuff. I mean, sounds intriguing, right? So how do you actually program it or get started with it? Well, the first thing is you've got to give it a goal. So you can say something like, make this small AI model smarter. That's the goal. And then an AI agent will actually plan what to do, like different settings, code changes, edits the Python code for you, runs a short training experiment on a GPU for about five minutes, it reads the results, and then it decides what to change next and to repeat the loop.

1:27Andre Karpthy:So in some ways, if you've seen my video on the Ralph loop, where it basically would do engineering 24-7 and you'd wake up to new stuff happening, in simplest terms, that's what auto research is helping it do. You give it a goal, the AI agent does a thing, you tell the AI what better means, cheaper leads, more clicks, higher sales, better model school, And then the AI keeps changing things, testing them, and it only saves the changes that improve. So what's really cool about it is you wake up, you grab the best version, and then hopefully you turn it into something you charge for or, you know, you give it away.

2:07Andre Karpthy:I saw this tweet by Toby, who's the CEO and co-founder of Shopify. Auto research works even better for optimizing any piece of software. Make an auto folder, add a program MD. that's just a markdown file which is really the foundation of what you know how you're going to be using auto research and a bench script make a branch and let it rip so that's why i started paying attention to auto research right when andre carpathy legend and toby's and and more people you know start playing with it i'm like okay i gotta pay attention so i created this little visual for for how to think about what auto research is so you set the goal uh the the ai plants and experiment.

2:48Andre Karpthy:It edits and trains the code and settings. It runs a short training on a GPU. By the way, this is an important, I should mention that you need a NVIDIA chip to actually run auto research, or you can do it in the cloud. I'll talk about this at the end of the episode, but you do need that. You can't just run it on, let's say you have a MacBook M1 or something like that. It reads metrics. It says, is it a better result? If it's not, it's going to log the attempt and it's going to discard the config. If it's yes, it saves it to the config and then just plans a different experiment and it just, you know, hopefully gets better on your goal, whatever it is.

3:31Andre Karpthy:So let's get into, we're going to get into some of the ideas, business ideas around it. But right before that, I just want to say, here's a simple mental model for how I'm thinking about auto research. So imagine you have a research boss you can boss around. Number one, you write a clear task. So for code experiments, maybe it's improve this model test score for business, figure out the top five competitors for product XYZ and make a short report. Step two is you give the bot access to the code, a GPU for ML experiments. You obviously need to give it access to the internet and documents if you're doing reading tasks.

4:14Andre Karpthy:The bot then runs a loop. So it plans, it acts, meaning it might run code or search. It reads results. It updates the plan. And then you just come back later. It could be 12 hours, 20 hours, 6 hours. And you see if it's logged everything, charts and metrics. And then it gives you a written summary in normal language. So think of auto research as a research bot that runs experiments for you while you sleep. tries lots of ideas fast and keeps the winners. Quick break to invite you to something. Now, this isn't an ad. I just want to invite you to a free event because I think that you're going to get a lot out of it.

4:50Andre Karpthy:I wanted to take one hour of time where we just talk about building businesses in the age of AI. People say SaaS is dying. I actually believe the quite opposite. I think that SaaS is just evolving. I think right now is an incredible time to be building software startups that help you craft your dream life. And for all those reasons, I said, let's just book one hour of time. It's going to be 11 a.m. March 12th. That's a Thursday where we can go and lock in and just talk about building businesses in the AJI. I'll include a link in the description in the show notes to join. And I can't wait to see you there.

5:26Andre Karpthy:Okay, how do we use it? Here's some ideas for you. So the first idea for you I have is a niche agent in a box, you know, products. This can be multiple products. And by the way, I put out these ideas. I want you to do these ideas. I think that even if they don't turn into businesses, you will learn about these tools and that is going to help you outperform 99.9 % of people on this planet. So you package tiny auto research loops tuned for one painful niche. So the example I think of is an Amazon listing experimenter, an email sequence tuner for realtors, a pricing optimizer for SaaS. Those are auto research loops and ideally in a niche that you understand well.

6:12Andre Karpthy:And then you charge a monthly fee. So the value prop is this thing runs experiments for you 24-7 and just shows you the winner to click accept. How valuable is that? And how many different niches are there that this plays into? The hard part is figuring out what's the pain points and then obviously you want to be quick to market. So here's a visual of it. Pick the painful niche, design the tiny auto research loop, run experiments automatically, see which setup works best, turn best setup to a simple agent product, and then you charge that monthly subscription. Number two, you're going to want to, here's an idea, print money using an A-B testing for marketing.

6:53So it's very similar, but instead of doing it for realtors or whatever, you're doing it for ads and landing page experiments.

7:06Andre Karpthy:So landing pages, so the agent writes variants of headlines, layouts, and offers, pushing them to traffic measures, which one converts better and keeps iterating. So this is like conversion rate optimization around landing pages. Think of tools like Optimizely. That's a SaaS tool that when I first moved to San Francisco, I remember how big they were and everyone was talking about Optimizely and A-B testing and it's like, well, this is the future of that. Auto-research for different landing pages. You can also use auto-research for something like ads, which auto-tests creatives, it auto-tests angles and audiences and then it keeps the combos that lower CAC or raise ROAS.

7:48Andre Karpthy:So you profit by running this for your own products. If you want to build your own products and just use this internally, that works. Or offering an always-on experiment engine to clients as a retainer service. For 5K a month, I'm going to give you the best landing pages every single month and it's just going to come to your inbox, that sort of thing.

8:08unknown host:Visual of it, business goals.

8:11Andre Karpthy:The goal that you're giving the auto research is more sales. It's generating things like pages and ad versions. sending traffic to the versions, measuring conversion and revenue. Does any version beat the current best? If it doesn't, then you're going to keep the current control. But if it does, you're promoting the winner to a new control and you're asking the AI for new ideas. All right, hope your creative juices are starting to get flowing. You're starting to understand a little bit more about how it's working, how you think about goals, how you can think about agents, and how you can set up these loops.

8:45Andre Karpthy:Number three, research as a service. Auto research's recipe is basically a loop for doing research. Because you're searching, reading, summarizing, and you're comparing, and then you're repeating. How do you point that at money problems like market and competitor research for startups? Constantly updated reports on who's doing what, pricing, features, and gaps, super valuable. Investor and M &A decks, fast technical and market diligence summaries, super valuable. Compliance and regulation tracking for niches. I don't know, crypto, healthcare, finance, super valuable. So you can charge per report, like a one-off, or you can set up a monthly subscription for always fresh dashboards.

9:28Andre Karpthy:So visual, define client research question, auto research searches and reads, summarize and compare findings, creates reports and dashboards, deliver insight to client, and the client pays per report or monthly, whatever you decide. Number four, power tool inside your own product. So if you already have built a SaaS or workflow, embed an auto research style agent so your users can press optimize, just like a big, I envision like a big button that just says optimize. And the system runs a mini research loop for them. So for example, tune prompts, pick best pricing, rank suppliers. Then you can charge higher tiers for this feature or you can use it as a wedge to upsell pro and enterprise plans.

10:12Andre Karpthy:So maybe that's a part of Pro and Enterprise. Maybe it's something that you just send an email to your entire list and you're like, hey, we have this really powerful tool. Imagine you press this button. It's like bending spoons, right? It's like bending spoons. Bending spoons is not the private equity group I'm talking about. The idea of you can bend a spoon, right? It's incredible that you'd be able to optimize, press a button, and this would happen. So visual over here, have an existing SaaS, add an optimize button. Users run many research loops. Tools suggest better settings or prices. Users see better results.

10:47Andre Karpthy:Offer higher price programs and enterprise plans. Number five. This is a saucy episode, by the way. This is saucy. All right. Agency that sells, we run more tests than anyone else. Because auto research lets you run hundreds of experiments instead of a few, you have a simple pitch. We do 100 times more testing than other shops for the same or lower fee. A niche example, a Shopify store conversion lab, B2B SaaS pricing experiment service, email subject line and sequence optimizer. You charge per month and a bonus if you hit specific KPI lifts, rev share performance fee. People love that. Of course, they're going to be interested in, yeah, if you can lift this KPI, we'll give you some bonus.

11:31Andre Karpthy:So here's the visual. Start an optimization agency. Use auto research to run many tests. improve stores, pricing, emails, and funnels, show clients more experiments and wins, charge monthly retainer and performance fee. Number six, and we've got about 10. So we're almost done. And then after we're going to talk about just some cool, interesting stories around auto research. And then I'll end with how you can set this up very briefly. So auto quant for trading ideas. So you can use auto research to run small, fast back tests of many simple trading rules. So LLM based, factor screen, sentiment filters on one GPU overnight.

12:18Andre Karpthy:So you can keep the few strategies that look promising, then either trade on your own account or sell signals and strategy reports. So depends if you're a trader, maybe you're doing yourself. Or yeah, you can just, you know, sell this as a digital product. or basically a digital product. So you define the simple trading rules, you run many back tests overnight, you review the strategy performance, you keep only promising strategies, trade your own capital or you can sell the signals. I think finance is changing a lot and I think with things like auto research,

12:55Andre Karpthy:it's going to be an unfair advantage for a lot of people. So I think you're going to see a lot more digital products that people sell and also just using their own money, trading themselves instead of giving 1 % or whatever to a financial advisor. I'm sure also, by the way, a lot of people are going to get burned by this too. They're just going to blindly just trust in auto research. You need to have a human in the loop and you need to manage that, obviously, accordingly. But yeah, you can just see, there's definitely going to be some people who are going to get burned. You just give a bank account and just let auto research trade for it.

13:40Andre Karpthy:I mean, it would be an interesting test, that's for sure. Number seven, always on lead qualification and follow-up. Point an auto research style agent at your CRM, so like a Salesforce or something like that, and inbound leads. Let it test rules and messages to see which leads are most likely to buy. It auto-grades the leads, suggests next actions, and drafts follow-ups. Salespeople only focus on high-value deals, so it's more revenue per hour spent. Visual over here for you. Connect to CRM, auto-research test the leads, rank leads by likelihood to buy, draft follow-up messages, sales focus on best leads, revenue per sale increases.

14:22Andre Karpthy:Eight, finance ops, autopilot for businesses. Use the loop to grind through invoice matching, expense report generation, and exception detection with continuous small and proven rule and prompts. You can sell this as we cut your AP expense time in half, either as software or as an op service with a small team and agent. By the way, I can totally see someone starting this and this gets acquired by one of the large fintech companies or one of the large banks. so visual here ingest invoices and expenses the auto research improves rules and prompts matches invoice and detects exceptions it generates clean expense reports reduces manual finance work and then you can sell it as a software or op service or you start maybe you start as op service and then you kind of evolve into the software two more for you number nine an internal productivity lab for your own org i thought this was interesting so treat your company like Carpathie's GPU lab.

15:20Andre Karpthy:Define KPIs, so like response time, close rate, ticker resolution, and let agents iterate on workflows and templates and routing rules. So you just get fewer meetings, less manual grunt work, and then you personally touch only the high impact decisions when everyone else rides the improved process. So the goal here is defining the key metrics. Auto research is testing the new workflows. It's improving templates and routing rules. you're cutting meetings and manual tasks. That's good. Team focuses on high impact work and then higher productivity and ideally higher profit. Last idea for you, done for you research or due diligence shop.

15:58Andre Karpthy:So you use the research loop to chew through docs, filings, product pages and reviews and keep an evolving living memo for clients like investors, acquirers, execs. You make money by selling fast, well-structured briefs and a monthly update pack instead of one-off manual research logs. The goal, get an investor or acquire a question. This happens all the time. Auto research reads through docs and filings. It summarizes that product market and risks and maintains a living memo for the client. It delivers a brief and updates packs and the client pays for reports and ongoing access. I would pay for something like this.

16:39Andre Karpthy:Hopefully someone builds it. All right. So those are a bunch of ideas for you. I also saw a couple of interesting things this morning. My good friend, Morgan Linton, who's been on the pod before, he says, I woke up this morning and all I can think about is auto research. So many ideas swirling around in my head. Not sure 99 % of the world realize the incredible breakthroughs Carpathia is making and just sharing casually on X. Right now where my mind is going is medicine. It feels like in many ways, clinical trial design is itself kind of like a hyper parameter search. I know right now trials cost tens of millions of dollars minimum.

17:20Andre Karpthy:It feels like an agent swarm could optimize treatment protocols on small proxy experiments, promote the most promising candidates, and then move to humans to review. So humans still very much in the loop, but later on and experimentation going much deeper, happening faster and for far less money. I think for me, while I'm not a doctor, he's an engineer. What I'm the most excited about when it comes to AI is the impact it will have on human health and critical areas like disease treatment. It might be a crazy idea, so a real doctor can jump in the comments and slap me on the wrist here. I looked at the replies.

17:56Andre Karpthy:I didn't see any doctors come in. But I don't know. I just kind of ended up thinking about what Carpathia has discovered here could have some pretty profound implications. so only halfway through my coffee though but woke up this morning and this is what I'm thinking about so thought I'd share I agree I think there's a lot of really interesting not just like business profit ideas but also just like medicine science research so I'm excited for people to take this and to continue with it I also saw this tweet here what's after auto research it's Carpathia's new open source project agent hub So Karpathy also launched AgentHub.

18:36Andre Karpthy:What is AgentHub? It's GitHub for humans. Sorry, GitHub is for humans, AgentHub is for agents. So it's basically a GitHub for agents. An agent swarm collaboration platform, a very promising direction. I'm watching him speedrun a one-man billion dollar company. If you look at the GitHub for AgentHub, it says first use cases for auto research, but it's a lot more general than that, exploratory project.

19:12Andre Karpthy:He says,

19:35Andre Karpthy:So maybe you've gotten to the end of this episode and you're kind of like, okay, I think I understand what auto research is. I think I know Carpathie's a G, Toby's a G, all these smart people are playing with it. How do I get started? Well, to get started, I'd recommend just tell Claude Code to get you started. So I went ahead and I basically was I gave Cloud Code this GitHub repo, the auto-research GitHub repo. Wow, 25 ,000 stars already, so this is crazy. It's really growing quick. I gave it the link, and I was just like, I need help installing auto-research by Carpathie. and it says here's how to install it and set up auto research by Carpathie.

20:36Andre Karpthy:You need an NVIDIA GPU. So I talked about that in the beginning. It was tested on an H100 but other NVIDIA GPUs should work and you need a UV package manager. So you have to install UV, you clone the repo, you install the dependencies, you prepare the data and run a training experiment. In my case, I don't have an NVIDIA GPU. I'm actually using a MacBook and an M1 Pro. I know I need to upgrade to a new Mac. So I was like, so wait, I need an NVIDIA GPU to do this. But there's a few options. Cloud GPU, so you can rent an NVIDIA GPU from a service like Lambda Labs, Vast AI, RunPod, or Google Cloud.

21:25Andre Karpthy:Some offer free tier with GPUs. This is the most straightforward path. So that's the answer to people who don't have an NVIDIA chip. Just rent it on one of these services. I personally use Google Collab. Why? I just know Google the best and trust Google the best. It also says you can try it via Apple Silicon, via an NPS backend. I'm like, no, I'm not going to do that. So that's what route I did. I went on Google Collab. The easiest way to get started, you go to collab.google.com, You create a new notebook, you change the runtime to change runtime T4 GPU and you run a bunch of commands. That might be complicated, sound complicated.

22:09Andre Karpthy:This is what collab looks like. You listen to what Cloud Code tells you to do and you just paste it in and you can get started. If people are interested, I can spend more time I'm with this, with auto research, as I'm learning, sharing more about it. But I just wanted to give you a quick primer on what it is, why it's important, what are some ideas on how you can actually use this thing, and then how are people installing it. You can use Cloud Code as your helper to get it installed, and you're going to want to rent a GPU in the cloud, at least to start. So hope this has been helpful. This is another solo podcast that I'm doing on the Startup Ideas podcast.

22:59Andre Karpthy:The last time I did this last week, I had a lot of comments that said, yeah, Greg, I actually really like when you just come in solo and just start telling us what's on your mind and stuff like that in real time. So I'm here. I read every single comment. So keep commenting, keep liking, keep subscribing, and I'll keep putting this out there for you for free. Yeah, I'm excited to see what you end up using this for. of course it's early right like this is this is brand new people are still trying to figure out what are the use cases but i always find that you know in the in the fog in the fog people don't really understand where the opportunity is is when there's sometimes an opportunity so one thing i've just learned in my career is just like when i see people like carpathy doing things like this you want to pay attention you want to tinker with it you want to have some fun with it and you want to see what it's all about so thanks again for you know giving me your time um hope this has been clear share this with a friend uh who you think would see it valuable and if you need any if you need any ideas more ideas on startups to build uh you know with ai ideabrowser.com definitely your place to go and i'll see you in the comment section and i'll see you next time you know have a creative day