“Vibe analysis”: How Faire’s data team uses AI to investigate conversion drops, analyze experiment results, and convert raw data into executive-ready insights

18:38Tim Trueman:It can interpret what the customer facing experience and intention would be. And then you can create these really interesting business and market facing assets out of that. So I just think the ability to query your code base and your GitHub history for any use case, including this one is really useful. Yeah, I love that. Great. Now, what do we do after this? So you've identified, you have a conversion rate problem, you've identified maybe a couple sources of the issue, you're going to go talk to your colleagues, you're going to look at the code. How do we actually do some analysis? I know we said we were going to do some vibe analysis, and we have seen very few numbers.

19:12Tim Trueman:So Alexa, how do we do actual analysis of data when we've identified a problem or an opportunity we want to go after? Yeah, so obviously like a quite classic analytics task. I'm going to take us through, you know, we launched a new product feature and we actually want to understand how it did. So I'll take us end to end from understanding how the feature was built, analyzing its performance, and then producing a summary that could eventually go to our exec team. like Tim kind of touched on without AI, especially the context gathering would mean hours spent digging through all the specs and PRDs, writing SQL queries from scratch, and then, you know, spending a lot of time writing and editing a doc.

19:56Tim Trueman:So with AI, I can pull context similar to what Tim just did directly from the code base, I can generate queries, and I can draft a synthesized doc. And so I am going to start sharing my screen. And while you pull that up, I have to say people think that why I got into AI in a deep way was because I thought it was so fun to code. And it was actually it made my sequel so much less ugly than it used to be. It was like my number one use case. However, many years ago, I was like, thank God. Now I don't have to bother my colleague with my disgusting SQL. I can bother AI with my horrifying SQL and it can make it a little bit more efficient.

20:40Yeah.

20:40Tim Trueman:I mean, even just ChatGPT for the last couple months has been a game changer for SQL queries. The problem with ChatGPT is you had to spend a good amount of time giving context, like the exact table names, the exact field names. And so using, I mean, it's not, its sort of most marketed use case, but using cursor, which is what I'm going to show today, to actually create, edit, write SQL has been pretty game changing, especially because it's so context aware. And I will talk about that. So cursor can take like three to four minutes to run some queries. So I'm going to just kick off this prompt, and then I'll explain the context and what I have done.

21:21Tim Trueman:So while it's running, I will set the stage. Last month in July, we redesigned the signup flow for a new payment method that we have been piloting. And this process of signup is successful when a customer links their bank account for the payments. And our old flow had been live for a few months, we had a hypothesis that we could improve it. So we redesigned the flow. So because this is a pilot, we actually didn't have enough retailers or users to run an A-B test. So I just needed to do a pretty straightforward, you know, how is this performing before? How is it performing after? Historically, again, that would have meant a lot of digging through documentation or more realistically just pinging an engineer to ask questions like, okay, what did we build?

22:08Tim Trueman:Who sees it and why? What front-end events are omitted that I can use to analyze this? And while I do work closely with our engineers during the nSpec phase to figure this out, those details are easy to lose track of, especially we're often coming back to analyze things weeks or even months after the feature launched. I will say that I probably would start with Notion AI context building, similar to Tim, but we already showed that. So I'm skipping straight to the code base. And if we go up to this prompt, my prompts are way less pretty than Tim's. I don't like spend a lot of time on them. I feel like with cursor, you can always iterate.

22:49Tim Trueman:And so I wanted to understand the setup wizard, which is what we called this new flow. I told it to research our code base and I essentially asked who, what, where, when, why. And so if we go to this answer, we can see, okay, it is, you know, looking in to the code base and, you know, I'm not an engineer. I don't really know what this means, but it, you know, we called this in our code, the first run user experience. and it tells me about some flags, cannot be sub-users. There's just like a lot of detail here. And it's telling me when users see this flow, what happens during the flow, the order of steps that happen.

23:27Tim Trueman:That's like pretty important. If I'm gonna analyze a funnel, I need to know like in what order did things happen? And then if there is a success event, like when the setup is complete. And then it gives me a bunch of events that I can use to analyze it. So this is already such a game changer. Like in the past, I would have leaned on secondhand sources like Notion to piece together how it was built. With Cursor, like you were saying, I can go straight to the source and have it translated into natural language. And that just gives me a lot more confidence because it reflects what's actually live and not what someone remembered to write down.

24:02One thing I want to call out while you're going to your next step is one of the steps that I see skipped by engineering teams is good event tracking when they release a feature.

24:14Tim Trueman:because you start up front in the PRD and you define a tracking plan and then it gets to implementation and people forget it should be a front-end event, should be a back-end event. And one of my favorite follow-up AI tasks after something has been released or it's in code review is I do a quick prompt and I go, is everything appropriately tracked in this feature? And I get either Cursor or Devon to go in and put in all the right events and make sure that the schemas are normalized. So for all the data analysts out there, be annoying and do a PR for your own events on new features. So you're not, you know, stuck with what the engineer has built for you.

24:50Tim Trueman:That inspires me to, I can take the N spec and just put it into any AI tool and say, what front end events do I, or what events do I need to ask for to be able to measure the success of this effectively? Because right now I'm just doing that in my head. That is not something that I have. Don't do it in your head. That's the title of how I AI, how I AI. Yes. Don't do it in your head. So, with this next prompt, I, again, not the most, like, sophisticated prompt. I'm just saying I want to understand at a high level how this feature has been performing. And I give the quick context of, you know, our goal is to make it better.

25:29Tim Trueman:That's pretty obvious. I just want to spell that out. And I, like Tim, I'm giving a fair amount of discretion to the cursor agent. I'm saying, okay, come up with the ideal output fields. I have some ideas, but like, you know, it's up to you. And then two, I do find that telling it explicitly to create a file, it sometimes forgets to do that and just writes the SQL directly in the conversation sidebar. Use the MCP connection. Like I went through all this trouble to set it up. I want it to use the Snowflake MCP connection and then actually QA the file. And that's what's so powerful about this cursor agent and the Snowflake MCP is not only is it writing the SQL, which is what ChatGPT has been doing for me for the last year, it is running it, looking at the output, and then making like its own sniff test sense check decisions, which is just so cool.

26:23Tim Trueman:Okay, and then another thing I want to call out as we are running this, the reason why I have a fair amount of confidence that this is going to work relatively quickly is because I and our data team have done a fair amount of work to create what's called a semantic layer. And so first, our amazing data engineering team, like six months ago, decided we were going to create like a general company semantic layer. And a semantic layer is essentially just a translation for an LLM of like our business terms, tables, fields, filters, metrics, et cetera. And AI can look at those files and understand like what our tables mean.

27:02Tim Trueman:This general one covered like our most used generic tables, orders, items, users, et cetera. And so they connected it to a custom GPT. And anyone in the company can go ask pretty basic questions like, what was the average order size in Europe last year and get an answer really quickly. And so that's been a huge unlock to save our analytics team time of like, we're not answering these questions for people, they can self serve. It's just democratizing data and, you know, saving us a lot of time so that we can focus on more deep analysis. And for deeper analysis, like, we needed something more than just these basic tables.

27:42Tim Trueman:And so I, with a lot of help from one of our data engineers, she's built a specialized semantic layer just for like, my scope as a test. So I was, you know, we're the I was the first one in the company to do this, but we're planning on kind of rolling it out to all of the areas of scope. And, you know, basically, this semantic layer just defines the tables that I use the most, the joins, the filters, the metrics. And because it lives in our code base, it's like in our data science repo, cursor can just tap into it. And it just makes the zero shot ability like insane of running people. I've seen a couple of these.

28:20Tim Trueman:And yeah, I don't know what yours looks like. But they really just look like defined terms tables. This table means this, this field means that if you're trying to query average order value, this is how you do it. And it's almost your documentation in a little bit more of a structured form around common queries. And what I think is nice about this is its ability to be managed by code. You can change it, you can update it, you can add new things. I also think for the data engineers out there, it reduces a little bit of needed complexity on the data warehouse setup because previously you were creating these like aggregate tables and these like defined metrics and you're hoping people were writing queries the right way.

29:00Tim Trueman:And now you can define these canonical queries and know that no matter kind of like what your tables look like, they're going to get to the right answer, which I think is quite nice on the data engineering side. Yeah. So this is an example of like what you were talking about. It's just a very structured JSON file. And from what I understand, I did not do this, but I had the engineer explain the process to me. And honestly, LLM's helped a lot with creating this. You know, he fed in details about our data warehouse and just a million queries that I had previously written and it kind of helped spit out this type of thing.

29:33Tim Trueman:He also used Langchain to like change the names of a bunch of the reports that we had into question form. Because obviously when I'm querying this, whether it's through a custom GPT or cursor, I'm often asking a question. And so I thought that was pretty cool. Like translating it to a question makes the semantic layer work so much better. Oh, this is going to be my next project. This is so fun. amazing glad to inspire so to go back to the actual sequel that was run and i will actually just run this let's see hopefully this and just in case people miss this you did call out the snowflake mcp which was what we're seeing right now which is a programmatic way to hook into running queries in your snowflake data warehouse so you can not only generate the sequel here but instead of like copying and pasting it and going into like snowflake cloud and running it or whatever your visualization tool is you can just run it right here you're getting your tables right here so again like you're you're eliminating that context switching you're eliminating the copy and paste and you're getting your data right here yep exactly and so i am oh this is interesting this actually i am looking at this and it's i think it showed a mistake um but you know i asked it to QA itself, normally this has done, does a very good job.

30:55Tim Trueman:But one of the quick QAs that I do for something like this is I want to see no skip steps. Oh, actually, you know what? I remember from the context, this is a temporary, this is a step that only some people see. But usually when I'm looking through this, you know, in, if we were not doing this demo, I would spend probably a lot longer QAing this, but I just want to see drop off that makes sense, right? Like I don't want to see zero, zero, and then one or then zero. And so that's just a quick QA that I can do. You know, it's not the AI's name on this analysis, it's mine. So I can do that. The other thing that I have done to really make sure that I can QA this effectively is I in my cursor rules, I tell it to comment every single CTE so that I know what the and sorry CTEs are like sections of sequel that often are created when you're writing sequel and I just want to know each step of what is happening so that as I'm looking at the sequel I can say okay the agent said it's doing this and like looking at this code I can actually tell that it's doing this.

32:04Tim Trueman:So engineers cover your ears because engineers hate, hate, hate, hate, hate when I say this. They hate it. I love overcommented AI code. And let me tell you why. Because when you are not writing this code, you really need to understand the thought process behind how the code was designed. And having AI comment the code that it writes gives you a natural language way to understand if your understanding of the implementation matches the actual technical implementation of the code itself based on the AI's own reasoning. Fine. Delete it if you want to. I don't care. I know all the arguments against overcommented code.

32:44Tim Trueman:And I think there's a lot of benefits for human review. And it's also great context for AI when they go back and work on it. So engineers, you can now uncover your ears. You can yell at me on Twitter if you want to, or an X if you want to. But I do the same thing where I say, go ahead and comment in the code so I can understand how you decompose these step by step. Yeah, it's pretty, pretty awesome. It's also I even have a custom GPT in chat GPT to comment code I've written before I just insert code. And then, you know, if I'm ever handing off dashboards to someone, I really don't want anyone to be so confused that they have to bother me.

33:19Tim Trueman:You know, my goal is to have it be quite self serve. Look, those lines of code are not going to expand themselves. Let's get some comments. This episode is brought to you by Brex. If you're listening to this show, you already know AI is changing how we work in real, practical ways. Brex is bringing that same power to finance. Brex is the intelligent finance platform built for founders. With autonomous agents running in the background, your finance stack basically runs itself. Cards are issues, expenses are filed, and fraud is stopped in real time without you having to think about it. Add Brex's banking solution with a high yield treasury account and you've got a system that helps you spend smarter, move faster and scale with confidence.

34:06Tim Trueman:One in three startups in the US already runs on Brex. You can too at brex.com slash how I AI. So I'm going to kick off my next my next prompt but basically like we're going to skip ahead a couple hours here because up until this point, like my goal was to get this kind of clean base query that I could use for dashboards in mode, which is FAIR's BI tool. You know, a lot of what we are doing as the strategy and analytics team is creating, creating tables that then can be used for pretty charts to tell a story. And so let's pretend that I spent a few hours with cursor, like refining queries, I actually did one for the old flow and the new flow, I actually did do this, this is also a real use case like Tim's.

34:55And then I built some visualizations in mode.

34:59Tim Trueman:What's really cool is that there is actually a mode MCP. And I can tell it to view a dashboard directly. For those who are listening, here we have on the old on the left hand side, our legacy flow and on the right hand inside our new flow, you'll see that there's one step that is only present in some of the entry points. This is split by entry point. And basically, it's just showing, you know, like, what is the overall success rate and success rate by step for each of these flows? And so this is what I have pointed the mode MCP towards in this prompt. So if we go back to this prompt, and I'm just going to tell it to run.

35:47Tim Trueman:this tool. Okay, so I'm telling it again, like, hey, go look at this mode dashboard and use this MCP. I also give it the direct SQL that's, that I wrote with cursor, that's powering that dashboard. I'm just asking it for some detailed takeaways. And next steps, I give it a little bit of context. And I tell it to ask clarifying questions and use the MCPs if necessary. The MCPs, I think I'm not sure if we've defined it yet, but model context protocol, I believe is what it stands for, are like so powerful. I think that that's when this has felt like magic the most. Like at first, I assumed that they were similar to APIs where everything needs to be defined.

36:33Tim Trueman:Like some engineer on both sides needs to go to find endpoints that there's a very specific structure. It seemed like a lot of work, these models just like know what to do. It's just wild to me. I will say that there's a lot of work on our data engineering side to get some of these MCPs set up. So I think Ben on our analytics platform team has just spent a lot of time on this. Like I don't want to minimize that step, but as the end user of them, it is like, it just feels magical every time it can just access something. And so if we go into the results over here, next key takeaways and next steps.

37:13Tim Trueman:Cool. So we looks like we did a good job. Yay, fair. And it gives like a pretty detailed list of, you know, the funnel analysis, insights and concerns, actionable next steps, etc. Like Like this is already a pretty good sort of output to start with. But at the end of the day, like analysis like this only matters if you can communicate it clearly, right? Like you need to sort of convince people of whatever you are trying to communicate. So we also have a Notion MCP and I'm going to ask Cursor to create a doc that captures our findings in a structured way. And I want to pause really quickly because we have done this in maybe 15 minutes where you have taken a problem, kind of like a pre and post analysis of a feature change.

38:06Tim Trueman:You have written SQL, have not used a WYSIWYG analytics tool. You have written straight up good SQL, traceable SQL to do a funnel analysis of that on a daily basis. Very interesting. You have made a dashboard for it so that your business users can use it. You have then done a meta-analysis of that dashboard using the MCP to actually read the dashboard, do a first-pass analysis, create a summary not only of the results, but of recommended next steps. And then you are going to publish that to your business using Notion. Now, I have to say, I have worked with a lot of data teams. And most of them spending their time saying, what is the priority of this analysis?

38:47Tim Trueman:We have a backlog. I need data engineering. And fine, here's the dashboard. like it's like the ones that like get promoted three times in a year that go the extra step where they're like and here's the analysis and here are my recommended next steps and I made it pretty so you can share it with your boss and I just think like I was watching this and I was like oh man I'm gonna promote this data analyst like they're pretty they're pretty they're pretty good and so I just think the ability to level up the quality of your work and think they're the interesting things the interesting thing isn't like did I write this sequel join correctly and The interesting thing is, like, have I thought through all the edge cases?

39:24Tim Trueman:Do I have any creative ideas on what we could do next? Can we improve this analysis for the future? And so I really like this end to end flow because it just shows how you are leveraging up into higher strategic tasks, as opposed to spending your time sort of in the tactics. Yeah, I mean, it's I totally agree. And we are almost done. But like you said, you know, we need to we need to communicate this. And so one thing that we have done on strategy and analytics is our chief strategy officer, Dan, like he really cares about synthesized writing and all the leaders on his team care about synthesized writing.

39:59Tim Trueman:And so we worked with him a couple months ago to actually create some guidance on how to write at FAIR. Like FAIR is very much a vertical doc culture, you know, pre-read culture. We're not creating a lot of slides. We are writing a lot of docs. And so we have this sort of like use answer for structure key principles doc. And then we also have a template for what docs should look like. And so actually in this prompt, you'll see like, I tell it to follow these rules that are in these docs. And that's like another thing that I love about Cursor is you can just tell it what rules to follow in a variety of ways.

40:43Tim Trueman:Okay, Alexa, I'm going to give you an upgrade here, which is you should reference these files in your cursor rules so you don't always have to answer. that's a great I should I mean I wanted to you know show the full flow but um the reason I don't is because it would have actually done it in the previous step oh yeah because it would it would have it would have known and then I wouldn't have gotten to talk about it but yes I will I will do that once we are done it's showbiz folks that's what this is um and so the last thing is I am going to pull over the doc uh this is one that created from a previous time I did this just because I wanted to highlight in yellow, I gave instructions in this prompt to tell me what to add.

41:23I think

41:24Tim Trueman:one thing I want to get across is this, I don't think that cursor yet, or AI can zero shot, like an executive ready doc yet. Like there's, that is where I think that we still need to do three to four revs of, of sort of editing, adding analysis, making sure this makes sense. Like these, Tools have so much context, but we have, we still have some context that is just this like, je ne sais quoi, like humans are still valuable. And so this is like a pretty good start. And I think what's cool about Cursor is like, I cut out some of the middlemen. I got to this point, like really, really quickly, but we're not just creating like AI slop docs all over the place.

42:08Tim Trueman:We are, you know, just accelerating how fast analysts can do things like this. we you know and the other thing that's really helpful about I would run this through that guidance three or four times it can be hard when you're been so in the weeds of an analysis to like take a step back and make sure your story makes sense and so that's what LLMs are really good for so it can like cover my blind spots well you know what's more painful than running this three times through your guidance is sitting three times with your SVP of strategy and having them tell you this makes no sense and you need to go back and edit stuff.

42:45Tim Trueman:So again, I think what a nicer way to get to a higher quality output than having to. It saves me time and it saves the leaders on my team time and hopefully improves the quality. You know, it's fundamentally improving how, you know, we are doing work on analytics team. And one thing I want to call out for folks that are maybe listening and not watching is Alexa, my friend here, is smiling. This is fun. This is like interesting and it's fun. You're not sitting here saying, I have no role to play anymore. The machines are going to take over. You're saying, man, it was really boring to like dig through tables and write all this sequel that I know how to write.

43:26Tim Trueman:And I've done it a couple of times. So let's let the machines do it. And now you're able to focus on interfacing with the business, having impact. And it's just, I think it's fun every time I get in these tools. I feel like it's magical. I feel like it's really fun. And so I want to call out, we got smiles across the board here on How I AI. I didn't show this, but the type ahead, like if when you're actually editing the SQL, that's also so fun. It's just fun. It knows what you want to do. So yeah, this whole process is very fun.

43:57Alexa Cerf:I think what's so powerful about this is not just like making the good analyst just incredible. It's also democratizing data. So this is something that can be done. SQL can be written by people all over our business, whether you're in sales, designers, anyone else can write this. So the people with the context can do analysis just like this. And then the analysts can do the really complicated stuff where these tools could help them get really into the weeds.

44:19Tim Trueman:for people early in their career i've said this before and i mean it to be true if you want to know the inflection point of clairevo's career it is when she learned sequel um true i mean truly i became unstoppable at that point and so lowering the barrier to entry on data analysis is just going to create a whole bunch of really high high impact folks awesome okay alexa so we just saw how cursor can do end-to-end funnel analysis all the way to the proverbial front door of your svp strategy. Tim, let's talk about another kind of analysis, which is experimentation analysis, my favorite.

44:55Alexa Cerf:Yeah, you should have close to your heart. So look, we've talked about the big picture, we've talked about like a really detailed sort of actual analyst of how they do their day job. But I think one of the other things these AI tools are just so good is just accelerating process, like automating away some of those routine, lower impact steps in the analytics journey. And so as a good example, we want to show you a quick agent rebuild, which automates the process of writing up experiment results. So across FAIR, we might be running, I don't know, hundreds of A-B tests on the product a month. And each of those experiments needs to be monitored, assessed, documented.

45:31Alexa Cerf:And that just takes up so much time for analysts. So if we don't stay on top of this very quickly, it's our team that can become the bottleneck and slow down our launch velocity, which is the last thing anyone wants. And I know this is something that's happening up and down the country around every single tech company. So we thought it'd be a good example just to demonstrate. So let me show you how I built this. One thing I want to really, really stress here is just how straightforward these things are to build. Once you've gone through the pain of setting up cursor, getting your MCPs in place, actually spinning up any new agent you can think about is just so quick and so non-technical for anyone to do.

46:06Alexa Cerf:So it all runs off a cursor rules file. So if you don't know what these are, They're literally just a type of file, an NDC file, that these agents know to look for and know they're likely to contain instructions. They're really easy to set up. It's basically plain English. So you just write a simple one-line interesting description of what it is. So format for writing experiment result using EPO data. EPO is just the experiment tool that we use. It basically takes our data, does a bit of analysis, slaps a UI around it, and writes it up for us. so you then select when you want to apply I just selected apply intelligent I trust the model to work out when it needs to use it they do a pretty good job and then other than that it literally is just writing out what you want the agent to do now this might look a bit complicated I'll generally write this in a few minutes in plain text what I wanted to write I'll have cursor to then tear the thing down and I'll rewrite it a couple of times and just get it right in the format I want but ultimately it's just a step-by-step guide of what I want this thing to do so I've just said for those who are listening, I've said, if you're asked to write up experiment results, do the following things.

47:14Alexa Cerf:So ask the experiment name if you haven't already got it, and then go collect the data you're going to need. So use the EPO MCP we've set up. So go talk to our experiment space, pull in the actual results of the experiment, and then use our Notion MCP that we've already talked about to go pull in all the other contexts that you might need. So any other documentation that's going to help it interpret that data and write up this report. And I've got a little bit down here, you can see telling it exactly what kinds of documents to look for. So PRDs, experiment docs, technical specifications, that's what's going to help it look for.

47:48Alexa Cerf:And then I ask it to basically write out those results in the format I give it. And then I'm pretty prescriptive about the format I want because I want this to do it really consistently in the format we want with really tight takeaways. So actually, I've asked it to create it in just a local file on my cursor on my computer. and that just means I can actually look at it before it goes create to the notion docs can take a peek refine the prompt if I need to but that's just a fallback and then ultimately it's going to turn into another notion doc so everyone else in the business can see it and it's going to do all this incredibly quickly and we're going to essentially see what this thing looks like in reality so let's just run it on an experiment result so I've just said please write up the experiment results for and I've given it the name of the experiment which is vertical product iron images and straight off it's gone off and it's found uh it's written itself a nice to-do list it's found the epa results so it's just called the results and it's found its results great it's found the the rules and now it's going to start working this all out for me which is great to see and then while it's doing all that we'll just have a look so the format we've gone through we can just show here so basically the rest of this is all just showing exactly what the format the thing's going to look like.

48:58Alexa Cerf:So I've asked it to give me the document links, exactly what I want if I click into more context, the brief summary of the experiment, and then the key bit, the actual metrics that it's got from EPO. So it's going to show me the actual results, the confidence intervals, it's going to pull out the most important ones, and it'll give me a nice little color coding for it. And then I just want the actual answer from this. So I actually want it to do the work of interpreting what we should do next. And so it's written the takeaway section. So I want to clear, should we roll this out? Should we roll it back?

49:29Alexa Cerf:What should we do? And give me the reasons why, like why are we doing this? And are there any other interesting insights that you found that we should call out from this? So let's see, right. So it's look, let's have a look at what it's doing here. It has found everything we need. It's starting to write out the doc, which is nice to see in this little thing. I'm just going to go ahead and queue up. So turn this into a notion. So as soon as I've run it while we look at the actual results it will start writing the notion doc and let's have a look so straight away in a second while it's running that I have got a write-up with all the right context I need so it's got the links I needed it's got the context it's pulled the right data good the nice thing is this result so this was just literally sharing vertical images rather square images like a really standard growth experiment like which one performs better and And you can see a nice fat sig lift of about three and a half percent for the treatment.

50:26Alexa Cerf:And then it's pulled out some other interesting business metrics. And let's have a look at these takeaways. So it's saying, great, roll it out. The right answer because of that lift. And it's also pulled out some interesting things. So it said data science prediction models are also actually positive. So it's saying not only have we got more retailers, they're actually higher quality retailers, the ones we've got. So this looked good as a first pass. This looks great.

50:50Tim Trueman:and just to call out one thing here personally like we we have a standard format for doing these where you have to type the confidence interval and type the emojis and that is like work that is not valuable for our team and so it's pretty awesome that like it came up with takeaways but also saved us five minutes of like fiddling around with emojis and decimal points yeah i mean ai as a translation layer between a SAS interface or a SQL query into natural language in the format that you like, that your boss likes, that's just a time saver in and of itself. So I love using AI as like the universal format translator.

51:34Alexa Cerf:So as you can see, I've just asked the Notion link, it should produce the Notion. So let's just open that up and let's put it on the screen. And look, straight away, I've got a nice document I can share around with everyone with all the right color codes. the takeaways. And even as a little bonus, let's see, it's done. It always has struggle getting things in a little toggle. But right at the bottom here, I've even asked it to spit out a slack with an even more summarized version. So I can just drop this into the right review channels and straight away, this can go and get approved. Now, are we going to do this for every complicated experiment?

52:04Alexa Cerf:Probably not. There might need to be a bit of analysis, but for the simple ones, straight one shot, even the complicated ones, this accelerates you, but also anyone in the business can start doing this, which means we can pass more and more of these things down to engineers, PMs, other people to write this kind of stuff and do the analysis for them, which again, can just massively accelerate our launch velocity affair, which we're really excited for.

52:24Tim Trueman:Yeah, I'm sorry. And I know this is my brand, but I feel like AI is just accruing to every task. Sorry, PM, it's your job now. I do like that, that little trend that's happening. This is amazing. Love it, have done these kinds of analyses before. They have not been this easy to read and they certainly haven't been generated in 90 seconds. Really useful tool for experimentation analysis, a call out to the experimentation tools out there that I know and love. If you have not made an MCP for access to your data, you are limiting your customers. And so I do think sort of AI integration of SaaS tools is going to be a way that teams start to evaluate the quality of tools that they're working with.

53:06Tim Trueman:So just something to think about if you're out there building data analysis tools. Okay, we are going to wrap up very quickly with a final, we're going to do a bonus. We usually only do three use cases, but yours are all so good. We're going to do a speed run through a bonus use case, which is actually designing and analyzing kind of unstructured data in a user survey. So Tim, you're going to whip us through how you could use AI to make surveys and survey analysis a lot better.

53:33Alexa Cerf:Yeah, I'm going to do this really quickly. We don't spend time with this, but let's just show. I think it's just another one of those incredibly common analytics use cases that everyone has to do. And they are just so time consuming. You've got to design the survey correctly, code it into a survey platform, then analyze all those dots. It's really time consuming. But end to end, AI can just like transform the whole process. Let's show another one. I'm just going to stop it. I'm not going to run these. I'm just going to go straight to my backup. So let's just start on design. So what I love doing is I think you can do it on the curse.

53:58Alexa Cerf:You can do it on many things. I think ChatGT projects is really good for this. And again, incredibly accessible. Everyone knows how these work. It's just a great way of giving context so if we switch over to this one which chat to petite it's lovely and taking a bit of load you can see in files what i did was give it a bit of background information so what is our bit of business so this was a survey we want to design on fair direct tools so that's our tools that we give all our brands to help them accelerate their sales with their own customers and so i've given a ton of information to the model that just says like what actually is fair direct what are these tools what's the strategy and then i whenever i do a survey like this um whether i'm doing ai or not i'll start with hypotheses that's ultimately what you want to test and so this is my way if i just open up those hypotheses so this is what i fed it into i just gave it a list of simple hypotheses on what um what we want to learn we do aligned we've got everyone aligned on some hypotheses there's 14 in here and then really simple i'll just call that one like um higher sales on fare leads to more usage of these tools, things like that that we asked.

55:06Alexa Cerf:Now I've just given that into it and all I did, if I just look at this prompt that we ran, so this was a simple prompt. All I did was drop it in saying you're a specialist at doing this customer insight surveys, design me a 10-minute survey for the thousand brands to test those hypotheses. I said these are the inputs I've given you, here's a bit of design requirements that we want and I asked of three things. I said, turn those hypotheses into a full questionnaire that we can go out to our customers. But also, don't just do that. Give me the coding file that turns that questionnaire into the actual, in this case, Qualtrics, the platform we use to actually run these things, can actually design that straight away in one click, and give me an analysis plan for some work to do there.

55:46Tim Trueman:I have to pause you really quickly, because this whole episode has been Tim saying, I just did this really simple prompt. And then you see this like 1000 word hyperstructure structured, very organized prompt. And Alex is like, I mean, I would just go in there and be like, maybe a nice survey. I love it.

56:07Alexa Cerf:So I'm a big believer that 99 % of my prompts are going to be one line. And then if I'm going to send a model, a big model is going to do work for 15 minutes. I'll probably ask another model just to turn my one line into something more detailed.

56:20Tim Trueman:held yeah i want i want the a b test of alexia you run this exact same gpt with a tinier prompt

56:28Alexa Cerf:and you tell me if you get the same quality see what happens see what happens baby i'm just uh i don't trust it quite as much as alexa does just yeah okay so what do we get from that so very quickly from a list of hypotheses i've got straight away a really nice first pass of a survey that's going to ask a lot of questions it's about the right length like this can just massively accelerate the process and then once we've got that right it's also given me that coding file which i just scroll on screen these things are painful to write so just having this a one-liner to tell exactly how the system should prompt this and write it out is just like saves hours of time for our research operations team and it even then translates that into an analysis plan that says this is what the outputs from that are going to look like so straight away this whole thing can go from a list of hypotheses into something we could probably get out to our customers by the end of the day now that's like shortens this enormously but what happens when you get the results back that's the other thing this can do and so again i'll do this incredibly quickly and just show you the final result but i did a very similar prompt as well so all i did i'm going to show you the file i dropped into this just show you how painful this is so i just gave the same hypotheses and look how bad this is like it's the raw output from qualtrics like these usually take a lot of cleaning it's one line for every respondent and then one column not just for every question but for every possible answer to every question so these things are incredibly dense for anyone's worth them and they take a bit of time a bit of playing with so the only other thing i gave it was a sub helper file which was basically that sort of coding file that i just showed you so it's the what's the question id what's the question language what's the answers and then is it i just added these two columns which is like is it a demographic question or an answer and is it a single choice or is it a multiple choice that's all i gave there and then i've written another one of my uh fun and simple prompts um so uh role task here just analyze the survey results you find the right most interesting things in this data and then judge the predefined hypotheses um so i want a table that basically says like for those hypotheses was it right or was it wrong uh and then again i always end on little quad check rest i don't want it to go away 15 minutes before and come back with something that isn't very useful.

58:42Alexa Cerf:And let's have a look at this just very quickly. So I've got a nice little summary out front. And then there are my 14 hypotheses. And it's got a nice table that says proved, neutral, disproved for each of them. And it's even because I asked it to give me a nice confidence score. So I said, one, it's really confident in this. Five, it's not very confident at all. And you can kind of see the different levels throughout this. And then beneath it, I've got for each of these, actually the specific analysis that I asked to do so just throw all the insights it found to back up those findings so like is this the only analysis we're going to do on this survey like almost certainly not but day one I've got the results I've thrown into this and within a matter of minutes I've got a much much better intuition of what all that day's showing so while I might go and do some analysis on this I can be so much more targeted on exactly what we want to what we want to look into where I want to spend my time.

59:35Alexa Cerf:And straight away, we can start sort of sharing some of these findings out with people very, very quickly.

59:40Tim Trueman:Oh, no. So I'm reflecting now after this episode, like, okay, I've told everybody to ship a bunch of features. And now I'm gonna be like, do a bunch of analysis. Like, in my mind, I'm like, oh, my gosh, I'm under using AI to actually understand my business. And it's so accessible. And if I can just write 17 point prompts like Tim, I can get really high quality insights, but I do want to call out just reflecting on this whole episode in your four workflows. What I love about what you're showing us is so many people think that AI is an input to producing a thing, but haven't done that full circle back to analyzing the thing, sharing the thing, communicating about the thing.

1:00:22Tim Trueman:And I think you're showing both sides. You can create with AI and you can analyze and communicate with AI. And I think looking at both sides of that coin is really useful. Okay, we are going to do the one and only lightning round question because we have gotten long on this episode. And I want to get you all back to all of your agents and MCPs and analysis. We're going to go back to prompts one last time. We're going to figure out your personality around prompts. Alexa, Tim, when AI is not listening, when your MCP will not call the tool, what is your prompting technique? Alexa, what do you do?

1:00:57Tim Trueman:I think mine's pretty straightforward where I think the problem that I run into most frequently is that I'm clearly running out of context. Like a conversation has gone so long that it's starting to be wonky. And so while I think, you know, level one is just starting over, what AI is best at is summarizing. so I'll say hey summarize like what we've done so far in this and you know 30 turn conversation and then use that to start over um because you know like like I've heard other episodes people say you want to figure out like where it got off track clearly I'm a pretty efficient person I don't you know I'm not Tim I'm not like writing out the entire prompt for 20 minutes like I don't have time for that I just want to say hey summarize what happened we're going to start over but I'm going to give it that summary so at least the new conversation can get some context from the old.

1:01:50Tim Trueman:Great. And Tim, what about you?

1:01:51Alexa Cerf:So much said for my prompts. It's all AI. It's all AI. Work my chat to you. So I generally will go and open up three windows on Cursor and I'll do three chats with three different models and put the same prompt in and get myself a cup of tea and see what comes back. That's the British stereotype in me and getting my cup of tea while I do it.

1:02:09Tim Trueman:Yeah, you run the A-B test is what you do.

1:02:11Alexa Cerf:Okay.

1:02:12Tim Trueman:I love this. Tim, Alexa, where can we find you and what can we be helpful with? You can find me on LinkedIn. My full name is Alexandra. And ways to be helpful. Our strategy and analytics team is hiring across the board. Our team partners super closely with PMs and our go-to-market team. We make strategic data-driven decisions. Super fun. We have tons of open roles. So if you like experimenting with AI, we are very AI forward. so you can learn more at fair.com slash careers.

1:02:43Alexa Cerf:And you can find me on LinkedIn as well and I'd echo that as well. Like come join us. If you love AI, come join us and show us how we can do it more here.

1:02:50Tim Trueman:Okay, we will link to your careers page in the show notes. Alexa, Tim, this has been so fun. Thank you for joining How I AI.

1:02:57Alexa Cerf:Thank you for having us.

1:02:58Tim Trueman:Thank you for having us. Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. See you next time.