How X built the best fact-checking system on the internet | Keith Coleman (VP of Product) and Jay Baxter (ML Lead)

25:29Your, your posts probably get, you know, a couple likes. And here you just put a, put a note on the White House. And they changed their public talking points based on what you did. Like that is an incredible amount of impact. So, you know, it, you could see why people are motivated to do it when they care about what's going on in the world. It, you know, you don't have to be a big, well known person to shape the discourse and information flow in a way that's helpful. It's insane. Like, there's so much to love about this. One is just the meritocracy of this whole operation of just anybody can, that is true and correct can participate and have impact.

26:10Also, it just shows you how much information we get that is just wrong. Like we had no idea how often we see things that are wrong. And now we do. Working on this product has made me realize just how many things I used to trust kind of by default that now I look at more skeptically skeptically. Definitely mean these days. Okay. Before we get to the origin stories, are there anything else along these lines you guys think might be really important to share really really interesting. Sure. I guess one other thing just is that although we don't actually use the fact that a post was noted in the core ranking algorithm.

26:46Which, you know, we think is a nice property. There is a really big impact just organically meaning not from the algorithm, but just from user behavior where people will like and reshare or, you know, quote, posts way less when when notes are applied. So just I don't know for people out there who typically run a B tests on big, you know, platforms, you may already be familiar with this, but like 1 % is typically an awesome effect size for any sort of algorithm change. We saw more like 30 to 40 % engagement rate drops for likes and reposts and a B test we were in when comparing showing a post with without a note, which is just crazy big.

27:32And then and then if you actually look that's that's just an A B test on the engagement rate. So that's not the network effect. If you capture the overall network effect of how post, you know, is spread less by that person's repost is good. If you look top line with a difference in difference as approach different multiple different external research groups have both found consistently that there's like a 50 or 60 % drop and total reposts, which is just nuts after note is applied. So it's having a really big impact on spread actually too. That's so it's like that's so great to hear it's what I would want to see and it's incredible impact basically like a AI image of something false would just go crazy on Twitter and did before community notes came out of now, which you're saying is just adding that context not actually like gallery you're saying the algorithm doesn't demoted if there's something incorrect.

28:24It's just people are like, OK, this is false. Why would I want to read this? That makes sense. Yeah, the notes just totally take the wind out of these stories. So like the thing will be going viral note appears resharing drops 50 to 60 % and like that's it. Like it just you can fit at 50 to 60 % per generation is the virality quickly goes to zero. And by the way, there's a very mixed feelings about this next one, but authors become 80 % more likely to decrease or sorry to delete their posts after they get noted, which, OK, that's great because like less less moos info out there, but I'm paying about because those are usually the best notes like if I think if the note was so just good that you had nothing you had no other option but to delete your post those notes don't get seen by other people right because that's our post that's our isn't there's an argument by the way that like seeing that's just because you might see the same misleading claim elsewhere off X or somewhere else on X, you know, it might be good to actually show better to have seen the post what's the note to then not see it at all.

29:28Yeah, unsure about that. That is so interesting. Yeah, I can be so sad if I was that community or no writer and just man, it's so good. They just can't even keep the post up. OK. So coming back from today's world where you're this like small amount of code is changing the way people understand the world and what they believe in making the White House rescind their announcements, zooming back to the beginning of how this whole project started. What I heard just briefly is Keith you were just kind of tired of managing PMs you wanted to just work on something yourself you wanted to work on something impactful away from corporate BS and you basically just started looking for something that was.

30:14Impactful important and you found this talk about just how it all came to be a beginnings of the story. Yeah, so I mean for me the beginnings actually go back to why I joined you was then Twitter in 2016 I was I had a startup and we were we had some acquisition offers and one of them was from company Twitter and it was 2016 it was the middle of the election between Donald Trump and Hillary Clinton. And there were like something like three televised debates but every day there was a debate happening on Twitter and it was very clear like this is where people are talking about these things that matter where you know information is being shared where they're ideas are being formed.

30:59And as a user it was obvious that I could get good information there but it was also obvious that there was kind of questionable information floating around and I remember just looking as an outsider thinking like wow like this is a really hard problem and it also seems really important. So we ended up going to Twitter and the company was an attorney round at that point so like my first three years was just helping to get the company growing again. You know working on everything that was the consumer product. You know getting user growth going back and people wanting to work there again etc but a few years in I was reflecting on what we had done you know I think we had done a lot of good work getting momentum going but it and people in the us and in the industry had tried things to kind of deal with with misleading information but nothing was really working like was obvious nothing was working.

31:55Nothing could handle the scale of the problem nothing could handle the speed and a lot of people just didn't trust the existing approaches the existing approaches were either fact checkers or internal trust and safety teams making decisions about what was or was not misleading and like a lot of people just didn't want or trust that to be the way this was decided which is very reasonable and so. You know I'm looking at that I was I was still managing a large PM team. You know that's a whole story in itself I I felt like I would that job required a lot of energy in and I and I didn't feel like I always saw the output that I wanted to see from it like I didn't see the change in the product I wanted to see and you know I was contemplating should I go start a company.

32:44I was sure I do something else and I kept coming back to this problem like man like the how is the world going to deal with the with this information quality issue like what we get on social media wherever we get it and like you know I'm at I'm at this company where you can make a difference on this problem like why not go and try some crazy ideas and see if like one of them might work and so I came back I had a kid I came back from paternity leave I went to my boss cave on. I was like hey cave on how about I just stop doing my job and I go work on this instead you know this being try some crazy ideas to see if we can deal with with misleading info he was stoked and so I went off and started working on that you know started with just reading well any research I could on the problem and existing solutions what was it was not working or the issues and then into prototyping and then you know it ultimately led to us building and piloting and I was like I'm not going to do that.

33:43I was building this idea that became community notes amazing okay I have so many questions and we're going to keep going through the story but when you joined Twitter was kind of the and it was called Twitter at this point I'm going to try to call it X now which I know is important to your boss what a era of Twitter was it at that point like it was cave on run joined or and who is the CEO because it has been many. Okay yeah I started I came in December 2016 so Jack had recently recently come back as CEO to turn the company around and just to give you a sense of like the state of the company something like a third of employees were leaving every year so just imagine like a third of your team gone every year you know the stock was in the toilet the product was not really growing and so Jack was working on a turnaround and cave on was there already cave on was running periscope with a much easier.

34:35I was working on a video stuff and you know that that group continued to you know Jack Jack was there up through the start of the community notes that bird watch project and yeah. Okay and it was called bird watch I don't think we've used that term yet but that's an important point it was called bird watch initially. Yeah so it was originally called bird watch when when we started the project but obviously so much famously the name changed along the way. Yeah maybe let's just tell that story real quick and I know we're zooming in for but just I have this Twitter thread that I saw would be between Jack and Elon when they're debating what to call it and Elon's like bird watch sounds creepy I want to change it.

35:18Is there anything there you can share. Yeah the story there the story there's kind of like Elon Elon came in acquired the company and you know we had just launched the product relatively recently US it'd been in pilot for a year but we had just made it available US wide and he I guess he'd been seeing the notes and he uses soon after soon after the exhibition he DM me and he was like hey this community notes thing is awesome and I was like oh I'm glad you like it like let's you know talk and so we talk the next day. And he kept referring to it as this community notes thing and I was like you know it's interesting that you keep calling that calling it that because that's actually the very first thing that I called it like the very first fig mom lock up I made depicting this thing was called community notes it just I don't know why it just felt really natural and so that's what we had that's the first prototype we had tested.

36:13You know later the project change it changes change the same to bird watch but you know Elon was like hey let's just call it that and so the next day we just change the name and you know it was it's always you're notable for the team when you change your name but really the team was excited about it I think it is a much more understandable name Jack has made fun of it calling it like the ultimate Facebook name or something like that but the most boring Facebook name. It's boring right name. We're just funny because they're now you know launching community notes but but I think it is a very understandable intuitive name and I think it is served the product really well there's there's a reason it was the name in the very first mockup.

36:57Yeah I think descriptive names just make sense. This connection with Elon and I want to talk a little about just how you've dealt with so many strong personalities over it and kept this alive throughout so many changes but before we get to that you you did something that I think a lot of people product leaders and leaders just people that manage people dream of give up all this power in air quotes and career trajectory and influence and just like forget all that I'm going to go back to just building something awesome small team is there any advice there you could share from that experience that you think might be helpful for other leaders to share to hear to help them maybe do that same jump because that's really difficult in practice easy to talk about hard to do.

37:42Yeah I think it is a difficult jump I've done a bunch of times in my career and I've always been very happy with it where I started with a small team that kind of grew into something bigger and then I was like you know this is like we're kind of dealing with a lot of big production set teams really big I want to go back to do something like crazy and new with a small team again and so I've kind of done that like soft teeth leap a bunch of times but it can be hard because certainly the natural like the classic career path is sort of I don't know rewards or you know running a large organization or being a manager things like that but I think at the end of the day you got to work on stuff you love you got to be having fun and you I think people want to be having impact and I think there's one myth that that can get in people's ways the idea that the more people you manage or something or the larger your scope is the more impact you have I definitely do not think that is true.

38:40If you just look at I mean look at community notes for example if I had stayed running a large consumer PM team like what would I have produced like 16 more pages of OK ours like I don't know you know a bunch of documents and I think building community notes has had way bigger impact on the world it's become the industry standard for how to deal with this now which is super cool people love it is the first thing that is plausibly dealing with the Internet scale you know issue of information quality. You know I I think it's unquestionably a bigger impact than I would have had if I were just do whatever doing some standard management track thing like I was doing before and I think that's true so many other you know small companies and startups I was just reading some of the screenshot I think is Blake sholes linked in the other day who went from like director of coupons or something to building the first supersonic from group on yeah yeah yeah and I you know those stories are everywhere when you look and so I definitely have found that you know for me I love building hands on I love trying crazy new ideas I love the zero to one experience it's fun to scale things up to you and it can be fun to operate it.

39:54You know it's scale but doesn't you know where are this team is a good example one that operates at a very large scale but that is still very small. Yeah I think the way you guys operate is what more and more companies are trying to do. Remove middle management layers create small teams that just execute and build impact and just like I see's whenever I see I have a comment and you two were like what is I see somebody is going to explain individual contributor non manager is when I say the word I see. So let me follow this thread and when I asked people about how you set up the team to operate effectively and and protected initially there's this term thermal that came up a lot they was like a thermal team if that's how you describe it yeah what is thermal yeah so.

40:40And you know someone who's working a larger company probably knows that things can get kind of bureaucratic or bog down decision making can be slow like there's this large planning cycles people can like try to like take someone from one team move them to another like a random arbitrary times that can disrupt a project like all sorts of things like that. You know our company this is a number of years ago when we started this project we had a lot of founders in the company like cave on is an example founder who is helping to run the company and he had this idea like hey why don't we create this program called it thermal where we could have teams that were somewhat isolated from that they could run on through their own process they would have like one clear owner.

41:20The team would be entirely dedicated to that project and we would just sort of like repeatedly make funding decisions as to whether to continue the effort and so why was it called thermal by the way what was the idea I think the I think it was like an old bird analogy of like thermals lifting you know the bird time their wings Twitter obvious through one point obviously had a lot of bird analogy is blessed it's hard. And so you know that was one of them but the you know the idea I love the idea as someone who you know liked the startup environment and so when we were starting this project I was like hey cave on like why don't we make this the first thermal project and he was like yeah let's do it and so so we started with that way of operating and it gave us you know from day one a lot of freedom and autonomy that I think was really important to make the product work.

42:14So just be very specific about what would would make it what makes it a thermal project and you set that up and this is asking from perspective if a company wants to build their own something like this what does that look like yeah I think there's a bunch of key attributes so cute one key attribute is there's one clear driver of the project who's effectively like founder. I guess it may be maybe you can have two or something like this like a really clear there's like driver of the project and also there's one clear decision maker that they go to. Outside of the team outside of the team and that was true back when we started and it is true now like if we need need something or have a question on something talk to Elon and it was like that from the beginning it's like that now it's a big reason we're able to make decisions effectively quickly in a simple way and probably has to be someone very senior not yes you need someone senior who can make the decisions you need me.

43:08Whatever they are so I think that's really important that clear decision making structure another was 100 % focus so everyone on the project is expected to be totally focused on it. That a lot of companies that can be easy to have people's attention sort of spread across a bunch of things and it makes it hard to get stuff done like you'll go to like at you'll talk to whoever that person is you ask them for help on something and they'll be like yeah I'll help you I got to finish this thing you know and it'll take me like a week or two and then I'll get to it like a week or two delay totally changes the momentum of a project when you know we were 100 % focused.

43:48We talked we talked in the morning it's like hey Jay why don't we like try this thing in the algorithm just like yeah then like then you know that afternoon or the next day we're looking at results and so because of that total focus the rate of iteration goes way up and then you know beyond that there was also just the ability to use whatever our own sort of like decision making process was we didn't need to write OK ours or you know follow others like standard practices obviously let's say we're going to do that. Like we had to make sure we were responsive or responsibly building the products and everything but you we didn't use need to use standard standard practices and I think that's another great example like OK ours I understand why they can be helpful.

44:31But they can also be you know not necessarily the right cadence that which to set goals like I don't I think it's really unclear that quarterly or annual goals are actually like the right pace like we would set our goals for what like we would set the goal for the next milestone that matters. And we would work on that and we reach that milestone we would have an idea of what was coming after and then we after when we hit that we set the next milestone whether that was two weeks a month. Three months like whatever it was like we set our own pace and goals at that pace and that just I think is a lot more natural for the development of something the whole OKR determination when planning process took longer than it would take us to pick a goal and then execute on it and finish it.

45:14How big was the team early on that you set up how many engineers started with just me and then we when we decided to build the thing it we we figured we needed about five and we want to be as small as we possibly could is clear we needed someone on ml doing scoring it was clear we needed someone to do some client engineering work someone to do back and engineering work. There may have been like one or two other we needed a designer and a researcher to help us understand the customer base and make sure we were we were building the thing in a way that was actually going to resonate with people and so I think that was I think it was like back in front end.

45:57ML design research that was that was the original team from what I remember amazing so one basically one of each function a question for Jay actually is there's all this talk of small teams and. Moving fast but you know some type is just any more engineers to build the thing is there anything you've learned about just how to keep a team small while moving as fast as you are and not need any higher more engineers and we need to hire more engineers I think in the in the beginning when we are iterating on you know what what should even the requirements be it was is definitely good to just have it you know like one and engineer but I think at some point we got clear on what the goals.

46:36Of the the algorithm should really be and we try it you know we were I think at the very beginning it wasn't clear that we needed to build this bridging based algorithm right the actual first algorithm that I put into production was very focused on anti manipulation is this kind of pay drink variant but it didn't solve the problem of you know bias basically so if there are some if there are more users on one side the a pay drink type graph algorithm can actually amplify those biases so. I think you know after building that prototype and getting data from that was clear that. You know the the bridging based algorithm was going to be the way that we needed to solve it and at that point basically I set up a bake off basically made you know this this kind of like the you know Kaggle competition or something so that was like the key time where it was really important to pull another engineers that is such a cool story.

47:36I want to follow that thread before I do that you just mention you guys y 'all thermal what does that mean is that like yellow like a version of your just okay just going to ship because we're thermal project ship it okay I have. Marketers I know that you love to you guys so let me get right to the point week studio gives you everything you need to cater to any client at any scale all in one place here's how your work for look. Scale content with dynamic pages and reusable assets effortlessly fast track projects with built in marketing integrations like met a C API Zapier Google ads and more a B test landing pages in days not weeks with intuitive design tools connect attracting and analytics tools like Google analytics and Samrush and capture key business events without the hassle of manual setup manage all your client social media and communications from a unified dashboard then create schedule and post content across all their channels.

48:30If you're working on content rich sites with studio no code CMS but you build and manage without touching the design and when you're ready for more week studio grows with you add your own code create custom integrations with wicks made APIs or leverage robust native business solutions drive real client growth with Wix studio dot com. Okay so coming back to this algorithm this actually really interesting because I've never heard any of this so I was going to ask just what inspired this actual algorithm and you basically did an internal competition amongst ML engineers to see you had the most successful algorithm Netflix.

49:06Broad contest style Kaggle style yeah yeah I think well so I mean this particular idea of of finding you know content that is light by people on opposite sides of the of a polarizer either who who typically disagree now this was not an idea out of scenario right like I think Keith had found some of Chris bills work he had done he had you know made this list of accounts that were often liked by people who you know are on both sides politically there is you know other other projects like a pull us out there that that looked for agreement among you know people who typically disagree but I think the yeah it wasn't obvious that our project definitely needed to use that from the very beginning but then you know when you when you want to in compare it against these other type like page rank seems obviously you know it's designed to be kind of manipulation resistant as naturally naturally like if you just have a voting ring of people who all vote themselves up then page rank and filter that out very well but like that just wasn't the main attack vector I guess so we we got we had to get some real data from the pilot to realize that okay the real thing going on here is people are polarized and and so it was only once we got that the real data from the pilot that I think it was it was clear that the the Virgin based algorithm was was the direction we really needs to go I want to come back to the way you operate the team I hear that you run the whole team off single Google doc that's like a four year old doc they just keep adding goals to the bullet points is that true there there is a very long running doc that has had to be chopped and purged because it was breaking Google docs in Chrome at various points in time it's sort of like a note taking doc it's really where we coordinate what we're doing the team meets on a daily basis we spend whatever amount of time we need to get on the same page about what we're building it can be we might talk about anything from you know what's most important right now to what are which we work on next to what are we trying to launch right now and why is it not launched like what's in the way of launching it and we might review new modeling or scoring algorithm update and you know try to understand what's working in it what's not so we'll just cover whatever we want and whatever feels most important and we like you know as you said we set our goals very dynamically so it whatever seems like the most important thing for us to work on now and next is what we spend our time on I think that's serve the project really well versus how feeling attached to like some kind of quarterly goals or something like well look at like what is going to help people the most or like what's the biggest problem right now what are either one of those and we will go tackle it and we can change our roadmap you know multiple times in two weeks based on what we see some hearing no juror no son on no Monday dot com no yeah I mean we we have to use juror like coordinate with some other teams like sometimes when we file a request we have to make a juror ticket but no I I am not a fan of heavyweight task management I love like being on the same page be able to keep most things in my head and having a really lightweight to write down the things that you know I can't or the team can't keep it in said we we did use a sauna briefly but my memory of it is that it spent you spent more time in the meeting grooming a backlog of irrelevant stuff then actually you know talking about the proper priorities so I think it's nice in the Google Doc that if something becomes relevant it can kind of just fall off without needing explicit backlog grooming.

52:57So just that maybe summarize a little bit of how you guys operate that might inspire other companies to set teams up like this so I'm going to go through a few things I you shared one is one person in charge of the team like the founder almost there like basically the founder of the team they have one very senior essentially sponsor slash decision maker that they interface with in your case you on no big deal in other cases it could be the CTO CPL someone like that. The team is focused 100 % on this product and and goal you keep the team very small so you start with one person of each function one front and share back in ML person designer researcher yeah.

53:38And then Google Docs almost basically for your project management is that roughly like yeah it's basically running with Google Docs stop don't use big complicated products I think that's pretty good recipe on the Google Docs you now take it the people can do what they want. So now go for I think this this first ingredients are really our key structurally and then you know beyond that it's a matter of having an ambitious goal it gets people fired up to go do great work yeah awesome I think there's a lot there that a lot of people kind of like think they should do and they set these teams up and they don't actually do and it feels like each of these is just a really key ingredient to it's actually succeeding it definitely really helped us succeed I don't know that the project would be here if it was not for some of those elements.

54:24That's that's a powerful statement like this thing that has changed the way the world understands what is true would not have existed if you didn't set it up in this specific way yeah I think you know I don't know if I would have started would have begun the project had I not known we had sort of that structure that ability to make decisions the autonomy the speed the ability to go fast and you know working we started with that in 1 .0 and and it's been continued in if anything furthered in an X I mean X is a whole company operates with a lot of those attributes and I think it's one of the reasons the product is successful I think it's a big those are big reasons why at least I Jake and speaker itself I have so much fun working on this I I love working on it you know it's great to wake up every day and solve these problems we get to you know we get to do them efficiently make decisions quickly build stuff that helps a lot of people it's awesome yeah this this like whether thermal or Elon way of operating is definitely more fun and and the fact that like that combined with the the awesome mission is super important for internal recruiting like I remember like when I was first trying to keep about this back in early 2020 you know I had another project I was you know it worked on a few but one was like personalized the number of push notifications that we sent and it was a drove a lot of D .A.

55:53you without like losing up that's a significantly so you know that that was like setting me on track or you know if I kept working on that I could have probably gotten a promotion from that with low risk or I could take this huge career I mean it's not as big at a career as like joining or founding an actual external startup but there is still career risk I guess and joining team like this so so just I think all of the same aspects of recruiting that apply to external startups imply internally and you know if you can have an exciting vision that is key related to that and your list Lenny one thing we missed that's super important is that on this project and I think of successful projects like it in startups is that people are self selecting to join we did not assign anyone to this project like people reached out to join or they applied to join the job you know I and the team interviewed every single person that joined the team and we're like we want that person on the team they want to be on the team and so people are totally bought in to the goal mission the way the team works the other people they're going to be working with and that makes a huge difference so I like a great time to do that is at the start of one of these things like don't if you're going to try something crazy like I would I'd do me tough if you're just assigning random people to but if you let people opt in and self select you're much more likely to be successful and one thing that I have observed at X which really surprised me was that this is also possible at a large scale you know one of the things Elon did when he bought the company was he basically asked people to self select to stay like you had to click the button and you send an email out that was like hey Twitter 2 .0 like for forking the road right forking the road forking the road exactly is like so we're 2 .0 you know now X it's going to be hardcore we're going to do ambitious things you're going to work your butt off you know what and you had to click on the form and say yes I want to join and I think that was really important for the company because you want people to opt into that you want that people to be saying like yeah that's what I want to do and the company is going to be a lot more successful if people are unsure it's like better for them probably to go do something else and where they're naturally more aligned and happier and I thought that was a great approach to taking a large company and getting it down to people who are really excited about you know working together on a mission so you know for us we did it from day one which is an easy way to do it but it's possible to do it later as well.

58:32I love that you described it as fun and I think a lot of people when they see Elon laying off a bunch of people being like very hardcore himself it's people don't imagine it as a fun place to work and it's clear how much you guys love working on this like how fun it is and how interesting it is and it's interesting to hear that because I think a lot of people don't feel that externally is there anything else along the lines of just working for Elon within an org Elon runs that may surprise people about just the way of working that's interesting or surprising or where you think other companies might want to think about adopting.

59:07I've always liked lean teams but this has made me my experience it acts has made me change the way I would I would think about running a future or you know if I were start a company and change the way I think about starting that company I would be even leaner than I would have made it before. I've been amazed with just how much the team is able to accomplish with the small group and I think because of a small group like when shortly after the acquisition we had this product called spaces it was it had been in the product before but it was it was pretty small scale and Elon wanted to run these large spaces I forget who the first people he was going to bring on wherever he was going to be there you know ultimately these things have gone on to host politicians and things like that and he's like guys we got to scale this up I forget the number he's like we need to be able to scale like a million people or something like that I'm getting the numbers wrong you'd be able to scale way up this is the kind of thing at 1 .0 that would have taken a year if it had ever happened and the team did it in like two or three weeks and it was really exciting and inspiring to see like we I didn't work on that but I watched her from the outside I'm like wow with this tiny team motivated behind a big goal that was like hey guys it's not like are we going to do this it's we are going to do this they got it done in two or three weeks that must have felt amazing for them it's certainly exciting to see but it's I have definitely come to appreciate just how lean something can be and not just get by but actually thrive because it's that way I think the point you made about people opting into that or is important because I think a lot of people hearing that be like I would never want to be asked to build something like that in two weeks and I think a lot of people do and what we love that kind of experience especially working with Elon especially is shipping something of that scale but I think there's an important one there just like okay I don't want to do that I have other things to do in my life other than ship spaces so I think that's a I think that's a key point you praised of just there's an opt in step totally I think the opt in this important and it may even be that you want to opt in one part you know one point in your life and maybe another point in your life something else is better I think you know whatever it is you're choosing to do it's nice to be opt in and to feel like it's aligned with how you want to spend your time something on my mind and I don't know if you guys want to go here but it's something I think a lot of people think about is when Elon came in he let go of 80 % of folks and everyone's just like Twitter is dead it's all going to fall apart there's no way they can run this thing with that small of a staff and clearly it they were wrong clearly it's working great it's like becoming a like a massive deal in the world and continues to grow is there anything about that that you were surprised by or anything about just like how it it continues to stop and it's not going to be a big deal.

1:01:54So well in spite of that big shift. I think the the leaner team the the reduced kind of like process in bureaucracy is a big reason it does move as fast as it does it's easier to get stuff done faster here and yeah I mean I think that's I think I think it's that that shrinking is actually a big reason for the increase pace of launch. Is the increase pace of experimentation one thing that I noticed that as a result of that is the people who are here they seem to all really feel like owners like they take like they take the sense of responsibility that an owner takes in the product they'll try to track down what's wrong fix whatever is needed jump into any to help build or fix improve any system that needs help even if it's outside of their space and there's the flip side of that to you for people who've worked it they come up with a lot of things that are going to be a lot of things that are going to happen.

1:02:54So if you're a company is they may have experienced this thing where there's like a nut you want to change some something in some other system or product. And so you reach out to that team and like maybe there are little resistant there may be like oh we'll get to that next quarter so they have their own goals to hit yeah that's exactly like they don't really necessarily want to help you're there busy here you're like hey guys we need to do this thing with that other system you work on and they're like great here's the code here in the docs. So send us the fab if you have any questions and we'll get it in and it's just the thing you can just jump in and get it done and that kind of collaborative effort like the sense of like shared ownership I think.

1:03:34And from my experience came from or result with a result of the shrinking of the team down to people who you know want to be there and work together to build this thing so I think that's been a really positive impact it's not always easy certainly like a lot of people have a lot of responsibilities but. You know they're here because they're up for it yeah I think one other thing that's key is what when you are forced to have such a small team you know deleting well this is important anyways but deleting code is more important than writing it a lot of time so I think so often maybe due to promotion incentives or just regular human tendency you.

1:04:12You know engineers have a tendency to add these little incremental wins that actually had add you know more of a long term maintenance cost than is clear because you just run a little one month a be test you see this. You know significant win you don't realize the maintenance burden you just added to your team for the rest of eternity until you turn the thing off so I think there's a lot to be gained and you get forced to do this. By the way when you have such a small team it's just deleting you know auditing auditing parts of your system and deleting the things where the maintenance cost is worse than one than the gains so I think we did have to do this across the company after the big way off and you know systems are leaner now and they can be worked on by fewer numbers of people.

1:05:02That's an amazing point I'm ever you know being like here we have to throw away the whole thing we have to re architect everything stupid the way it's built and sound like that actually worked. Yeah well we didn't you don't have to rewrite everything from scratch I mean some things we did I guess rewrite but I mean just even deleting the unnecessary craft and keeping the rest of the core system. That's awesome. I love the word creating kind of a formula to run these sorts of companies and teams there's so much here I want to go back to the building of the original product. I kind of took us on a long tangent and an amazing tangent but I heard a story of when you launched bird watch at that point you specifically wanted to keep expectations very low and there is like a gif in the thing and it just look like clearly this is not ready for prime time talk about just how you do that how you launched it in a way where people weren't like and it's never going to work.

1:05:52We were very disciplined I guess you could say about having the product prove itself at every given point you know when when we built the first mockups we had just these are just like pictures of. You're depicting what community notes might look like we showed those to people across the political spectrum we saw like hey people really like these whether they're on the right or left like they seem very open to reading these community notes even when they're critical to people of their own side. So we're like all right that gives us confidence that if we can build this like if we can actually make this is a reality is going to work.

1:06:27Then there's a question of like can we make it a reality like our will people in the real world be able to write notes that are of this quality. And so you know we built we had an internal pilot test version of this where you could like write notes and we first basically ran this through like a Amazon M Turk type of. It's just to see like if you just like put some normal people in there like will they be able to write these notes and it not they weren't all those notes weren't good but like it was clear that there were people out there who could write good notes so I'm like okay this is possible like what will happen if we actually do this out in the real world and like let's write let's run a pilot and find out and so we took that pilot that you know we run the M Turk kind of test on.

1:07:14And we were released it to at first a thousand people to you know totally on public and we didn't know what was going to show up like you could imagine the notes could have been terrible and and so we were talking like well what do we do like we're going to put this out there everyone's going to have all these questions they're probably going to be really skeptical like and we know it might be a total dumpster fire. And so like what do we do to like set expectations of perfect like we felt like we could probably get there in the end we just didn't know is happening first we wanted set expectations and so like well why don't we just stick there's like the page we see a post in the notes flow like why don't we just stick a dumpster fire gift like on that page and you know you go there you like hey you know anything you see below here might just be a total dumpster fire at least it would show we we were aware of that as a possible risk.

1:08:08In the end we did not do that it cracked me up but we thought it was kind of like he didn't actually launch okay this that we had we had mock ups of it and every time I looked at the mock up I laughed but I ultimately we so much to explain on that page like what is this thing and how does it work ultimately like okay this is probably going to like distract from the point so we pulled it's I somewhat I kind of wish maybe it is seen the light of day at one point but yeah ultimately we kept it simple we focused that. I'm not going to do it again it's just like a huge I'm explaining what was going on here but again you know we as this happened many time with tennis with the project you know we put the pilot out there and the notes were good like they were all good there's a mix bag but like there was gold in there and from the very early days with just a thousand contributors it was obvious that that people could write notes that were informative that were neutral that spoke to controversial challenging topics and that if we could just identify as those from the rest this was going to work like it was going to work as well as the very first mock up sweet made so that became the focus that is how do we sift out the gold from the rest I remember there's a I think you may have sure that with this with me when someone noticed you guys were testing this and they took screenshots and tweeted and I think you a lot of replied like this cool yeah yeah so in the in the very in the very early days when it was just a big one prototype we were running these like user testing dot com unminorated studies I guess of what are the participants sent one to an NBC reporter like wrote a bunch of stories on it anyway like that day there's a lot of chatter about it on on the service and Elon this is like this is what this is back in you know time perspective this is I think 2020 so two years before any acquisition stuff happened Elon is just a Twitter user building rockets electric cars and other cool stuff and stumbles on this thing that depicts the prototype that we've been testing and he writes back yeah definitely worth trying IMO and I ever think and I was cool back then and it's interesting to see like he's obviously had a very consistent point on it I think the you know the idea was appealing and he you know it's obviously a bit of a big fan of it in the product and a bit of big supporter of opponents so yeah it's kind of cool that it came from that support has been from the very early days before he was ever involved in the company and I love that moment I must have felt really wild for Elon to be commenting on this Figma prototype retesting it was cool it was cool oh man so when we were preparing for this interview asked you guys what's what's the main thing you want to make sure people get and understand about why community notes has been so effective and Keith you specifically said that it was the principles behind how you wanted to approach this and how you continue to stick to this throughout and we'll talk about how you kept it alive throughout all these different CO changes the leaders but just talk about these principles like what the actual principles are and why that was so key to it working out.

1:11:16There are a number of principles that I think when we first shared them with people at the company seemed maybe a little bit crazy but I think they are they are the reason the product works and I think they've been very important and we do we come back to them regularly today all the time. Probably the craziest one is just that this thing is going to be the voice of the people it's going to represent the voice of people it's not going to represent the company's voice so it is not a tech company deciding what shows it is the people deciding what shows and that had a lot of implications on the design like first of all there is no we don't have a button that will change the status of the note or of a note so if a note is showing that the company is going to be changing because the people have rated it and found it helpful it is going to show like we can change that and that is the kind of thing that like for we first proposed this that's unsettling to people they're like wait so like something can go up and like we you know the company can't take it down or you know what we can't change a status get it stop showing and we're like yeah and like it has to work that well if it doesn't work well enough to do that then it doesn't work if there's a problem with the note there's like one of our one of our key principles was if there's a problem with a note that's so bad that it's not going to work.

1:12:32So what do you want to do something about it is a problem with the system like we need to redesign the system to be showing good notes and so so yeah we had to you know get everyone comfortable with the idea that there was no button to change the status of a note you similarly we as we talked about earlier we wanted this to represent all of humanity and so we didn't want to be arbiters of who can come in and be a contributor and he can't like so we we open it to everyone you just have to be really basic and you know we can't do that. So you have to have an objective criteria like you have to reverified phone to help reduce the likelihood of having like bots or things like that participating but beyond that it's random selection and it stills that way today and you know again that people who took some time to get people comfortable with it but I think that the the fact that this is the voice of the people and reflects their output through an open and transparent process is so key to both wide is good like white works but also white is true.

1:13:33So I mean that's number one and and you it's you know well I think we'll forever be at the heart of the products. Another one that people thought was kind of crazy was transparency. You know we're like we're this we the previous approaches to dealing with misleading info they it felt to a lot of people like sort of black box tech companies or media companies or leads or whatever making decisions like they people people need to get comfortable with this they need to trust this so the whole thing has to be out in the open like the code that decides what note show has to be out in the open all of the data and ratings that that make it happen have to be out in the open people should be able to do that.

1:14:15So we should be able to take the code and data and replicate the whole service and that that we have done exactly what we've said we've done and they should be able to audit it they should be able to go and look and say like hey I think this part could be better like if they think we're biased they should be able to work you know work with the data and point it out and if they if people have good observations that should factor back into the code. And this is again something that's kind of difficult to get people comfortable with that everything is out there you can't cover anything up but I think that's so essential to people trusting it so yeah I mean we we we set these out on day one we go back to them constantly because we're we're always evolving the product and we're always like got to make sure every new changes open like whenever we update the code there's we're update the scoring system there's not being get have went the data is published daily so you can now let's see what we're doing.

1:15:06So yeah I think those have been those have been really central to the thing working yeah and and by the way these these do not come without a cost right like the the it's actually really hard from an end perspective to actually open source the actual algorithm that's running on the actual data. You know because the with the way large scale services like this usually architected does not you know naturally lend itself to like being run like as a script by someone who's downloaded a T .S. V. So I should have to take a weird architectural decisions to make this possible in a way that probably wouldn't have been if we didn't start with this assumption from scratch would have to maybe rewrite the system to make to make it like this.

1:15:48What's an example that for instance there's a matrix factorization that we that we train right usually you would train a matrix factor is a you know train your ML model once and then service I guess with a separate service but you know we didn't want to have people externally spinning up services you know to be able to replicate replicate the you know system that we had so I mean basically I don't think it would have been actually very cool if we had open source the code in a way that wasn't actually run a bull I guess you know by someone just you know at this point you can download Python code and run a script you do need a lot of RAM right now but how much time we talked about like 500 gigs it will take like a day if you don't do anything special speed it up.

1:16:44Yeah, but yeah, cool possible is the key thing and people have done it like the talc Peter and how to blog post where he you know talks about his explorations you know making sure the algorithm really does what it says it does and I think just the fact that a handful people have done this you know there's enough people who have done it that there's someone you probably trust who is very right now it's rolling out to me. I love just like as you describe these principles just I can imagine a PM at a company being like okay guys here I want to do this project can be completely like there's so much that idealism to it that rarely works in real life could be open source you're going to give it to everyone we don't have actual control over what it's going to do don't worry about it going to just change the way people see this thing that we've been very careful about and it works and I think that's very rare and it's really impressive and what I'm hearing partly is that the sticking to those principles was actually really fundamental to it working and not kind of bending over backward bending over when someone's like not on that we can't do this what if we change those part.

1:17:53I think if we had if we had broken with any of those principles like if there was anything black box if there was you know whatever I the product would be a lot harder to trust and so I think it's because we've just stuck to them so cleanly simply that you know people people can't trust it. You've talked about a few moments when it was like wow the White House change their announcement because of a community note we talked about the dog as a cat. Are there any other moments that after you launch of like holy shit this is working this is going to actually work all along you know we saw it we saw it working like we were pretty we wanted to be confident whenever we expanded it to new you know audiences or new countries or whatever like we want to be confident was going to work so you know maybe you hold a breath a little bit just to see that it would do what we expected but we always expected that that said there were definitely stress stress cases I mean the one that comes to mind is the start of the Israel Hamas complex in 2023.

1:18:56October that was probably the largest deluge of misleading information I've ever seen shared on the internet at one time it was like overwhelming that just the number of photos and videos and whatever coming out related to that was it was insane and just to you know to give you an example in the first I was like first three days or something of that conflict we had 500 notes covering all sorts of different you know like out of context imagery like people someone said like hey this is happening here it's actually from like 2013 in Syria there were people making fake like battle footage in the video game simulator armor three so they're like notes explain how the stuff looks really looked realistic and you unless you saw the know you like wouldn't really know and there are all sorts of claims about what was going on in the ground and you know that was definitely the product was still pretty new at that point we don't we'd expanded in the US less than a year before that we had been rolling out throughout the world that year and then this large event happened and you know I felt like we were we were just enough prepared at the right time for the system to be able to handle that like probably one of the most important things we did right before that was launch the ability to write notes on images and media and have images and videos and have this matched to other posts.

1:20:25I remember at that time thinking like wow I'm glad we launched that feature a few months ago versus still had it on the shelf because it was really important in that conflict and I think even it was like just a few weeks before we launched a major speed up in notes to you like like when we when we first built the product the number one focus was always quality like we knew we knew that notes the product would live in die by the quality the notes like that was the thing we could never give up but we also knew it needed to deliver speed and scale but we're like we will was get the quality in the right place and we can speak we can speed it up and scale it out over time and we had actually just launched a speed up that took like three hours off the time it time a note needed to go live and I was you know I think a matter of weeks before that conflict happened so again like super glad that was out there the in the first few days of the conflict the median time from a post going live to know showing up with five hours we just like crazy fast difficult fact checking is like two to four is really common to see it take two to four days these days we're showing up in five hours we're like we are so glad we got those things out before this happened in the service a lot more helpful one other thing that was I think a nice to you see working then was I go like one criticism of community notes some people bring up is well if you always need agreement from people a typical disagree then and these super polarized settings like that conflicts being like probably number one then then you know you won't see any notes but actually the reality was there were the you know tons of notes about that conflict so I think there is this kind of you know nice nice property where actually and maybe this is a surprising fact that there's more agreement out there cross polarized divides than maybe conventional was themselves right and people and the places where people agreed were really objectively true and verifiable like I guess maybe this is more true the more polarized the setting is but where the agreement actually lands you and basically notes that are very neutrally written very focused on the facts and and these verify information.

1:22:44I love just like there's a talk this talk for a while of just like no more there's no more facts like nobody believes there is a single true fact anymore like everything is subjective and I think community notes is exact up proves the opposite yeah fact matter there are facts we can all agree with even the most controversial topics yeah we we saw this really from day one it's the we when we would show those prototypes to people just depicting the idea it was really obvious that people cared more about or they cared a lot about getting understanding reality what was going on and they were willing to disagree with like their side so to speak to recognize that and I think that's not always that obvious to people the world does feel really polarized but people definitely are willing to cross parts and boundaries to to get to you know accurate information and that's why the product works.

1:23:38It feels like as we rely more and more on what we know and understand but the world is becoming social media online and moving this quickly feels like it's like we're I'm so thankful this exists because otherwise it just be what do we trust anymore like this is like it like this being out aligns with we need this thing to exist at the same time. And it feels like at the same time there's also this people just like I just don't trust I think people have shifted from I trust what I read to okay I shouldn't just believe everything I'm reading. Is there anything there you're noticing but just how people.

1:24:17Think about news they see and their shift of just like I'm not going to believe everything is there anything they've noticed by just like human behavior just the way we've shifted understanding what is true. We've been done any research you know to look broadly at at how people's perceptions are changing there but I certainly have found you know myself that particularly seeing notes I am more skeptical about what I read at first and I think that's been helpful and we hear that from people that they think about things a bit more and you know I think that's a good secondary effect and benefit of something like this which is the way I'm going to say that.

1:24:59The more you see the patterns of how what you're reading can be wrong the more you can thoughtfully question it and try to get a better understanding what's really going on so it's like you know historically this was called like media literacy but. And basic idea of like can you understand the ways in which things can go wrong and try to cut them yourself. And another aspect I think we help with that is discovery of the community notes I think often you know before community notes you could have just been living in a little news filter bubble or maybe there were fact checks out there that you should have been reading but you weren't discovering them right so the fact that the note applies is directly attached the post and visible by anyone who sees the post helps you know cross those those filter bubbles and can kind of.

1:25:50I think for some people it's the first time they've actually seen counter arguments to you know to to claims made in their own little echo chamber that's incredible yeah I love the point you're making about how they actually teach us people to. Be a little more skeptical of the things they read like it's an education system more than more than just here's the one thing is wrong. I love that okay just a few more questions there was an audience question asked on Twitter we all asked them to it or what people want to know about community notes one was actually why you guys switched anonymous contributors was the decision between behind that.

1:26:26Yeah that was you know we had this pilot where we were testing that with a small number of contributors like that 2000 contributors and we learned a lot to that pilot probably the biggest thing we learned was related to. You and on the midi or suit suit an emity of contributors we had originally assumed that it was important that people contribute under their like real handle or the real name or whatever it was we actually like the first prototypes depicted that we kind of thought that would be important for like people trusting the note. And actually was totally wrong we were like it was the the best option was actually opposite of what we first tried.

1:27:01We found a few things one people were hesitant to write a note on a controversial topic because they didn't want to get like a factor harassed online and so some people were comfortable doing it doing this but others were not and so meant there was more potential good notes to be written. And then we're getting written and this is very clear feedback from the pilot to and this is super interesting people are actually more willing to cross partisan boundaries when they are anonymous or pseudonymous then when they are under their real name and it intuitively makes a lot of sense like if you publicly are using your name you are you you feel are affiliated with one side versus the other you might hesitate to kind of like the perceived is breaking with that side.

1:27:49But you may actually you know for example find a note helpful that's critical of that side and yeah there's a bunch of studies that show when people are anonymous they're much more willing to cross bound partisan boundaries work with the other side agree with the other side and we saw that too and so by allowing people to just to be synonymous you actually get more honest answers about what they really think and helps find disagreement that really so counter intuitive. Yes you know you never hear the opposite always and so interesting it's the opposite yeah yeah I think that's the same principle plot.

1:28:19It's the same principle as to making the likes private is just thinking that yeah yeah I like a lot more stuff that's a little. It's tough yeah it would have liked yeah it allows it allows freedom for honesty which is pretty great. And and we you know one of the criticisms of pseudonymity is like you know can generate maybe like people have reached their quality the quality threshold that they put out there but we have so many quality mechanisms in the system that that wasn't an issue so we could keep quality high while opening up for that honesty. Another question you touched on this a little bit which is around navigating the existing trust and safety apparatus of Twitter which to as you describe basically previously was like we make decisions on what is true and not and there's every company works this way.

1:29:05You guys basically upended that like here's a completely different way of no control over what is what we say is true and not talk about just that experience of kind of overcoming that I imagine very difficult hurdle of like okay forget all that we're going to do it totally different. Yeah it was definitely what we were proposing was very different I will say that I think people were sort of open minded to it generally speaking and I think everyone had a sense that what was being done at the time wasn't really working that well or solving the problem. And you know people were open and you know ideas so that was that's like a good foundation but I think one thing we did that was probably very helpful in that is we wanted the product to prove itself at any point like first it had to prove that people could possibly find notes helpful then I had to prove that people could possibly like write these notes that would be good quality and so anytime that we were proposing doing something with the product like running some research test or running the pilot or expanding the pilot we always had the data that had convinced us that that was a good decision like we we were stepping into the next phase of expansion that made sense and so I think we probably rarely proposed anything that deemed unwise like because we were holding such a high bar for quality ourselves and I think you know I suspect that went a long way.

1:30:32So it's partly what I'm hearing is like take it step by step to prove this is actually working and partly be confident it is working you to yourself before you try to convince say the trust and safety team this is the way to go. Exactly. Is there any was there like a moment along that journey of just like okay here it's like it shifted from no way this is a thing to okay wow let's actually consider this or is it this very gradual process. Whether other people were saying no way to well it's just internally of just like okay we're going to actually stop this trust and safety. We have operating and instead for lying can you know so is there like a moment of like okay let's actually make that switch or is that Elon actually is that the big switch the biggest change there happen in X.

1:31:19The biggest changes prior to that were just the decision to put this out there and you know and have it be operating at. In public at first us wide scale but yeah then the big the bigger switches came in the X period I think even even though there was you know original research you know before before birdwatch even started or community that's even started from external researchers showing that you know crowd source fact checkers can do you know lay people can do about as well as fact checkers and actually the agreement rates were kind of similar between the groups. Like I think even though that research was out there I think there are definitely a lot of people who didn't really believe it could work until it already worked.

1:32:03So basically prove it prove that it works yeah yeah that makes sense versus just a bunch of docs and strategy and thinking it's just like look it's actually working you can see for yourself. Yeah make sense okay possibly last question we'll see which fractals of questions you guys bring up here I referenced this a couple times this incredible achievement of keeping a promise. So I think that's a really good project alive through Jack and then I have this note and cave on running the show then and then parag running Twitter and then Elon and then Linda taking over CEO quite rare especially something this visible this impactful to everything that X is any lessons or keys to that actually working of this project surviving throughout so many work changes and leaders.

1:32:53It definitely has been a crazy time to be building something has been fun the craziness has been entertaining. I think we have one one reason the perhaps the product has has done so well and survived is the product itself nature of the product itself it is designed to produce information that is found helpful by people who normally disagree. And so even if you have CEO's or leaders who might disagree like there's a good chance actually they'll find it helpful the back wow this thing does produce pretty you know useful but so I think there's something in the nature of the product itself that you know when people see it whatever side they're on you know left right what are up down they're likely to find it pretty helpful so I do think that helps.

1:33:38I also think the team executed really well we had ambitious goals that were exciting they solved a real problem this is a real problem that matters in the world. At at every step you know as we talked about the product needed to prove itself and you know we would make sure it proved itself and we would bring the results that convinced us and we share those with people and so they would say oh yeah I agree it kind of proved itself let's take it. We take the next leap you know and we've we've done that all along the way and we continue to operate that way and I think like that focus on the outcome and goal that matters and executing against it really helps like the team did not get distracted by much all through the period during which the acquisition happened like there was a lot of opportunity for distraction this team was shipping like every week like we were super focused on the team.

1:34:37So on the goal let's make this thing work let's get these notes out there and I think people saw that execution and and we're excited to support it. Yeah like it's working why would we must have that and it's important and it keeps us from having to hire tens of thousands of people to a fact check. You know an interesting thing about that is no one ever asked us or brought up or seemed to care about anything related to cost savings in this process and I think that's like a probably like an assumption people have outside the company that like this must have been a reason there was interest in it but like that was never a goal is not not at all why the project was started is not why people were excited about the project and I think that's also you know for people outside you know you don't see the conversations it's like kind of a heartening thing.

1:35:26The thing to know is that the focus was always on solving the problem the other approaches even if you had 10 ,000 people doing it like it the real issues that they don't work that well because they're not trusted or they don't scale or they're too slow and so the goal was really always just like help people stay informed at scale. You know it's building the internet internet scale solution to an internet scale problem that people like something I heard about you Keith when I was asking people about how this worked and why this worked so well. The goal is that they describe you with it having a very low ego and that allowed you to give up this whole team and power influence and just the name forget it whatever you want we'll call it for community and that's great.

1:36:09Is there anything there you can share of just like how you think about that and how important that is as a product leader to have a low ego. You know for me this this project I feel like I get to do community service with this project like my I like my see my work is in service of the people in the community and like that's it motivates me the only thing that I care about is delivering the outcome that the world finds helpful. And so in some ways the project is is not been about like you go or is always right it's like about truth seeking like let's find not truth in the sense of like what information issue but like let's find out what's actually going to make this work.

1:36:52Like how does it need to be how does it need to be structured what is it what should it be called like whatever is going to produce the best outcome is what we should do and so I think you know I feel more attached to the product being helpful than to anything else. And so you know to whatever degree might seem like low ego is probably more a result of wanting to actually solve the problem. And I think I think partly what I'm hearing is just if you win and succeed good things will happen. Yeah. Certainly satisfying things will happen is very satisfying to have people appreciate it satisfying that like people on the left and right.

1:37:26You know love it is satisfying that even people who receive notes love notes and treat out the most I'm like that's amazing it feels it feels you know so good to have helped give people that and yeah it's very motivating it's great reason to wake up in the morning. It's absurd this is work but it's also like of course this would work of course something like this should work it's like such an internet it's of the internet you know that's why it works. Oh man. Where's community and that's going from here which are people who look what's happening where's it going what's what's the future. We're always working on you basically more better notes faster so we want to there's clearly opportunity to get more notes out there.

1:38:11We want to they want them to stay as good or better than they are we want to get them faster so we're always working on like core product changes to help deliver that like recently for example. We just released an update to what we call the community notes bat signal or the ability to request a community note so we anyone on X can say hey I think this post needs a community note and now they can even out of source. Explaining why so that when a perspective writers sees that it's much easier for them to write a note so we're always working on on core things like that core program improvements. I think there are also new frontiers that show a lot of potential AI in lm's or one it's easy to imagine a lot of ways that AI could assist the people in this task they're doing of trying to get information out there quickly.

1:39:06Maybe you Jay should talk about the super notes work that we've done with some folks outside the company. Yeah so I mean one cool thing about having public data encode is that external researchers can can collaborate with you and in this case the super notes at this idea that we can basically take existing notes as input existing proposed notes that aren't actually you know maybe they have some problem maybe they have to tell part of the story maybe they're worded and kind of a biased way. Basically take all these in have an lm generate a ton of different variants and then basically make the simulated jury.

1:39:49So to basically you know get a representative group of contributors of for community notes who would be writing the note and try to predict based on their pass ratings how they would rate these lm generated notes and so this way you can actually. Rather than just like having an lm write a note from scratch and hoping it's good you can kind of like simulate the entire community notes rating process and you know explicitly create notes that are like way to be rated helpful by people so I think ideas like that are very promising for the future and it's a nice way that the lm's and humans can work together.

1:40:28I think like obviously you know agents can can browse the web too and and that's one way that you know you could imagine agents assisting humans is you know maybe maybe checking whether a source is actually supported by the note or a note is actually supported by the source although then you get into things like well you know are people going to actually be as diligent you know right now I think they're very diligent because they know just some community notes contributor wrote this like I better check this before I read it a really helpful but you know hopefully people you know we can design things in a way such that people don't trust the output and actually verify themselves before issuing a helpful rating.

1:41:10Yeah that is such an interesting area to explore where you want to avoid AI hallucinating slop be slop versus make it easier and scale it even further what an interesting challenge. What's cool about this project in addition to this the AI elements is that it's being done outside the company like we talked earlier about like Trent the open source transparency like the key reason we we made this all open source was so people could see at work but the dream is actually that it's the it's not just that the contributions to the notes and ratings are from the people the dream is actually the products built by the people like what if the scoring algorithm were significantly or entirely written by the public like that would be incredible and super notes is probably the first very substantial potential change in like the algorithm of the way it works that was coming kind of coming from the outside and plausibly could be part of the core.

1:42:05So we'd love to see the product go in that direction as well sweet go super notes well guys the work you're doing is tremendous I think this is every product persons dream I think to work on something like this small team lots of support lots of impact like just like a neatly interesting and so I think this is going to inspire a lot of people. So let me just ask you is there anything else you wanted to share anything else you think might be helpful folks to leave them with. Sure I guess one thing that I just I thought was interesting over the course of working on this product is just there's I think in a similar way to how retweets originally were not something like jack came up with I think user just started doing it and then it became a core part of the product there's a huge way already in which there's just a lot of surprising things that people wanted to use community notes for that I don't think we really expected and it's it's kind of cool to see those you know user.

1:43:01User desires kind of merge I think like one example you know I guess we've always been imagining political type of misinformation but for whatever reason there's like you know a lot of people who love debating met whether messy or Ronaldo got more goals I guess it's kind of kind of a funny one that there's a community moderation aspect right so I think we also thought that you know this would be specifically for. Adding context to misleading or potentially misleading information but what you can see is that there are some notes that go beyond that towards like calling out content that they think is spam or something so I think that's that's just I guess another just another dimension in which community notes is a product that's like driven by the people itself that's so beautiful basically they're trying to keep Twitter or slash action.

1:44:01So I think that's a very complex healthy and they're just like no this should be taken down this tweet as spam yeah I love that was their answer on the messy versus who is the other. Ronaldo Ronaldo okay. There like a definite fact there it's come with that is that just unknowable yeah I guess I guess that's an interesting one because it's it's a case where raiders are actually very polarized. I guess it actually kind of fits into the core algorithm where there's some people are just die hard messy fans or Ronaldo fans just so I think it could be on politics. So we actually you know it's specifically modeled that topic as well as some other topics so we can like estimate people's opinion on that particular debate it's kind of kind of funny that's something like that would emerge but I think that's the most controversial topic on X for now versus messy.

1:44:50It's a controversial one. Oh wow. Who knew? Okay Keith is there anything you want to tell. Yeah you know community notes is cool itself but I think what it points to you about society is actually even bigger. Society often feels really polarized. You feel talk about it all the time like no one can ever agree on anything but actually like community notes shows you people really can agree on quite a lot even on super controversial topics related to politics and everything. There's a lot of agreement that's why notes work and I think that's a really big reason for optimism about the world is it well it might feel polarized there's probably like an 80 % you know set of people that agree on quite a lot of things and imagine if we could use the same kind of approaches we use with notes but to find agreement on legislation or policies or things like that that people want the government of the world to do like possibly we can get a lot of things.

1:45:50We get a lot more momentum behind these ideas that the people really want and everyone would be a lot happier like maybe 10 % of the people on the edges wouldn't be happy but like I bet there's a lot of agreement that we are not identifying and if we did it we'd all be pretty happy. So I don't know I think it's easy for people to feel pessimistic about the world but I think this is this probably just a good reason to be optimistic about the future. What an incredible way to end it. I can also see Keith White people want to join you and work with you and work on this team. Appreciate it. You do want to join.

1:46:21We are hiring an ML engineer you get to work on these amazing problems with us and have a lot of fun. So we're accepting applications x .com slash community notes. Okay great. I'm glad you give the girl. Oh man you're about to get flooded. Guys thank you so much for doing this. Is there anywhere other than that place to go off? Join the team as an ML engineer is there any other place you want to point people to either your socials or anything else? I'm Kate Coleman on x. Please reach out if you have any feedback or want to help us out whether you're going to want to work here or want to do something from the outside.

1:46:57Would love to talk. Yeah I'm at underscore J .A. Y. Baxter underscore at x. Yeah I think in particular you know besides just using community notes it would be great to get more substantial contributions. You know like full requests. Collaborate on projects like super notes. I think that's the most exciting type of stuff people do want to contribute. Ship some code guys. It's amazing. Guys thank you so much for doing this. Thanks for having us. Thank you so much. Bye everyone. Thank you so much for listening. If you found this valuable you can subscribe to the show on Apple Podcasts, Spotify or your favorite podcast app.

1:47:41Also please consider giving us a rating or a leaving review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at linnie's podcast .com. See you in the next episode.