How are web standards made?

0:10Hello and welcome back. It's springtime and we're back with a new episode for Search of the record, a podcast coming to you from the Google Search team, where we talk about search and maybe have some fun along the way as well. My name is Martin, and I am a... What am I? Developer? No, not a developer. Search relations engineer, I think is the official title. Someone excited about... I don't know. But I'm not alone in my confusion, I guess. With me today is Gary. Hi, Gary. Maybe you're a unicorn. No, I'm not sure. That could be your title. You're a house elf, right? Yeah, I had many titles. I remember maybe 10 years ago, because I'm such a cheerful person, Miley Oye, who was working with us back then, she gave me a title, Chief of Sunshine and Happiness.

1:18and I was very happy about that because it's super ironic. No, that's very accurate. No, it's very accurate. It was such a joy. Martin, stop lying live. Oh, wait, this is not live. Go on. It's off the record. We can say whatever. Oh, God. Anyway, so that was my given title and then the house elf, I don't know where that came from. I think it was someone asked me what my title is, like someone external, and then I was just fumbling like you did with whatever our title is and we don't know. And then I was just like, you know what, house elf. And then it just stuck. I think I gave myself, internally we had this tool where you can look up people working at Google.

2:13I think my title that I gave myself is Open Web Cheerleader. So that's a fun one. By the way, Open Web. Oh, yeah. Remember Steve? I thought it was cool. What? Remember Steve? Our search engine? The best? My butler? What? No, the search engine that we built. You know, the toy search engine that we built and haven't been speaking about for a while now. Oh, my God. that was like 20 years ago. Yeah, it feels like it, but it's, you know, I was wondering, like, Steve had a cool feature that I think other people should use more. Can we make that into, like, an internet standard? It's already a meme, I believe, but we could make it a standard, no?

3:02Should we do that? What? You've been at a standards meeting recently. I know that you've done that, And I know that you have worked with someone to make robots.txt a standard. Wouldn't it be cool if we made more web standards, like more? Well, I'm not even sure if it's a web standard. Is it a web standard? Is robots.txt a web standard? I don't know. I'm confused.

3:31So I think we have to go back to what a standard is first. And then we can qualify them with Internet Standard or Web Standard or whatever. Oh, okay. But I think it largely depends on under what organization you are standardizing something. I work with the IETF, the Internet Engineering Task Force, within that a couple working groups, namely AIPRAV and most recently the TLS, like the, what does that stand for? Transport of AI Security. Security. Oh, okay. All right. And yeah, let's, like if you want to go and explore this, then we should probably look up what a standard is, right? Okay, so I think a standard is kind of like an agreement amongst a bunch of players in a certain field.

4:47Let's say like HTML is a standard because a bunch of groups have agreed that HTML has certain elements and is built in a certain way. And what are the things that it has and hasn't? And I think for HTML, it used to be the W3C, the World Wide Web Consortium, but has recently been, well, recently has a couple of years back moved to a living standard under the W3C. Yeah. So I think like a bunch of people come together, form like a forum or a group where they agree on certain things to be true or to be part of something, I guess. I don't know. So in my head, it's typically a document and drafted by consensus.

5:46So basically, someone proposes something to a group, and then the group agrees that it's a good idea and we should continue with it. And that's the approval state. And typically it has to be under some sort of institution or consortium or something, like some governing entity, I suppose. So yeah, basically what you said, but using fancier words. and we have quite a few such entities slash consortiums that govern standards that we are using on the internet namely like the thing that I'm working with is the IETF but there's also the W3C for example that creates standards And I think that they typically agree upon what they are governing.

6:56So, for example, the ITF is governing, well, internet-related stuff that is lower in the stack, in the internet stack. So for example, transfer protocols like QUIC or TCP IP or those kind of stuff, HTTP itself. And then I could be completely wrong because I haven't worked with them, but W3C is more related to the markup perhaps. Yeah. That is used on the internet. And then we have JavaScript that I don't even know where it falls. But that could be like a completely different consortium. Because for example, C has its own thing. Like its own entity and own governing body. So I would imagine that JavaScript could have the same thing.

7:56It has the TC39, the technical committee 39. But I'm not sure which organization that is under. Because it's clearly part of a bigger thing. But I'll figure that one out. Eventually. Eventually. Right. So what are we standardizing? So I thought we had a pretty cool thing where Steve could be told what cool things you have on your website by basically having a cool.txt. So that's not really markup. It's like robots.txt, but better. What? Nothing is better than robots.txt, Martin. No? So it sounds like a sitemap. Oh, so maybe we already have that. Yeah. Huh. Where does the sitemap standard live?

8:46I mean, we could try to standardize sitemap. Isn't it standardized? It doesn't. Stepping back, like robots.txt was de facto standard for what, like 20 years or something, or 25 years, something like that. And then eventually we standardized it under the IETF. Why the IETF? Because that's what I was familiar with, like no particular reason. I'm a nerd and I read RFCs, requests for comments, and basically internet standards to understand like how the internet works and what are the new things that happen on the internet, like new protocols that we can use for stuff. So basically, we just went with the IETF.

9:40Okay. So similarly, Sitemap, which was created, I mean, the standard was created, well, de facto standard was created around year 2005, 2006, or it was announced in 2006, but was created in 2005, something like that, that is a de facto standard, meaning that there is no governing body that adopted it and pushed for it being an actual standard. Okay, but it's the de facto standard because everyone kind of agreed that we are going to do it like that. Yeah. So basically, it's so widely adopted in its original form that basically it became a de facto standard. I don't know what a better word is for de facto, but it's basically something that people just adopted.

10:48Like writing is not a standard, but what? It's an informal standard because once you have a document and a larger organization adopts it, it's a formal standard. Yeah. Perfect. Informal standard. Then we don't use Latin anymore. Informal. Fine. Fine. If I had to guess where we would go with, or if we wanted to standardize it, then I would guess whoever governs RSS and Atom and those things. Because technically, it's doing the same thing as those formats, basically giving you a list of URLs plus optionally some extra metadata. And probably that's why we are also using, we Google search is also using RSS and Atom as discovery sources.

11:50Confusingly enough, they have RSS advisory board. The RSS advisory board owns RSS and that's a separate standards body apparently. Wow. Who owns XML? I think XML is... What did you search for? Atom XML. I searched for RSS, and RSS is owned by the RSS advisory board. Huh. Ah, Atom as well? Atom, I don't know. Let me see. Actually, Atom is IETF. How about that? Oh. And I found out who owns JavaScript, and I feel very, very dumb. Okay. Anyway, so we should start. Well, we should stop Googling, I guess. And good podcasting. I think this is the real vibe of this podcast, really, figuring things out as we go along.

12:51JavaScript called ECMAScript. It's owned by the ECMA, the European Computer Manufacturers Association, which is very confusing to me. Why does the European Computer Manufacturers Association... I didn't even know that we have computer manufacturers in Europe, but anyway. We used to at least. Okay, so this is kind of weird. So basically Atom is owned by ITF, well developed by ITF, and RSS is not.

13:29which means we can actually choose whatever we want. Like if we wanted to standardize sitemaps, then technically we could go with the ITF or try to go with the ITF and see if they would want to adopt it. And that's because they already have something that is similar to sitemaps. Hmm. Okay. So it kind of fits in the big picture. And is that how you would choose a standards body? Because if you want to make something a standard, you probably... Apparently, you have lots of bodies to choose from. the RSS Advisory Board, the IETF, the IEEE probably, the ECMA apparently, the W3C, the WattWG. What does it mean?

14:32Why should I go with one body over the other? So you would probably go with one body or the other because it fits their purpose better. So for example, I guess sitemap.xml is not that great because there you can choose multiple stuff or multiple bodies. But if you say you wanted to create a new or a replacement for TCP, then there's only one place for it where you would do that, where you would develop it, and that's the ITF. And that's because the way I understand it is that the people who would know enough about previous standards, related standards, are there. So basically, you have a community that is expert on the topic, and then it's more likely that you are going to end up with something that is actually usable on the internet because the discussions will lead to a better standard.

15:58Okay, so whoever is best positioned to help you make the right decisions in the standard and get it adopted as widely as possible, that's where you should take it. Okay, got it. Yeah, that's the way I understand it. Makes sense. Which might be completely off, but in my brain it makes sense. That does make sense, yeah. And then once you picked a standards body, then what? Yeah, that's a good question. In the case of IETF, I think there are two ways. Well, actually, three ways, I guess. So at IETF, you can publish something like an informational RFC, which doesn't become like a standard. It's basically an opinion piece, if you like.

16:58And pretty much anyone can publish that. I think, and there's minimal scrutiny with approving them. And then you have, if you want to actually go for a standards track document or end result, then you have two options. One is finding a working group within the IETF that would adopt or babysit your draft that you are writing. Meaning that you find the group that has the most expertise for the stuff that you are writing. So if you are, I don't know, when you were developing or when we were developing QUIC, QUIC, which is like a kind of a replacement for TCP IP, not quite. But now with HTTP 3, it's actually getting used quite a bit.

18:17you would look for the working group that developed TCP IP, which is a very low number standard, probably not one, but somewhere there, because it is the very basic building block of the internet. And then you would go to that working group, and you would ask them, and, hey, what do you think about this new idea that I have? Here's a draft that I wrote up.

18:52Would you see this as a good fit for your working group? And then if they say yes, then you knock yourself out and start developing further with input from the working group. They will raise lots of concerns. they will have very good feedback about pretty much every letter in your draft. And then eventually after probably years of iteration, you end up with something that you can get consensus on, meaning that everyone agrees that this is a good standard proposal and it should become a standard. Okay. So if you can't find a working group or you don't know about working groups in general at IDF, then they have a dispatch list.

19:51And you can email that dispatch with your idea, including a link to your draft. And then people will start arguing about which working group should this belong to. or whether it should be independent track. That sounds interesting, but it sounds pretty low barrier. How many people are doing that, just like emailing their ideas? Is that happening often? I don't know. I don't follow the dispatch list. I'm following HTTP, TLS, and AIPrev. But I would imagine not so many. And I think there's a good reason for that. And that is that technically the internet is in pretty good shape when it comes to the technologies that make the internet happen.

20:54So there's not that much need for new stuff and replacement stuff. And yeah, it just doesn't happen. And then when there is something new, then people would directly go to the respective working groups instead of dispatching their draft. Because whether we like it or not, the community that develops the internet is pretty tight. So they kind of know where to go when they have a new idea. it's not like Gary woke up one day and hey I want a new AI nonsense dot text and I will make it happen on my own they just know where to go so you go there you find a working group you build your request for comments thing and then you get like lots of feedback and at At which point is it that this becomes a standard?

22:13Well, it probably takes years for something to become a standard. And when I say probably, I'm underselling it. So basically, it just takes years for something to become a standard. There's a good reason for that. Basically, standards shouldn't be issued lightly because they are going to govern something. There's lots of things to pay attention to when you are developing, especially internet standards, because there are, for example, bad actors on the internet who are going to try to exploit the stuff that you are developing. Let's say that, I don't know, robots.txt.

23:09There's a risk that someone could create a buffer overflow in our parsers, for example, and then exploit that to their advantage somehow. Like, it cannot happen because it happens in isolation, or the parsing is done in isolation, but let's say that if we hadn't put a 500 kilobyte limit on robots.txt file, then people would be able to cause a buffer overflow, like try 4 gig, to like a 4 gigabyte file to see if that would cause damage to the parser, namely a buffer overflow, or try with 64 gigs or whatever. And then once you have that buffer overflow, then you have access to memory blocks that you could exploit to your advantage.

24:14And these are things that when we are developing the standards we pay attention to. So basically, when I'm reading a draft, then I would look at how I would exploit stuff that the standard is describing and then make recommendations to the draft or to the authors saying that, hey, I think like this particular bit could be exploited. How about you add a 500 kilobyte limit to the parsing limit? Because then basically the plan of the attackers or potential attackers is foiled from the beginning. It's like with 500 kilobytes, you're not going to exhaust memory. Or if you do, then you have a different problem.

25:14To me, often it feels like people are nitpicking on stuff, but they are nitpicking for a very good reason. And that is that these standards have to work everywhere for a long time without fault or as little fault as possible. They are going to nitpick every single little thing in the draft and make recommendations about how to improve it. And that can also go really weird because sometimes it's nitpicking about the language that's used in the draft. Oh, okay. So for example, you haven't explained clearly enough how parsing should be done when there's an empty line between two rules, for example, in robots.dxt.

26:08And then you would go back and refine your draft and add more words to a sentence to better explain that kind of stuff. I think in our case, we were very lucky because we had our tech writer with us. And Lizzie is really good at noticing deficiencies in the language that we are using. like when you or I write something. On first read, when we spit out our drafts, on first read, it's like the England is very weird sometimes. Because England know our first language. Yeah, and then Lizzie would come in and she would clean it up and ask questions about like, this is what you meant or this is what you meant, because depending on where you put, I don't know, the comma might mean different things.

27:11So yeah, that's one thing. And the other thing is that, especially in ITF standards, we use certain keywords that have weight. And that would be stuff like should or may or must. And if you're reading RFCs, those are capitalized. And that's because those are special meaning keywords. Well, not special meaning, but they have special weight in the standard.

27:51and when you're reading it as an implementer, then you have to understand that if something says must, then you actually have to do it. So let's say that in case of robots.txt, You have rows and then the rows contain a key value pair. Right. And then in the draft, it would say that the key must be separated from the value using a column. And then when you are writing your parser, then you know that there is no wiggle room there. it must be, it has to be a column that is the separator. But then, like for example, with the parsing limit, like we do want to allow some wiggle room. Like for example, if you know that you are absolutely certain that you cannot get a buffer overflow from two large robots.txt or very large robots.txt files, then you could say that we impose or parsers should parse the first 500 ,000 kb of robots.txt.

29:17And then you know that, okay, it's not a hard limit. It's basically like a lower limit of how much I have to parse. And then if I want to parse 700 gigs worth of robots.dxt file, I can do it. Like there's no standards limit set to that. Like hard limits. So yeah, those keywords mean a lot. Okay. I don't actually know where I was going with this. It takes a long time for a standard to get made because they have to be long-lived. And eventually all these comments are addressed and everyone in the working group is happy with it. So does it automatically become a standard? Okay. So you can push back on comments.

30:10Like it's, hey, I took your comment in consideration. Here are my reasons for using the current language or the current structure or whatever. and I think your comment should not be applied to my draft. And then you can go back and forth and convince the other person to basically accept whatever you already have or you address the comment, yeah, and then basically implement the change that you were asked to implement. And then once there are no more comments, from the working group, then there's something called a lost call. Meaning that... And I actually monitor that list most rigorously, I guess.

31:10Basically, the working group believes that... or the shepherd of the document believes that all comments were addressed and there haven't been new comments for a while. So if you have something to say, say it now or be silent forever, because we are going ahead with standardizing this. Another interesting thing that there's a bunch of different directorates in the ITF that will need to review a document or a draft during the last call. So, for example, the structure of the RFC actually matters, or the draft matters, because it may be used to, or the drafts may be parsed by machines, and then it needs to follow some very specific structure, or even just the publishing engine that they are using that will need to be able to parse it.

32:21References, for example, it matters where you put them. It matters what kind of reference you are using. There's informative, where you are just informing that, hey, this, I don't know, like from robots.txt, we reference sitemap.xml as an informative reference. But then there's also normative references. So for example, when we are talking about HTTP headers, we call in a normative reference to the HTTP standard, like 91.10, because we make claims about how something should be parsed, for example, in the HTTP header or how something should be interpreted in the HTTP header. And then a normative reference, basically a link to the doc that defines some behavior, let's say.

33:22And then someone extremely familiar with the topic that you are discussing in your draft is going to be asked to review your draft to see that no one actually missed anything that can be missed. So basically you get like a final review. And then once everyone is happy with, like every directorate is happy with your draft and there is no more last call comments, then it can be moved forward with standardization. And then there's at least two kinds of standards that I can think of, an IETF at least. And one is proposed standard, which basically it is a standard, but it's not immutable. Meaning that technically it could be changed or that's how I interpret it.

34:19And then there's actual standards like internet standards, like STD1, standard one, which defines immutable standard that the internet uses. Okay. Like you can add extensions to TCP, but technically it's immutable. Like the way it is right now, that's how it has to die. I think that's reasonable. then you can at least rely on that. And if you can still extend it without changing the underlying fundamental standard, I think that's fair. Right. Yeah. There might be limitations, but that's it. Okay, so then everyone agrees or you have explained why something got excluded and then it becomes a standard by basically being reviewed the final time by the directorates and then there's a last call period and then then we have a standard okay that's pretty cool you said that takes years yeah is that is that because the the consensus takes so long or do you need to have like a reference implementation or something or how does that how come that it takes I think both.

35:43I think it's both. So you have to show that the thing that you are working on actually works. And for that, usually when we are in a TLS working group, you would have adoption calls for new drafts if they don't have someone to work with already. Like, let's say that Martin came up with this new brilliant idea and needs someone to implement it as a test to show that it actually works. Like, have a proof of concept, I guess. Like, you need to show that it works. And then the other thing is that, especially with certain drafts, there's lots of back and forth on the mailing list about particular sections of something or even the general topic that you are discussing in your draft.

36:54And then argument is not the right. Basically, it's just like civil and constructive discussion about the draft or sections of the draft or multiple sections of the draft. And, you know, the internet, like people have opinions about stuff. And then you have to decide whether you address the comments and then you are able to move forward with your draft or take a step back and maybe revise that whole paragraph or the whole draft to exclude the part that people are upset about or nitpicking on. So basically there's tons of iteration going on and it makes the process very slow, but for a good reason.

37:56As we said, these standards actually are used by sometimes the whole internet. In case of TCP, for example, the whole internet is using it. So it has to be ironclad. There's no bigger room on there. Yeah, I mean, there's the saying, if you want to go fast, go alone. If you want to go far, go as a group. And I think this is one of the examples where going slower improves the quality and longevity of the outcome. And all of this is public, right? It's not happening behind closed doors. Okay. No, everything is public. And also our meetings are public. So technically anyone can join in and listen to what we are talking about.

38:49or even just say words in the meeting. Like there's no formal membership. You can just show up and contribute to standards. I don't know how it works with other standards buddies, but at least with ITF, you can just show up and say what you have to say in our meetings. For formal meetings, there's usually an entrance fee, but otherwise you can just show up I mean the fee is probably to cover the cost of the location and all the logistics of making it happen that's fair I mean like the last ITF meeting I went to that was in Bangkok it was a week long including the hackathon and they or we were using three floors of the hotel like literally all the meeting rooms that the hotel had available.

39:54So like it must cost an enormous amount of money. So they have to cooperate somehow because they don't have a profit. And as far as I know, that's pretty much the case for most of the standards bodies because I know that for W3C, it's very easy to set up an account. It's free. You can start your own working group and then do your thing. And then eventually you come out with something that looks like a proposal and then other people are being invited to comment on it. And then that whole process happens, but it's all public as well. I'm pretty sure the TC39, which governs JavaScript or ECMAScript, is doing more or less the same thing.