Unlocking Legal Insights: World's Largest Open-Source LLM Dataset Unveiled

0:00The headline here is that Allen Institute for artificial intelligence has pretty much unveiled what they're calling dolma so this is the largest open source language model data set this is amazing this is so heartwarming to me because this is what i think ai should be all about this is what's really going to help advance um ai forward whether it's this or you know future iterations of this they're setting the example here and i'm i'm here for it so the allist institute for artificial intelligence um is the it's ai2 um recently unveiled this massive open source data set which is tailored for training artificial language models.

0:34So it has a 3 trillion token size, and Dolma really, I think, because of that has emerged as the largest open source data set in the domain period. So the creation of Dolma is part of AI2's OLMO initiative, which is really just geared towards establishing an open, transparent language model, which is definitely a very stark contrast to someone like OpenAI, who has a very black box when it comes to the data they put in there for a number of reasons, right? They say it's because they're like, well, you know, it's our copyright or it's our intellectual property. But really, I think a big problem is people would be mad if they knew where all of the data came from, right?

1:16If they're going and scraping all of your Reddit posts and then using those to train and then charging you for them, people would be upset about that. So I think that's another reason why OpenAI is very non-transparent. So in creating this current data set, the Allen Institute really kind of stuck to five core principles. Number one, openness, representativeness, size, reproducibility, and risk mitigation. Those are kind of the five areas. And I want to go over what those mean kind of according to them. So openness, AI2 responded to the researchers' communities pressing concerns about restricted access to pre-training CORPA and associated language models.

1:55Essentially, they envisioned Dolma as a transparent data set that is open for thorough examination, right? What a novel concept. You can literally go see what's inside of it, examine the contents. If there's something problematic or something illegal or something copyrighted, it should be able to be, you know, removed theoretically. And so I think this is really interesting. And this is a first step towards that. So as far as, you know, the whole representativeness. They aimed for Dolma's content to stand toe-to-toe with datasets employed by both private and open language models. Ahem, you know, open AI or closed AI.

2:32The third thing is size. So recognizing the potential benefits of large datasets, AI2 incorporated findings from research that underscore the correlation between larger training datasets and heightened model performance. The fourth thing is reproducibility. So AI too is really focused on community engagement, ensuring that any tools created during Dolma's creation are open for others to replicate or modify. So honestly, they're doing a huge service to the entire community with this kind of open source, open concept approach. And I think a lot of people really appreciate that. So the fifth thing is risk mitigation.

3:11Designing DOMA with a strong emphasis on privacy, the institute ensured that its web crawled content could not be traced to actual individuals. This is interesting, right? Because on the one hand, they're saying we have a very open data set where you can find, you know, everything that's in it. But they stop one step short of actually saying, you know, who said what in the data set, which is interesting, right? So like if they went and scraped Reddit, for example, they'd tell you maybe what subreddit stuff appeared on but they wouldn't tell you what the user's names are now that's interesting because perhaps that's for privacy reasons um but it does yeah i guess there's some tricky tricky aspects of that when it comes to like attributing certain content to someone and the the reason i say that is like if they ever were to do some play that i've heard this isn't just like some moonshot like i've heard people float this idea where essentially you compensate people when their data is uh used but really what that would look like is more like you're compensating like reddit as a whole and not an individual contributor that actually wrote a post that's getting used or you know so i'm not sure um how that plays into it but it's probably for legal reasons and whatnot they got to keep the the privacy of the individuals that have written something private beyond just like the website it came from so in any case this is i guess it's just another reason that if you are a content creator you need like your own platform your own website to host a lot of your stuff and then it's probably a lot easier to claim it when these AIs inevitably suck up all your data and train models off of it.

4:41So I think developing Dolma wasn't a very easy task, right? This is the biggest open source data set ever. And so I would not expect this to be easy. The team grappled with some really critical decisions surrounding data sources, like I just mentioned. And then of course, pre-processing methods, language inclusion, and strategies for personal data exclusion. and essentially by employing a bunch of different practices that are pretty standard for data filtration and duplication um they were and this is also coupled with you know measures to kind of weed out potentially harmful content they essentially produce this data set with a really rich um like conglomeration of everything on the web so that was english literature from product Gutenberg scholarly papers from PES20 wikipedia entries and code snippets from github unlike many private databases that really kind of shroud their components in mystery dolma really prioritized transparency here so even among its open data set peers dolma i think kind of stands out um just due to its sheer size and also its unique licensing terms so i'm sure a lot of people are interested right?

5:51What is the licensing terms here for this? So licensing under the Allen Institute's impact terms, it's classified as a medium risk artifact. So for those looking to tap into DOLMA's potential, it comes with conditions centered on transparency, risk reduction, and ethical usage. The license explicitly restricts its use in military operations, surveillance activities, or for generating disinformation, which I mean, sure, I'm sure everyone's like chat GPT should put that in their terms like can't use it for generating disinformation i don't know it's kind of funny like it just like it goes without saying but also you know people do it anyways so like the people that are generating disinformation probably don't really care about the terms but in any case this licensing approach kind of strikes a balance between accessibility and the potential hazards of circulating such an extensive data set or so they think but i mean i'm not 100 sure i'm not 100 sure what that means right because like let's say you download this data set use it for reasons you shouldn't and like let's say north korea creates an ai that creates disinformation based off this data set what are they going to do like yell north korea and tell them they have to give back the data like too late so i don't know um i'm not saying that i actually think it's a good thing they created this but i just i don't know so i think maybe in a lot of cases like this they put stuff in their terms of service just so when something bad happens they can inevitably say like it's against our terms of service we told you not to but whatever it's not it's not a big deal it's the internet all the information's out there anyways.

7:19I'm sure North Korea could build a giant web crawler scraper bot, or probably already has, and then sucked up all this info anyways. But in any case, I think prospective users are going to be required to share their contact information and articulate their intended dataset usage. Upon approval, they can download Dolma from the Hugging Face platform and for a deeper dive into the dataset. A comprehensive data sheet is also going to be available. Additionally, a GitHub repository complete with code and instructions on dolma's generation and exploration has been provided so ai2 has really big plans for dolma i think they're intending to augment it with more data sources and languages in the future which is going to be interesting right multiple languages beyond just english and with this whole endeavor the institute aspires to really champion openness in ai in research and kind of pave the way for superior language technology advancements overall all, I think, you know, like a big tip of the hat to the Institute for doing this.

8:16I'm really excited for more projects like this to come forward. If we have a ton of open source models, a ton of open source data sets, this is like, this is amazing because this is how the community can build really creative, impressive tools, not get stifled by the costs of API calls or other things that maybe we are unable to afford. And so I think this is going to be a really interesting, exciting place to watch in the future.