RWKV: Reinventing RNNs for the Transformer Era — with Eugene Cheah of UIlicious

29:45How I do so we'll cover later and this can be scaled to as many parameters as we currently what we have is dominant our main models is the 7b model and the 14b model which you can find on hugging face or respectively our demos we also have on there'll be the there'll be the rwkv raven models these are also instructionally tuned okay so there's world there's raven there's music oh my god this novel what what is all this okay so we before we were The current main models is RWKB4, POW, and Raven. So POW is basically just a POW plus model. What is POW plus? I know about POW, but what is POW plus? Random data sets that the community...

30:33How many tokens were? I would just say slightly 1.1 or 1.2 times the POW. Okay. Yeah, this is not instruction tuned and stuff. Yeah, the plus one is typically all the other languages. Subsequently, Raven are the instruction tuned model. This is the current main complete models. We subsequently have... And the instruction data sets are from... Typically, GPT-4, but then we scrub it for every move or the... As a large model. Yeah, this would be the uncensored... There's some other project that's kind of doing something similar, and they call it uncensored, but really they just scrubbed it as a large-end problem.

31:14Correct, yeah. So that makes it technically breaking TOS of OpenAI, right? Yeah. Okay, but that's a later problem. Listen, frankly, let's be honest, even if we don't remove it, someone is going to remove it. I mean, so there's ways around this, which is you get clean data sets that are not GPT-4. like so the one that I typically mention is Yannick Kulture's open assistant. I believe that was included as well. Yeah obviously all these release orders are all over the place. So okay Raven World. So Raven is the instruction team model. And then subsequently the world model is a new model that we are training.

31:56It's not 100 % complete yet. We have the focus on a new tokenizer and all the languages. All the languages. All the languages that we can grab from the internet. All the wikis in all the respective languages. What do you mean when you say all languages? 100 languages. Okay, fine. So 100 languages, it wasn't really a very precise sign. We just basically, whatever the wiki tool that allows us to download the ex-wiki languages, if it works, it's in the set. If it doesn't work, skip. And all the major prominent Oscar translation sets. So as you can see, PAL, red pyjama. Alright, what is OSCQR? OSCQR is just a common term that we use in...

32:37You can just search OSCQR in Haging Face dataset, and it just means translations. Okay. So, you can find English, X pairs, all the respective pairs. Okay, yeah. So, and then all tragedy data I can find. Okay, so 70 % English, 15 % multilang, 15 % code. Is there a strong grounding for why 15 % code? No, it was just... It was already there. Yeah. The focus of the world model was not to improve everything else. It was literally that 15 % multilang. We wanted to increase... It was English and code, and then you just added multilang. Yeah, we had a bit of multilang, but we wanted to bump it up. So this is primarily English?

33:18Whatever. Okay. Yeah. What I would like is basically like a visual of like, here's all the building blocks, and here's how they combine to create all these things. So we have the RMKV architecture code. So that's the main model building block. And basically we feed it the data. Power Plus, Red Pijama, then subsequently some of the code data. For the world model, we subsequently add on top of that all the translation Oscar sets and so on. And so you're training these things. You've mentioned that you're intentionally taking a hit on evals, on traditional evals like MMU or whatever. I wouldn't say intentionally.

33:53Also to clarify, I am not training it. I'm just part of the community. The community and bling someone training. But I would say it's more like the lack of care for the evals. So the reason why we add things to the data set was never about improving evals. It's about directly in response to user feedback. It's like, oh, not good enough at this. So they're like, okay, just touch it in. Yes, literally along those lines. So take for example, right? Even for Raven and the world model, as we go through the training stages, right? We specifically ask people in other nationalities within our Discord community to test it for their language.

34:36And our rule that we set is that, our informal rule is that the only person who can decide whether this improved word model is better in Japanese or Thai or whatever it is, is a native speaker. Yeah. Where does it take place? So it's mostly in linguistics, but sometimes we do a short term in general as well. Okay, linguistics. so why don't so do you have like a appointed ambassador like you have a hundred languages yeah you just have like a czar of Japanese a czar of Thai it's not so pointed it's more of like hey this is the Japanese model please try

35:15it's not there's no the Japanese model there's one model there's a world model so if you go to world model I don't know whether it's inside here no four sorry five is you should never put five on top because 5 is fully experimental. Okay. So, under file samplersions. I see. I see. I see. So, you see there's a Japanese specific tune, Chinese tune, Arabic. Then for all the other smaller languages, we actually ask them from the base world model. Yeah. A bit itself. we feedback on that. So, we actually released previously like 10 % train, 15%, 20%. Like, as it goes through the stages and then it's like, hey, is this working?

35:54Yeah. Is it regressing? so it's like evals but real humans done by real humans and not not systematically is there a reason that you release you also so you mentioned 7b 14 14b but i see also 0.1b 0.4b 3b 1.5b like what is is that useful for people or yeah is it just for research or 0.1 and 0.4 is frankly more for research but some people do try to make use of them nothing stopping them well i mean it's extra like these are just different architectures different dimensions yeah so it's actually extra costs to you to provide these things oh but specifically for the world model what because we are trying a new tokenizer we are and and the reason why we're trying a new tokenizer is that as as i think i'm cut is that one thing that we found more like i found surprisingly frustrating existing tokenizer was that it was very English centric.

36:50And the existing tokenizer you took from GPT-Neo? Yeah. Okay. I need to backtrack a little bit, just for people who are not following along. GPT-J was the original Luther reproduction of GPT-3 and then GPT-Neo was the bigger GPT-J? Yeah. 20B? Something like that. Yeah, I do believe they have a 20B model. Okay. and there's a check for I mean for those outside of the open source space in particular for the transformer I think one thing significant for GPT-NeoX was that it was one of the major models that had everything fully documented and they like why they make this change in the architecture and so on so forth and that became like a basically reference note for all other subsequent open source models because they were the early ones that were like doing doing a good transformer model.

37:41Yeah. And at least for the large language model. So GPT-2 was actually open source. You didn't, people didn't find that useful? No, people do find, do reference as well, but it's like the code is there. And why did you do this? Oh, I see it's not documented. I see. So in that sense, was OPT from Facebook useful? Because I've heard very good things about the logbook of OPT, where they had the daily logbook and they just published that. Yeah, those were useful as well. I think one thing that NeoX had going for it, especially the elevator committee, is that it's not just logbook, it's just like, you could just go and just go, hey, why you do this?

38:24And the person who trained it will tell you. Yep, someone there, hopefully. One of them. So that's why we had the 0.1 and 0.4 models because we were just in uncharted waters here. So a lot of existing tokenizers took space as a major delimiter to detect and split. And the tokenizer we are using is actually a lot more simplified. So as these tokenizers, I mean, they scan all the text, they do a statistical model of what pairs well with what and so on and so forth. We did a similar approach, but we instead of using this token pairs well with this and should be paired with that. We just made it a trial list.

39:05So basically, we find the... Try the data structure? Yeah. So we just find the longest matching string in that matching string that we have trained inside our token list. And then we just use it as a token. It's a drastically simplified tokenizer. And it doesn't use spaces as an assumption, which I know... Which is good. Yeah. And that helps a lot of the Japanese, Chinese, and character models, they don't have spaces. And I would even argue to fair say, if you look at the really large models, like OpenAI or Cardia, tokenizers are not really a thing. I mean, in the sense that the model can work even if you tell it character by character.

39:51It's maybe inefficient. There's someone tried? I mean, there was that jailbreak where the system prompt, you put the character, then enter, enter, enter. Do you remember that geographic? No, I didn't see that one. Yeah, so you can literally, like instead of like left to right, you can literally up to down. Okay. And you're just eating tokens for every character. No, actually you're eating two because there's also the new line. And the model understood it because there's enough dumb data on the internet that it has learned how to deal with this kind of formatting. Got it. Okay. And if these models are already understanding things at the character level.

40:28Everything else is just improved compute. Because we jump the multiple tokens. Do you have any idea of your dictionary size when you use this 3D data structure? Because the typical tokenizer is like 80 ,000 tokens, dictionary size. I presume yours will be bigger. Yeah, I can't remember offhand. Our previous tokenizer is around 50 ,000. It's the new X tokenizer. Then subsequently, I believe this is around the same size. It's want to change too much on that size but we just wanted to like just change the format yeah cool i actually kind of want to establish the credentials of this thing so who is blink is rando on the internet or like because again never heard of this guy until published this is real me right and you had like i have this paper to work with but it was only published in may yeah you found this before the paper and and so i think it's very unusual for a researcher to effectively launch to the wider public without a paper and just get some kind of pretty decent community going and then publish the paper.

41:38I think a few years back, with GBT2, Transformers started to pick up Steam. And I guess the whole world is starting to think let's just abandon neural networks. So we haven't even gone into the code part, but the main reason why neural networks were bad compared to Transformer was that when you train a token, let's say you just input a token, and train a token for data samples, you have to wait for the compute to finish for that token, take the state, and then you train the next token. And we'll get into how RRWK solves that. But basically the whole work at that point just concluded, yeah, neural networks cannot scale as well.

42:14Transformer, let's just abandon it. And everyone just went in that direction. And Blink, or Bupeng is the actual name, decided basically as an individual literally at the Elutter AI forum decided that hey I think we can modify recurrent neural networks based on the Apple paper the light attention that I showed previously to make to scale this up without to make neural networks scalable and paralysable in the same way transformers work because the reason why we branch away and focus on transformers is because neural networks were slow to train it was never I mean it wasn't so much about whether was it good or bad it was just no one wants to wait a hundred years for their billion tokens to train finish even if they can throw a GPU farm at it and and that's where he started taking looking into it like how to make the neural networks trainable in parallel and and specifically RNNs yes and subsequently with the AI and I believe there was also a few others like because he was doing it very publicly there came on board to sponsor the GPU computes required.

43:25Because even though I mentioned that on large context size, it is substantially cheaper, I think, especially if you run an open source Discord forum for an AI model, it's like every day there'll be someone who thinks that they can train a 20D model on a single GPU coming in. The scale is still large. Even though it's like 1.5 or 1.10 compared to Transformer, it still needs a lot of GPU. So that's where Aether, AI, and the rest, Stability, I believe also, is involved, stepped up and donated the A100s needed to train the basic models that RWKB had. So before those models were trained, we were only having, in theory, the toy models or the smaller models that this can match Transformer.

44:14We have no idea whether it can match Transformer at that scale. Yeah, and subsequently with the larger models, the 14D models and all that, we can compare it directly with NuEx model. And that's where this paper came out. So that's the history behind it. It's like he wasn't really doing it in silence. He was doing it from Elutter. Then he branched out because this became a big project on its own. And that's where other people started coming in. Let's go. So the part where we say that RwKV is a neural network that can be scaled, can be rolled out as a transformer, right? Yep. The key thing that you want to see, right, is this diagram here.

44:56Yep. This should be in the paper. No, sorry. Yeah. Yeah. Accordingly. So what you get? So when you do inference, when you are running inference mode, ideally you should run it as a neural network. So this is a layer. so classic neural networks is that you have a state the state could be start from blank you process a token you output a state and then you rinse and repeat and then as it keeps doing the output it makes a prediction in that one thing that so subsequently for RWKB what happens here right is that we can roll out this neural network side by side and then it runs similar to transformer but the key thing here is that the states are split across the layer.

45:42So this is what we call in this diagram here specifically, this is what we call the timings and channel mix. These are operations within the layer. Depending on how you view it, you could view this as individual layers or as how we view it. We view like this collection of layers as one layer block and each layer block pass the states to its sibling subsequently down the road. As you process the next token which is a similar RNN type feature. However, the key thing is you do not need to wait for the upper layers to complete before you can go to the next token. So what happens in practice? You have to jump to the diagram like this, this graphic here.

46:25This is not 100 % how it runs behind the scene. I like it. Yeah. Whoever put time into this, kudos. I made it. So this is how you can visualize it. So the first layer is the layer norm. The layer norm doesn't... This is standard layer normalization. It doesn't need to... It just doesn't have a token and doesn't need to wait for the other layers. But if you notice, right, subsequently to the right and to the top, these tokens, these blocks, right, need to wait for the blocks on the left. And this is like, once you go past the first few tokens, right, this cascades very rapidly. Especially, like, this is only like one, two, three, four layers.

47:05Most models have like 20, 40 plus layers. and the cascading patterns are happening. And in practice, once you start cascading there, you just saturate the GPU. And that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs. That was one of the key things. What else is the key thing? So other things is that, so I think you're familiar with LSTM, right? This is how traditional neural networks keeps things within memories. In RLKV, we have two channels. We call it the channel mix and the time mix respectively. Is there a formal definition of channel mix and time mix?

47:41Yeah. You can see the data from the respective time mix and channel mix move to the next segment. How time mix is designed, per se, was that it's how it retains, similar to LSDMs, right, where it processes a state and the input. It may decide to discard certain states and keep new things in the state. Time mix does the same thing, but with a different formula. so it replaces the LSTM in a sense and and it can decide to keep things indefinitely so this represents a long-term memory if you want to be with it that way but classically the problem with that is that it struggles with long distance correct does it have the same issue so that's that's subsequent it struggles with long distance because it also needs to keep track of both near-term memory and long-term memory so you split it up.

48:32Yeah, effectively split up. So channel mix is... Is this the perfect memory? Yeah, this is the closer to the perfect memory than it's the short term. So, so, time mix, it has trainable weights on what it decides to keep in this card. Channel mix, it has a very strong bias in it towards, like, just the next token. So, so, so, so subsequently, it was just like as, like, memories are stored in the lower layers, it just slowly shifts upwards through the channel mix. And this is the short-term memory, which at some point, as it just shifts all the way up, it will just disappear into the void. At that point, subsequently, then time mix should be retaining the longer-term memory.

49:14So we took a break for a bit, but now we're trying to cover, like, what is the big aha moment for you? And you said it was something to do with cost. Correct. So we have this chart on screen. There's literally a chart of quadratic scaling versus linear scaling in terms of GPU time spent in text generation. And you said it was at training time and at inference time. Just basically in everything that matters. All right. So, I mean, so look back to how RNN works. From a high level, we do an O1 operation on a token, create a state, O1 operation, create a state. So, this just scales linearly. You want to throw a thousand tokens at it?

49:53On inference, it just scales linearly. And subsequently, for Transformers, you take in a token, you process your first token. It may be O1 here. Subsequently, when you generate your third token, you need to compute your second and first. And then it rises. So if you do your 1 ,000 token, you need to compute back your 999 previous tokens. And as this keeps growing and growing, this is your quadratic scaling. and this is why we had this graph of the amount of cumulative GPU time that you need to spend to generate all these tokens respectively. And this is fundamentally just transformers versus neural networks.

50:34Yeah, on inference. The reason why and subsequently, neural networks did have the disadvantage of, let's say, not being able to paralyze well in training. But as I covered, RWKB kind of solved that by splitting, effectively splitting the layers, allowing you to train different parts in parallel. And like some people will go into the academic debate of like technically the second and third token is not paralysable until the first is done. But once you get into like, I can saturate a GPU land, it's just way better. It's just academic debate. We are done. And so training in essence has always, I mean, this is a bit of a transformer.

51:12On your network is I need to do an inference pass. I look at the logits. then I backprop to see what went wrong and I update the weights so the inference is the forward pass it's part of the training course as you backprop as well having needing to only look at the current cell tokens and the state instead of everything also reduce the amount of things that you need to backprop so it's just that there's so many factors involved in just reducing the overall inference and training time and that was something that appeared to me because in the long run I mean all of us want our model just run blazingly fast right?

51:48Yeah and also on minimal hardware. Oh yes Dan. Which as far as I understand you still have 14 billion parameters that's not going away. You still need the RAM to store 14 billion parameters worth of stuff. That's not going away. Yeah. Okay so RAM is unchanged. Yeah on the RAM side but the working memory is reduced. So typically you need more than 14 for transformer. I mean let's not touch quantization but in this case we don't need to keep like if you really really want to like save RAM you it is possible for you to do token by token inference so that you don't need to keep your states in history you only need to keep your current token state and your next yeah okay and yeah and that's actually like one segment of our community just purely porting RRKB to C++ based model or in the next yeah and running it on pies and stuff raspberry pies yeah it's interesting to watch there's a chart about performance and it shows that rwkb is competitive or actually better in some of the reasoning challenges which that's something i definitely would look for right like and it's it's fine if like your your speed is faster and all that but if the reasoning quality sucks then it's not a very useful language model exactly so so this is like literally us saying there's no trade-offs.

53:14Yeah, you don't lose out in that process. Okay. Big question then. Why isn't R2KV a bigger deal right now? So, one, we are not a commercial organization. Okay. This is literally the pure open source play. But you could have done the Stable Diffusion thing. Which, you know, Stable Diffusion launched. It was by a bunch of nobodies before that. It's from, like, literally split out from Luther. And, But they definitely had some hype. They definitely, like, you know, I interviewed Sharif, Shamim. The reason I ask you so many things about how did you find out how to give you, because I think the generalizable skill is how to be early in AI.

53:53Because being early in AI is very valuable. Because then you were there to see how things developed instead of, like, picking it up later like me. Anyway, so, yeah, why is it not a big deal? You don't need to be frank. Yeah. We just suck at marketing. Okay. That's fair. I mean, this is part of it. Yeah, this is part of it. So, like, maybe... But I don't think that is entirely the cause. Yeah, I'm sure you're definitely... I think the other major segment right now as well is that we were really late on the paper. Okay? Like, one of the weirdest things right now, weirdest thing right now I feel that is that RRBKB is starting to have its moment right now.

54:33Okay. Is that ever since that initial paper came out, there was ResNet, there's a... I think there's two more, there's a few more additional papers. coming up, one from Microsoft, one from other organizations that are literally exploring the whole idea, once again, of scalable neural networks. And they are citing RWKB as part of it as well. And I think for most, I think it's existingly, why switch to this model when, even though we have proven that, yes, it's scalable to 7 and 14, and that it can match transformers at similar param and training size. But all this is very academic because the community, right, the community at large, especially for the English-speaking community, right, they don't really care about this.

55:26They care about what's the best model that I can run on my computer, at least within the open source space. And even though we match in performance for things in the same data set, the keyword is same data set. Like, this benchmark is not even red pajamas yet. It's the power. And when you have models that are, be it like Falcon, being trained on much larger data set, especially for an English use case, it makes more sense to use that. I see. So there will be another paper coming that is RWKV trained on red pajama that will presumably be a larger data set. Yeah, and so on and so forth. So I think that's the...

56:06we are still in the stages of reaching that point where we train on the larger dataset. The only reason why we have a bigger outsized impact compared to the other models is frankly because half of our Discord came in not for English, it's for other languages. Yeah, that's great. And there is a definite very US and English-centric bias towards these models. And it's, to me, kind of poetic. Like there's nothing in the architecture of RDDKV that particularly bias it to be really good at other languages. It's just that as a community, you decided to prioritize it in your tokenization, in your datasets.

56:46That's it. Yeah, that's it. I would even argue that I'm more surprised that, especially on the European side of things, that we don't have more models that actually focus on even the European languages because there is like a softer jump to character Japanese and Chinese characters they're all romantic but I think back to the benchmark what excites me most still about this is that it just means that we just need to scale we just need to scale this model and we derive data to like 40B 40B 60B I mean params is one thing it's data sets and GPU time so you and I are talking offline about ideas for getting data, getting compute.

57:32So this is like a project that's ongoing. Okay, anything else for the future of Rwkb? The biggest one would be... Okay, so this is back to how, remember I said, evals doesn't hide or doesn't highlight everything. Being realistic on another weakness on Rwkb's side is that now with the rise of, let's say, 100k or 32k context size windows, transformable order, RfKB currently is trained to handle, let's say, 8 or even some people have already trained it to 16K sizes. It has, and well, as a neural network, it will happily keep going on for infinite context, man. It will just keep generating. Does it do well?

58:16The answer is no because if you didn't train it to handle that situation, and that's actually a charge. So, for example, if the prediction, the power test loss, right, it does improve over time, let's say, if we go down the context length, but this is if we train it. And what is not seen here is that if we were to, let's say, run it further, it'll just go back up. Because it was not trained to handle that. Well, it technically can run, it suffers from the longer context length, and that is the part where other TV, especially in Q &A tasks, in huge documents, like, you get closer to summarize giant documents.

58:54None of this is fundamental, it's just you need more money. yeah that's it and no there is actually a fundamental part so what what one of the things that i was doing i am actively helping within the committee right now is that we found that the existing way to scale the memory was not that efficient and we were just being realistic ourselves if we want to hit 100k we need to change this so so one thing one thing that i'm actually looking forward to right now is actually those experiments we have already we have already started scaling things to be able to handle things at transformer scale, be it the 4k, 8k, in terms of how it handles memory really well.

59:31And we found it, we want to extend it to be like 16, 32, and 64. And that is within our roadmap. And that's the exciting thing, because once we have that, it's able to handle long-term memory within those sizes, it removes what many people in the community felt, right, was the last architectural limit. Because once it's able to handle memories like context length, the same as Transformer, we know already to do all those, like, you know how existingly, like, people do, like, long conversation in Transformer, they just discard the rest and the sliding window. This is, like, the better version of the sliding window you have.

1:00:09The model can handle the sliding window perfectly. It can keep remnants behind it. Sure. And that's something that I'm really excited and invested towards because this is back to the full circle of how I came into RK. I want my model to handle 100k tokens, 4 megabytes of HTML, whatever I throw at it and be able to process it. But it'll be lossy. The later half will be lossy, but the key thing is extending the non-lossy part. And we are aiming to extend the non-lossy part. So, you have displayed today an impressive amount of knowledge just across all this stuff and you don't have a research background.

1:00:53Your advice to AI engineers getting as deep as you, who want to get as deep as you. So, I think your article articulated very well that there's going to be divisions within how we approach this. So, AI engineers, model trainers and data set curators and ML scientists. So I'll loosely define as a tree. I ignore the full stack because every company needs it. So within this tree space, there is actually a lot of ways anyone can come in without knowing anything. So let's just start with AI engineers. Don't be like, even though this whole topic, we even dived into how the layers work. We also showed how the math works.

1:01:33Frankly, for an AI engineer, you don't need it. your main thing that you needed to do was to frankly just play around with chatgbt all the alternatives be aware of the alternatives just be very mercenary swap out to cloud there if it's better for you or swap out to an open source if it's better for you and just play around the prompts learn bare prompting techniques like one shot, two shot, few shots and then from that on you can start building your agents stacking your prompts and in sequences and stuff like that and you are able to build applications that do anything in terms of the AI space and all this without knowing all this nerdy stuff with all the hard engineering because that's all you really need to actually build a product for the user, remember you are supposed to focus on making it for the user they don't care if it's RWKV or Transformer underneath the hood they just care that it helps them and I would say like Notion probably is like probably one good example of how they use it because we know underneath the hood is OpenAI but you really use it's OpenAI yeah so I obviously agree with all that let's just say that people are there already and they're just curious they want to do what you did so that's where you start going down the layers yes so the next layer the next layer you go down in is subsequently training the model from scratch fine tuning and incorporating the data set and this is this is from this is where you still do not need to know the math but you need to know like you need to have a rough sensing on how the model works and how the certain models and in this even within the open source transform certain models are better trained in certain sequences with certain learning rates and you just need to get a few of it so this is just like like the data set try it see the loss you literally did this yeah at least for RWKB and the That's a lot of work.

1:03:33Code Gen 1. Yeah, it's not a cheap work tool because you need GPUs. Okay. And that took you how long? I think Code Gen 1 alone was like six months. And then this other KB, I've been doing this for like another six months. And that is just pure experimentation. Like there's no right or wrong because like, especially if it's in a different domain. Like recently I was helping someone on the RGB Discord regarding the music generation domain. And my assumptions for learning rate and all the patterns were just completely thrown out the window because the music model just fundamentally is different in those sense.

1:04:11So that is the exciting thing is because it doesn't really have any specific rules and guidelines until you get until you travel to a certain space. It also means that you coming in is as fresh as anyone else coming in last year. it's really that kind of uncharted space for everyone and and especially as you start exploring new domains your existing your existing knowledge may actually matter because sometimes like i mean i think a few papers already covered this that's like how you train your model in certain sequences also matter like you want to train a certain set of knowledge and then and then you extend that knowledge subsequently but if you're talking about material science or genetics how am i supposed to know what is foundational knowledge what is extended knowledge i have no idea maybe you do and i'm just picking an example yeah so and the same thing for music and so on so those are things where even though you're outside the space is where you can come in just at the data set level now you want to peel off to the next layer let's just say let's just say you want to look into modifying the model the the the the the foundations of it i think one of the beauties about this current boom is that even though I did my toes early, like before the transform wave and into the early neural network phase, frankly almost everything that matters was basically in the past four years.

1:05:40Like there were a lot of things that fit in academics that were before that and you know and they were mostly dealing with models that were under a billion parameters they pretty much no longer matter and can you be more specific like okay I know I'm shooting myself in the foot because RRK is a neural network but if you're just trying to get transformers to work you don't need to know LSTM yes yeah you don't yeah there's a lot of like pre-knowledge in neural networks that is irrelevant in the transformer era and maybe some of it will have a resurgence but to get up and running is not a requirement.

1:06:21And I think this is where you could either go the very academic way of reading papers and stuff. But frankly, what I found was way more useful was Akapati. His series of videos. Serious a Hero. Yeah, that is really, really good. I think even though I read some of the papers and guides before that, it really helps that it starts from zero because you can see how it happens part by part. And even though we will not use the exact same code that he used because he re-implemented the backprop and all that, and we're just going to use Torch for that, that's where you get the aha moments on how these building blocks work and how it falls into place.

1:07:06And I had a fundamental misunderstanding of how backprop worked until I actually watched this video. Oh, really? Yeah. And I think that's the scariest and craziest thing about AI models. sometimes is that you can actually have fundamental misunderstandings but as long as you make the building blocks and you connect and okay loss is great it works yeah well so you know even the gods of the industry you know I don't know if you read the Swigulu paper so there's these like there's all alternative activation functions like there's ReLU and then there are people are always looking for different slopes and very famously the Swigulu paper had this line in there that was like yeah we don't know I don't know why this works but it works.

1:07:49Can't explain it. It literally happens here and there on the Discord too. One of the funny things that I'm doing right now in RWKV 5 experiments is that, okay, we are going to do this change, we are going to run this train, make your prediction. Will this model beat this model in this lost curve? As a game? As a betting? It's a very informal It's literally a buddy Kind of like Kind of bed But The fact that The fact that we can do This kind of bed Even though we like Understand The code It's like It just goes to show Like how often Like oh wait This didn't go What we predicted No one And And that's why Even if let's say You don't have a PhD Or So on so forth Heck Even if Math is not your special agent you're coming in as a developer.

1:08:40I'm going to say frankly, I didn't come from the research right now, the extremely math-heavy stuff is what I struggle with. What I do sometimes is I copy and paste the math into GPT-4 and ask it to explain to me. Which is good in plainer language. It's very good at that. Yeah. But the thing is, there is lots of value beyond that. One thing that I realized, and this is not specific to RWKB, this is what happens across a lot of open source models is that a lot of ML scientists when they really build this stuff the focus was more of like always get it to work it was never about getting it to work efficiently or getting the code documented or organized and stable diffusion literally went through this whole journey they had the giant they had the code and the model that worked and the community just started and engineers that came in with zero machine learning background, started picking it apart.

1:09:38It's like, no, you could replace this with this that does the exact same thing, but it's more efficient. Like, one of the major breakthroughs, for example, for GML, and this happened some time back for a bit, was that someone external from the AI community went in and implemented memory mapping. Yes, I saw that. I forget her name, but yeah. Justine.law is her URL. Yeah. And she didn't come in as an AI expert. She came in as a software engineer. Yeah. And these are all just very, very straightforward. You know, in her world, this is normal. Whereas for the researchers, they will be like, I don't want to.

1:10:19Wait, what is memory map? Yeah, exactly. Yeah, and there are a lot of things. Like, one of the jokes that I have right now is that every month there is a research ML scientist that's rediscovering the number 32. Why? Because be it like someone in the community writing the inference code, because GPUs, especially QDA GPUs, tends to work really well when they align to the batch size of multiples of 32. And if you've been in the gaming industry, especially when you write shader code, this is like well-known, just given knowledge. And people are just constantly rediscovering, oh, maybe if I just adjust my data set, my data size to fit this batch size suddenly I get 10 % improvement and yeah and I was like these are things that once again because they were so focused on just making it work that they won't know outside space and that's why I would say right if anything right now is the best time is that you don't know AI to have people from different background come in because your contribution could be from dataset level, how to train the knowledge, to shader code, to hack, how to memory map, how to cache data.

1:11:37There's so many gaps. Cool. Great. So yeah, thanks so much for being very willing to get on and talk with no prep. We did some prep, but it's very unusual podcast episode, but I really enjoyed it. We literally just met yesterday in Singapore. But I know you've been on the Discord for a while and I can tell you like you're very serious about all this. I think it's very unusual for someone like you have a job, but this is like a second job, essentially. But you are really enthusiastic and passionate about it. And I think that's very rare. And I don't want to encourage more people to do it. And so thanks for sharing.

1:12:09Thanks for having me here.