86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic

0:00In transformer models, the cost of attention grows with the square of the context length. So doubling context makes the compute roughly four times more expensive. And Alex and his team have solved this issue. When we've done benchmarking of Frontier Models with SueBench Pro, we found that 86 % of the steps were actually read steps, just trying to do that context engineering before execution review, which was only the last 14%. Is this the end of RAG? If you could put everything that's in your RAG knowledge base into the context window, you wouldn't need that more complicated architecture. Is that right?

0:41Yeah, it is. Although I wouldn't say it's the end of RAG, just because it's too extreme. All of these enterprises that are sitting on massive amounts of data, and a lot of that data has yet to be put to use in an AI product, they're getting pushed to do like a$10 million data a transformation project before they could even start to build a product on top. And value prop that we bring to the table is that is no longer necessary. Hi, this is Eye on AI. And today we're talking to Alex Whedon about his company SubQ, which solves a problem that plagues transformer models. In transformer models, the cost of attention grows with the square of the context length.

1:30So doubling context makes a compute roughly four times more expensive. That's what we mean by quadratic compute complexity. And Alex and his team have solved this issue. uh so alex can you start by introducing yourself to listeners and we'll we'll go from there sure i'm alex co-founder and cto here at subquadratic um i've been in the space for about 10 years i started working with uh language models since before transformers existed when lstms were still the hot thing. And our mission here is to always be the first to make the most computationally memory and sample efficient algorithms and build foundation models on top of those.

2:27Yeah. And can you explain, you've tackled this quadratic complexity with sparse attention am i right and can you explain what sparse attention is in simple terms and why it matters compared to standard transformer attention sure so um the quadratic bottleneck of attention is introduced because attention compares two token relationships um so you have 1 ,000 tokens, then there are 1 ,000 squared two-token relationships or a million squared two-token relationships for a million tokens. You have this quadratically growing set of math that you're going to perform with an attention. Sparse attention says we don't actually need all of those token relationships.

3:19And so sparse attention attempts to find which token relationships are important before running attention on only a subset of the token relationships. and one of the main challenges with sparse attention is that there's always been a quality trade-off as you can imagine like um in theory if you're not looking at every token relationship then you can potentially miss some important ones however you're also looking at a lot that don't actually matter um and so the challenge of sparse attention has been figuring out which ones actually matter most sparse attention mechanisms have used fixed patterns so um based off of the distance between the target token and all the other potential tokens.

4:02There's some fixed mathematical formula that determines which tokens should be passed through attention. DeepSeek Sparse Attention upended this by saying we're actually going to use a separate prediction model to determine which tokens are important. And that worked. And they showed that if you dynamically select the token relationships before running attention over a subset then you can do so without a quality trade-off the only problem is that prediction model that selects the token relationships uses full attention and um and so you still have the quadratic problem in fact that uh sparse sorry that uh that separate selection model uses more compute than the sparse attention layers in the full model um at just 52 000 tokens and then continuously grows to an even larger differential.

4:56And so what we've done that's unique is we have a sparse attention mechanism that, like DeepSeek sparse attention, dynamically selects the token relationships, but does so without the massive cost of full attention and does so without a separate selection mechanism. So the single model does both the selection generation at the same time. And can you give us a practical use case where this quadratic compute complexity makes it too expensive to run the model on something? I mean, code bases seems like an obvious, but are there other situations where you want a very large context window and that it would get unreasonably expensive to do that with regular attention?

5:52I mean, regular transformer algorithms. Yeah, absolutely. I mean, I think a very significant portion of enterprise workloads fall into this category. And by the way, I will just clarify now that we're not just after the absolute context window size. We're after higher intelligence within that context window, lower cost of context, lower latency for context. And so these are actually four different axes of context we care about, size, intelligence, cost and latency. and enterprise workloads are dramatically impacted by these. Everybody's making trade-offs. Everybody's trying to reduce the number of tokens that they're using as much as possible.

6:33Everybody is doing context engineering that requires a lot of effort and also narrows the scope of what that system can actually handle. But some of the top use cases that we see are, yes, code is obviously one. Thinking about code review planning and long horizon memory, those are all long context problems. Thinking about the fact that when we've done benchmarking of Frontier Models with Sweebench Pro, we found that 86 % of the steps were actually read steps, just trying to do that context engineering before the execution review, which was only the last 14%.

7:15And then financial document analysis is another really big one. Financial documents can be a couple hundred pages. um and if you really want to analyze trends then it's helpful to have many of those documents over a period of time or several different companies um we've in an enterprise you have hundreds of billions of tokens of textual data and anytime you want to to tackle a problem um it's helpful to have like a broad range of documents um the key point that we're that we're conveying um and working on with enterprises is how do we do coarser grain retrieval more retrieval steps at once um how do we avoid chucking documents as much as possible or make larger more comprehensive chunks etc to offset the quality loss that we often see with a lot of the context engineering that's done today.

8:15Yeah, and that quality offset, or you said earlier a moment ago that it's not only the cost, but it's the intelligence within the context. Is that how you phrased it? Because large context windows, many LLMs don't operate optimally within a very large context window. They don't see things or they get lost in the context. Can you talk about that? How do you solve that? even though you can now do very large contexts efficiently? It's a great question.

9:10So, yeah, the intelligence within the context window is a very important topic. If we just take a step back and think about what Frontier contexts has looked like historically, we didn't get Frontier models with more than a 256 ,000 token context window until March when the Opus 4.6 general access, 1 million token general access came available. And so until then, most enterprises were building with a less than 256K usable context window. You see that, so a lot of people that we talked to, like they're using 100 ,000 tokens or less and they haven't yet adapted their workflows or RAG setups to use more than that.

9:58Sometimes they feel a quality trade-off above that. With particularly hard problems, actually 100 ,000 is a fairly common cutoff, surprisingly. In the coding space, a lot of people set the compaction limit to 400 ,000 tokens because that's where they feel there is a qualitative trade-off within a 1 million token context window. Because of this, in addition to intelligence, I would say there's also an alignment problem within the context window that user preferences with how a model should handle, should leverage 1 million tokens are not super robustly understood today. There are just not enough enterprise applications built to actually leverage a full 1 million token context window to really understand what that should look like.

10:45And so as we push into the multi-millions of tokens, this is very much white space, not even just from a technical perspective, but from our product perspective. what the users want. There are all sorts of benchmarks that kind of measure human preferences at short context, but a long context does not exist. And then finally, I will just add that one of the challenges with large context intelligence and user preference matching is the amount of training that you can do at large context. And so that's one of the main reasons that we set out to create SSA in the first place was that it is cost prohibitive to do large scale training on large context.

11:26As far as we know, we are the only people to do robust multi-million token pre-training, which we think differentiates our models. So like the number of times that a model can see how to handle a certain situation at a reason over some context at large scale, as that number goes up, obviously the model gets a lot better. that's really hard to do with quadratically scaling compute costs so it could go a little deeper on that i mean you've trained uh and evaluated up to 12 million token context windows so explain how the how you avoid uh the problems that large context windows uh create i mean you you go a little deeper on on how how the model approaches that yeah so i mean i feel like i i kind of buried the lead on that with one of my earlier answers but uh i mean the problem with sparse attention historically has been that uh their fixed pattern mechanisms um like sliding window attention look at the last 1000 tokens that closest um and um and uh or before and after but um but these fixed patterns don't robustly model language so like there's no guarantee that the first and seventh token are always going to be relevant to each other um it's highly context or content dependent and so you need a mechanism that accurately models context, accurately determines which relationships are going to be important, no matter what the input looks like.

13:18And so while deep-sea sparse attention uses a separate model to do that, reintroducing the quadratic problem, we do not. We have a mechanism that is much more lightweight and cheaper and leverages the existing model that does the generation to do the selection as well um so it's super cheap super fast um but also dynamic so every input gets a different um attention map and um and then we train our we leverage that the fact that we can do this dynamic selection to achieve high quality results while also having efficiency differentials. We are around 40 times faster than flash attention before on B200s and B300s.

14:13And then we're using 64 times less compute. Both of these figures at a million tokens. So this means that we can do a lot more 1 million token training. with the same compute or two million token training or whatever. And so this enables us to introduce a lot of problems that models have yet to be trained on robustly. The reality is, is we're training on a lot of data that hasn't been including in training to date and introducing capabilities that have yet to be introduced into the models to date because we have this mathematical efficiency differential that uh just makes the math of doing this training so much different just because my audience isn't all um ml engineers um so so the model you use sparse attention um and you how does the model decide what tokens to pay attention to i mean you were saying that deep seek has a separate model that does that prediction but can you go into how your models with these large context windows then can decide which which tokens to focus on Yeah.

15:38I mean, that's the highly proprietary part that we have. But I would say, I guess I could say we creatively use some early information on the inputs in a way that doesn't create all to all comparisons, which is very high level. but i i feel that going deeper than that uh leaves a lot of breadcrumbs well one thing i i wanted to ask you and we've spoken a couple times but uh does this is this the end of rag then of having a vector database that the model searches and i mean could you if you could put everything that's your RAG knowledge base into the context window, you wouldn't need that more complicated architecture.

16:41Is that right? Yeah, it is. Although I wouldn't say it's the end of RAG, just because it's too extreme. So the way it changes RAG. What we're looking to do is simplify the way that people build products with RAG or, you know, agentic workflows.

17:06Examples are today, a lot of documents are chunks that don't necessarily need to be chunked. The chunk sizes are very small. Every 400 tokens is like super common. And they're like, what if you could have a multi-page, multi-section chunk instead? There would be a lot more contiguous context that would prevent misunderstandings. Top K of 10 for the top 10 search results is very common. Well, what if you could pull the top 50? What if instead of doing one type of search, you actually did like 20 parallel searches and dumped all of those search results into the context window? These are ways in which I think we make RAG a lot more, we can enable people to make RAG a lot more robust.

17:56If you have 400 million tokens, there still needs to be some way for the model to process that data. But we're saying don't compress everything into 100 ,000 tokens or less. You can now afford to because of cost and latency. And then also the intelligence and absolute budget of context. You can now pull a lot more context and do a lot more work with every single step in your agent workflow. Like I mentioned before, if we're talking about coding problems, Opus 4.6 required an average of 66 steps to tackle a few bunch of pro problems. And if we assume that we only needed like the last 10%, then we could have gotten away with like six or seven steps if we had been able to just put all the code in the Conics window.

18:48Or even if we took like three or four steps to figure out which parts of the code base are more important, being more aggressive with the context in each step. I mean, that is a much faster, much more efficient process that actually uses fewer tokens in aggregate. So it's more token efficient. And then it's not going to miss the details. We do hear complaints about coding agents missing details and definitely on the longer horizon. So not the end of RAG, but the transformation of RAG as we know it. And you talked about doing multi-million token pre-training rather than just post-training. Can you walk us through how that works and how that changes the model's behavior and practice?

19:47um yeah so pre-training is the stage of training where um you give the model its core knowledge and to some extent some core capabilities um but it's very knowledge focused and so the pre-training simply means you predict the next token over and over again and um and then the model gets penalized based off of how often that next token matches the actual next token in a corpus so it's crude and rudimentary but it's also highly scalable and it's largely seen as a requirement to extract additional capabilities um in the in the post training phase and so um research has largely suggested that your ability to create value in the post-training phase is limited by the pre-training that you've done.

20:44So if you haven't, for example, done a lot of pre-training on code, then you're going to struggle to post-train a model to be really good at code. And so that's why it's important that we've done this multi-million token pre-training, because it means that when we get to post-training, there's already a certain amount of that knowledge there there's a good foundation to build on and in post training we can extrapolate even further and in fact this is something we showed in our technical report for a sub q 1.1 small model um we actually have a model variant where we trained the model we pre-trained the model with 1 million token inputs up to 1 million um but a lot of 1 million token pre-training and we show that that 1 million token pre-training made it so that um when we did post training at largely also only up to 1 million tokens, that the model is able to extrapolate how to do retrieval style problems up to 12 million tokens without ever seeing anything in post training over a million tokens.

21:43So that's the power of pre-training. It enables the model to extrapolate much further than if it's seeing stuff for the first time in post training. Yeah. a lot of people right now particularly with mythos and and fable out now from anthropic and glm 5.2 from yeah jirpu or zai yeah many people worry about large context models making it easier to find vulnerabilities in code bases. Is how do you view that? And and. In in both offensive and defensive cybersecurity use cases. Yeah. I do think that long context windows, long context reasoning capability does help with finding vulnerabilities in code bases, but that's different than finding vulnerabilities in products.

22:48So finding vulnerabilities in code bases enables you to be more defensive. Finding vulnerabilities in products puts your product at risk. So they're two very different things. In our case, if you already have all the code in the code base, being able to reason over a lot more of it at once enables you to find vulnerabilities much faster and also find vulnerabilities you would not have been able to find otherwise. And so we see that as something that's really valuable for people being able to make their products safer and more robust. That is, some of our design partnerships have leaned in that direction.

23:30um i don't think that long context reasoning specifically provides nearly as much an advantage um for being able to find the holes in a product from the outside um because they're more about iterative tests um the context requirement is not necessarily significant um but the way this works is software is built layer upon layer of abstraction, of abstraction. And so, you know, there are probably like 20 layers of different code components underneath the code that you're writing at any given time. And so if I'm trying to hack your product, I'm trying to guess which open... I mean, there are other things too, but this is a big part of it.

24:26I'm trying to guess which open source software you've built on top of. And these open source softwares have known vulnerabilities. And so I'm trying to, I'm testing to see if any of those known vulnerabilities exist in your software. So it's not really a long context reasoning problem. To be honest, I think more than anything, it's actually a knowledge problem. it's it's an existing knowledge of all the the existing vulnerabilities and that's why you've actually seen that people can can get open weight models to to have mythos level capabilities for for hacking because uh the model needs to specifically know to be honest sometimes uh like the number of vulnerabilities that that lead to most of the threat it's lesser than some people realize um because everybody builds on a lot of the same software components so yeah but couldn't this uh i mean to find those well you said they're known vulnerabilities in open source uh code bases but with mythos or or uh you know or i mean with your model that that can handle massive contexts.

25:45Couldn't you then just all the various foundations that are managing open source code bases, they could just dump it all into your model and discover vulnerabilities that maybe they weren't aware of, patch them? I mean, isn't it advantageous for the open source community to have solved the context, compute complexity problem so they can scan large code bases and fix vulnerabilities? Yeah, I think that's a great point. I think partnering with open source communities to find and fix vulnerabilities faster and find it and fix the vulnerabilities have just been there for a long time where speed isn't the issue.

26:39It's the breadth of perspective that's the issue. That'd be a great use case that I'd love to work with folks on. Yeah, you've mentioned design partners a couple of times. what are the most compelling early design partner use cases you're seeing for a large context? Is it finance, code, internal documents? So unstructured document processing and analysis is like the number one right now, and the number two is knowledge-based products. within those two categories finance has emerged as a really important one just because finance companies working in the finance space are building such data intensive applications they're processing so much data that's kind of like the core criteria for whether a design partner early access user is a good fit for us right now is how data intensive is their application So finance is a really big one for us.

27:45We have had a lot of interest on the coding side. We are intentionally moving a little bit more slowly on the coding side because we've kind of realized like how much coding is like a budget game. Like how much data budget do you have? How much label data can you train on? and so uh that's an area that we're really interested in and are working on but with a bit of a longer time horizon and then um and then it's beyond that it's it's like general enterprise document uh processing for knowledge bases and unstructured inside extraction um those are the main use cases that we're seeing. One thing that's great for us is that I think unstructured document processing and knowledge-based products are in pretty much every enterprise out there.

28:42And we're also seeing that because we can work with data that is less structured and we can process more data at once, it becomes a lot easier for an enterprise to get their first value from data. So we have all of these enterprises that are sitting on massive amounts of data, and a lot that data has yet to be put to use in an ai product um or maybe it has but in in a very limited fashion and they're getting pushed to do like a 10 10 million dollar data transformation project before they could even start to build a product on top and a value prop that we bring to the table is that is no longer necessary at least not at that level like we can help you um either work with the data with a lot less structure or create the structure a lot faster because of our ability to process a large amount of data at once with a lot less curation.

29:42That's interesting. And as we move into agents and particularly coding agents increasingly, which is happening, uh most coding agents use very surgical context management i mean they they hop from file to file or uh section to section uh what do you think breaks first when we move to agents that can read everything at once yeah i think that will a couple things will happen um one is we will have systems that can generalize better because every time you add a search engine or you know vector database which is a search engine um or some conditional logic to route between the steps um this human curation really limits the ability of that system to do a lot of different things it makes it so that it works within exactly the scope that you've set it out to to be able to to to achieve and um and uh and the more complicated you make it the less the more narrow that's going to be so i think we'll see agents that can do a lot more um we will also see agents that are a lot faster because you know if you're doing 10 to 60 steps um it's pretty slow so uh i always say agents that are cheaper um we want you know a million tokens to feel like 50 000 tokens in terms of intelligence cost latency etc over time and um and so i think we'll see people build a lot more too, to be honest.

31:36Context engineering step is where the vast majority of human and capital investment is today. And so if we can get rid of that, then we'll see people, like the barrier to entry goes down, people build a lot more things. And when a single step can do 10 steps worth of work what are those 10 to 60 step workflows going to be able to achieve now i mean um it'll be a lot more powerful a lot longer horizon um i guess that's that's another point um we really care about long horizon agents and um the ability to manage context not just through a two-hour session, but across, you know, weeks or months of work is something that I think will be critical and will also require much longer context reasoning than we have today.

32:40Yeah. We didn't go into your background or the founding of the company, and it's interesting. Can you give us that story? Sure. Yeah. So I've been in the AI space for the last 10 years. Most recently, I was head of Generative AI, a mid-market and enterprise AI consulting firm called Tribe AI. I did over 40 projects with a series of mid-market enterprise companies with a medium-sized group of folks underneath me. And that's where I got a really good horizontal view of what generative AI implementation was looking like in 2023 and 2024. And that's kind of what convinced me that it was overdue to move to another model paradigm, that they were bottlenecks in the enterprise implementation layer that were going to be painful for a very long time unless we change the actual mathematics behind the foundation models themselves.

33:46Before that, I was at Instagram working on creator monetization products. Before that, I was at Stitch Fix where I built the first text-based recommendation system. Before that, I was at a research lab where I published some research and built our first production products for our sister companies, Glassdoor and Indeed. Before that, I was at academia.edu where I built some of our first AI products for our 200 million user base. Before that, I did research with the World Bank of Blue Cross Blue Shield. And so I've done a lot of different things. I'm not a domain expert. In fact, that's one of the things I pride myself in is I like to look at things from a very horizontal perspective what's the key value point that's missing across everything everybody's doing and that's really what we're trying to tackle as a company it's like how do we take computational memory and sample efficiency and turn that into downstream business value propositions um context is the first one but there will be many more yeah well and you mentioned memory are you working on memory absolutely so i mean we're working on memory within the context of ssa subquadratics parts attention the thing we talked about publicly we're also working on non-attention algorithms have been for a while um that uh make it totally necessary to have any form of kvcash kvcash is this massive thing right now where at the multi-million token range uh can easily use a lot more memory than the model it's the model weights themselves um becomes very hard to host the context without distributing across multiple cloud data center grade gpus makes it impossible in my view to do long-range modeling for robotics on jets and orans which have about as much memory as your phone and so we see this as a critical problem to solve for long context modeling, for general robotics intelligence, and just the ability to create much better frontier models than exists today.

36:00This long context, you mentioned that most enterprises don't even fully use 256K context. as you introduce them to multi-million token windows what what are you learning about user preferences and behaviors yeah that i mean that they're not well mapped out and they can vary um so we found for example people will tell us um that uh on a news article they'll ask what are the top three insights from here and the model will give three top insights that are technically accurate but they're not the three they cared about um so like this is the level of alignment that becomes necessary uh with long context reasoning gets very complicated so um one of the things we learned is that we are so far from from this area of modeling being mature.

37:09I think people say, well, the frontier models have a million token conics window, but like their ability to deliver what people want at that size is just completely, we are far removed from that today. And, and promptability is still a bit lacking also for these types of problems. And I think to some extent, like this problem can't be solved at the prompt level because it's just so complex and nuanced. We do actually need to solve this through training first, where we see a very wide number of problems that, a very wide number of inputs and outputs or problems through RL that show what user preferences are at scale before this becomes a problem that we can really solve through prompting.

38:01um so that's a super high level insight um beyond that um i think reasoning is pretty unsolved oh here's here's an interesting one actually um we did some we created some evals for code base question answering and financial document question answering um and we ran we did them at 500 000 tokens um a million tokens per opus 4.7 which is like 800 000 tokens for us um and uh and then as close as possible to two million tokens which obviously only we could do and so we found on code-based question answering um um the uh there there were like different levels of difficulties like we could say easy medium and hard and um on the easy and medium problems like the models most frontier models and our model could do pretty well pretty close to 100 percent and uh and there was really not much of a concern there on performance but it is quite easy to create an eval set that on financial document analysis that is well below 50 percent um at 500 ,000 or 800 ,000 tokens and um so it does seem like obviously there's a bias towards code I think that um the model companies have spent like more time in some areas than others and then also I think there are certain types of contexts that harder than others like i've been working with financial documents uh for a long time i've spent a lot of time working with edgar data um and there are a lot of things that make it hard um some of them are intentional but um but the structure of the pdfs can be very challenging for llms to handle um data can be really distributed across the several hundreds of up to several hundreds of pages.

40:12You've got tab viewer and non-tab viewer data. You've got tables that are really hard to parse correctly. All these challenges that I think make these problems harder than code in some ways. So I guess the TLDR here is that long reasoning capability today is very asymmetric across verticals. Yeah. Yeah. On the, you mentioned that you give these models, the standard transformer models, a large context, a large chunk of text in the context window, and ask it to surface the most significant things and it doesn't come up with what you think is most significant. How does this, is the reasoning stronger with your architecture or is it that it sees more of the context?