Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut - #769

0:00Sam Charrington:As context windows get larger and larger, one question that keeps coming up is whether retrieval augmented generation or RAG is becoming obsolete. If models can ingest millions of tokens of context and reason over enormous collections of documents, why bother with retrieval at all? The answer, it turns out, depends a lot on the application. I recently sat down with Alex Bowcut, head of engineering at Sphere, which builds AI systems for sales tax, automation, and compliance. exactly the kind of domain where getting the right answer isn't enough. You also need to know where it came from. I asked them this simple question.

0:33Sam Charrington:What's your take on the whole rag is dead argument that some folks make? I think for some use cases, it's certainly true. I think for us, or at least for this particular problem, because we are so sensitive to accuracy and we're so sensitive to the exact right citation, As of today, I don't think agents are just searching over the file system, grepping over it is at a point where we could switch over and not lose accuracy. I'm Sam Charrington, and this is the TwiML AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters.

1:15Sam Charrington:Let's jump in.

1:24A little bit about Sphere briefly, so this makes a little more sense. Sphere is a revenue-based compliance company. So we help companies with all of their revenue-based compliance needs. The main one of those is sales tax in the U.S. and internationally. That's called VATGST. And the way, you know, there's other companies in this space, of course. There's some big companies that have been around for quite a while.

1:51Sam Charrington:Tax is not a new problem. It's not, unfortunately, for companies and for consumers, I suppose. So this isn't a new problem. The incumbents, they face a particular problem, which is in order to support, you know, every jurisdiction in the U.S. because, of course, every U.S. state has different rules. In some U.S. states, even the cities have different rules. And then internationally, of course, every country and potentially province has their own rules as well. And so the incumbents need a way to understand how are products taxed in each of these different jurisdictions. And the way that traditionally they've done these sorts of things is they've hired these massive teams of essentially tax lawyers.

2:39They're tax experts. They'll call them tax content teams. But what these tax lawyers are doing essentially is looking through the legislation in Alabama, for example, and understanding how does Alabama tax SAS? And even more specifically than that, how does Alabama tax SAS that maybe has an API connection and has servers that are hosted within the state itself?

3:04Sam Charrington:So it gets very granular there. And this is a huge, you know, this takes a huge amount of human time to do. And you all, there's also like the moving target of it, which is, of course, like legislation updates that can happen at any moment. And so you have to constantly be updating and looking through the legislation again to see if anything has changed and updating your tax engine, essentially, to make sure that you're applying the correct treatment in all of the jurisdictions. And so that's been a huge inhibitor to growth for the incumbents and the reason why most of the big incumbents have stayed in the U.S.

3:41because, you know, they've kind of tackled this problem in the U.S. and to extend it internationally, you know, it's just like too much of a Herculean task for them to, you know, it's too much manpower, too big of teams to handle it. And Sphere has taken a very different approach. I think the time that Sphere was started as a company was obviously advantageous. We were started during the AI era. And so this is a very classic document-based problem. It just from a high level, you have legislation and court rulings and bulletins from departments of revenue. These are all just documents. And these inform, you know, the answers of how products are taxed are found in these documents.

4:24And it's just a matter of finding the relevant passages and understanding the relevant passages and then assigning a taxability. Right. How is this product taxed? And so in this new era that we're in, we looked at that problem and it was a problem that we thought was screaming to be solved by AI. So what we eventually built is what we call TRAM, which is the tax review and assessment model, which is like a system of a few different things that I'm sure we'll get into. But essentially, its job is to supercharge our tax experts. So what we found is that TRAM allows our internal tax experts to move almost two orders of magnitude faster through this process with less errors than the traditional just fully human focused approach.

5:14Sam Charrington:And did you work in tax prior to joining Sphere? I did not. So I've learned a lot about tax. And I think I was always like, you know, I would read, you know, U.S. Supreme Court rulings kind of just for fun, out of interest of kind of what's going on and how does the legal world work. But no, I didn't. I didn't come from like a traditional, you know, PWCE tax background or anything like that. What was your background? Yeah, just a pure engineering background, mostly in startups. So I started my career before moving into startups. I started in the semiconductor industry and then working on GPUs, writing CUDA kernels, and then eventually left to start a startup with a friend, which was like a web data collection startup.

6:01and we worked with like investment banks and private equity firms and things like this to help them collect really massive amounts of web data at scale to power their internal like analyses and signals. And eventually that company got acquired by Bain & Company. And so I was at Bain for a bit and then started another startup and eventually Nick reached out. He was the founder of Sphere, Nick Rudder. Nick reached out about Sphere.

6:32Sam Charrington:And at the time it was just Nick in a little one-person office here in San Francisco. And, you know, looking back, maybe a little inadvisable, but he convinced me it was a good idea to leave the other place I was at and come and join him at Sphere and kind of chase this dream that was, he had like this nascent idea of what would become T-RAM. And yeah, things have gone quite well. We did our Series A last year from Andreessen Horowitz. and continue to grow at kind of a remarkable level. Talk a little bit about the data landscape that you have to deal with. I'm imagining significant amount of complexity due to the global nature of it.

7:14Sam Charrington:And, you know, whenever I've talked to folks that are doing a collection of legal data, it always surprises me how much of that stuff is in non-friendly formats, like, you know, photo, like image-based PDFs or, you know, I've talked to folks, it's been a little while, but, you know, they had to go send people to go scan stuff. Like, how crazy is it still? Yeah, it's not great. It's not great. And a lot of what we do is working with government systems, obviously, which are, you know, they can be archaic. There's some that are better than others, but yeah, there's a lot that are quite old. so some you know sometimes things are well structured they're you know html pages that we can go and collect the legislation and that's great sometimes they're pdfs and those but they're well structured pdfs and so we can we can parse those easily and so it's always great when the sources are like that and then yeah there's this long tail of you know pdfs but they're just images right so you need to OCR them and use other techniques like that or even spreadsheets or text documents or word documents these things are much more common than than I think I had when I started working on on on the issue and so yeah the beginning of our pipeline like this data collection process accounts for each of these different file formats but yeah not not the funnest problem to solve and and you know trying to especially you know like spreadsheets and things like this quite difficult to try and like pull good information and like retain context when things are stored in a spreadsheet let's take a step back and dig into how your users use t-ram you mentioned that like you, you know, two orders of magnitude near improvement in their process.

9:14Sam Charrington:What is that process and how are they using the system? Yeah. So our end users. So if you look at our website, folks like Lovable or Replit who use Sphere, who are customers of Sphere, they're not direct users of T-RAM. t-ram is an internal tool that allows sphere that has allowed sphere to expand globally even though we're we're a small company we're just a startup and then also have higher accuracy and so um the the way that it t-ram is used is is by our tax experts so there's a web app that the tax experts use where they go in and essentially just review the work that t-ram is done. So there'll be a queue of work that needs review.

9:59So for example, maybe they need to review, you know, T-RAM has done what we call determinations, which is just, you know, this determining of whether a product is taxable or not, and some other features around there in a particular jurisdiction. So the tax expert will go in and say, oh, I see that California digital goods needs to be reviewed. And so there's a list of different types of digital goods and the model's output on whether it's taxable or not, some reasoning the model gives on why it came to that conclusion, and then also importantly, the citations that the model used in order to inform its decision there as well.

10:39And so the tax experts are able to look through really quickly. And sometimes they do have to make adjustments. The model isn't 100 % accurate, but essentially they review the model's outputs. They can lead feedback, things like that. And eventually they click submit. And the submission of that then puts those into our deterministic tax engine. So we have a tax engine that, for example, we integrate with Stripe. We're a first-party integration with Stripe. So our customers, when you're checking out and buying something on the checkout page, we'll be calculating tax. And that part of it is deterministic.

11:15There's no AI there. The AI has been done upstream from there.

11:19Sam Charrington:So when these tax experts are, you know, working on a piece of work, like what's the impetus for that? Is it, you know, they are, you know, solving a problem for a customer and that like, you know, drives work or is the work all driven from, you know, some new, you know, ingested piece of data from a jurisdiction that the system says, oh, this might have a change in, you know, know the deterministic engine so it would be twofold so one thing uh that would be like the impetus for them to go in would be um we're we're expanding uh products that we want to cover so every tax engine has done this uh you you are you can't support every product type from day one right you kind of have to break down and you know we're going to start with with clothing or we're going to start with SaaS.

12:15Sphere specifically, we started with like electronic services. So anything that's not tangible, essentially. So they would go through, we have this backlog, essentially, we're like, hey, we want to add support for all of intangible goods, which we've already done. Once we've done that, now we're like, okay, we want to add support for tangible goods. So we'll move through clothing and servers and things like this. So there's like this backlog And that's kind of driven by customer demand, right? If we want to sell to a clothing provider, of course, we need to support clothing. Then the other thing would be updates in the law.

12:51So when legislation is changed or new bulletins are posted or new case law becomes posted, then we'll scrape that data. It goes through our ingestion process, eventually ends up through the system with, if there's some action that needs to be taken, we'll make that recommendation to our team. tax experts. So an example might be last year, Maryland, or maybe a better example, Manitoba in Canada, they changed, they began to tax SAS at the beginning of 2026. That was flagged very early in our system. And so that was reviewed and essentially pushed into the deterministic tax engine with a start date of January 1st so that we were prepared well in advance.

13:40And that's one of the nice things about having, you know, something like T-RAM, this automated system is a lot of the time, traditionally, these things are done retroactively because, you know, you miss the update and then, you know, you scramble to try and add it, but it's already too late. You know, it's past January 1st or whatever. So, yeah, those would be the two different ways.

14:00Sam Charrington:Got it. I remember getting a flurry of emails maybe, I don't know, nine months, a year ago from like every SaaS vendor I use talking about some big change in the way taxes were going to be calculated. Yeah, I mean, it's especially on the SaaS front, you know, over the last four or five, six years, U.S. states and international jurisdictions are just getting more, you know, they want to they want a piece of the cake, right? They want to share the pie. And so they're changing rules to tax SAS, and they're also changing lots of other rules. There are certain international countries that require real-time reporting to the Department of Revenue or the tax authorities.

14:38So as you're transacting, they want a copy of it. And I think it's kind of like tax rates. Tax rates only go up. They very infrequently will California reduce their sales tax rate. And I think how involved the tax authorities want to be in transactions and how much information they want, that also only increases. It's not going to decrease. And so that's something sphere, I think, is kind of at the forefront as well, or these other features outside of just sales tax that are only becoming more common. And a big part of that is SaaS, kind of like you mentioned.

15:13Sam Charrington:So these human experts, in some ways, what it sounds like they're doing is data labeling. Did you think of it like that? We think of it more as like a legal review. So in some ways, I think what we do is somewhat similar to someone like Harvey. So Harvey AI, that's like a legal AI. And you'll let Harvey draft like a first version of a brief maybe. But then a lawyer or someone at the firm will go in and they'll review that brief and they'll check it for correctness and things like that. They'll do a legal review. And I think that's how we view what our tax experts are doing. It's not necessarily data labeling.

15:52It's a review of correctness from a legal perspective, because these are legal claims essentially that we're making. You know, we are claiming that, you know, SAS is not taxable in Alabama or whatever. So, yeah, I think it's more akin to a legal review. And so talk a little bit about the pipeline in more detail. You ingest this information from lots of different jurisdictions, presumably normalize it in some kind of way, or at least try to extract the information out of those image PDFs.

16:28Sam Charrington:What happens next? Yeah, once we have kind of, you know, the text from the document, and ideally that's well structured and like HTML and some PDFs will give that to you. So we try and preserve as much structure as possible. The next step would be for non-English legislation, we'll do an English translation. So it's been another big unlock is that, you know, we don't need tax experts that speak every language. You know, LLM's great, you know, notoriously great translators. And so they're happy to translate these documents. So we'll create an English translation. And that's kind of the starting point.

17:08From there, you know, we can't, many of these documents are very long. So we can't just like take the whole document and, you know, create an embedding for it and store that in a vector database or even necessarily with like a more with like a TF IDF, like full text search database. You might not want to do that either. So what we do is we break up into sections, smaller sections, and there's a naive way to do that, which is just like every end characters you chop and then you create a new section. And that's obviously not ideal because you lose very relevant context. And again, these are legal documents.

17:47So they're well-structured typically, as long as it's not an image. And so they come in sections and then subsections and bullets. And so what we try and do is our pipeline semantically chunk things into essentially sensible chunks that cut at normal places. And then we still retain the hierarchy of where that chunk came from so that we can reproduce it later. And then we also store like metadata, of course, and things about where this document came from, like the root document. And then eventually we have these text chunks, and we embed those both dense and sparse, and we store them in a vector database.

18:29And that's eventually what we'll then query over when we go to actually make a determination. But I think it is, you know, we don't, I could probably talk for the next 60 minutes about kind of this process of chunking. I think we spent a lot of time there and it's a very important part of this process. I think if you do a naive implementation, you leave a lot of accuracy on the table, essentially.

18:54Sam Charrington:Yeah, I wouldn't mind having you dig into some of the work that you've done to kind of assess the lift on the semantic chunking. And what you've seen there, I think, you know, as you alluded to, like a lot of folks will pull a rag library off the shelf and it'll give you like three or four ways to, you know, chunk number of characters and whatnot. And, you know, folks will do one of that, but there's often ways to take advantage of the inherent structure and the information that you're trying to capture. Yeah. How did you approach that? Was it just obvious that, you know, hey, we're going to do this based on sections because it's a legal doc and they're right there?

19:36Sam Charrington:Or did you like iterate on that for a while? I think it was obvious that like, you know, when you look at one of these documents as a human, it's very obvious how like if you were going to break it up, how you would like to break it up. And so it was clear, I kind of, I guess, what should happen. The question is, how do you make that happen in a way, you know, doing it for one document is easy. doing it for you know the millions of different documents that we've pulled in a way that's general generalizable is much more difficult and so I guess the details there go into you know as as we ingest these documents we have a number of different buckets you could call them of different structures of legal documents that we have parsers and a lot of these are LLM backed parsers, but like bespoke parsers that for that particular type of document.

20:29So we will either from like the metadata of the document or through an LLM call determine, hey, what is this kind of document? Is it a case law? Because a ruling from a judge will look different than like the legislation, the statute law, which will look different from like a bulletin or like a notice that the Department of Revenue releases. And so we have bespoke parsers for each of those. And some of those, yeah, they involve like LLM tool calls. Some of them are fully just algorithmic because the structure is all there and it works fine enough. But I think that's where kind of the devil is in the details on it was, yeah, as a human, you look at it, you know exactly what to do on each of these different examples, but how do you do it in a way, yeah, where it's generalizable across all of, you know, across languages, across jurisdictions.

21:20Sam Charrington:Can you dig into a little bit more detail on the dense versus sparse aspect of what you're doing? We started with just a dense representation, which felt correct at the time. And if we think about the query that we'll eventually run, you know, if we're looking for relevant passages about SaaS, you know every jurisdiction has a different you know ignoring even different languages of course but like every jurisdiction even in English might have a slightly different way that they describe SAS right and especially in legislation legislation reads very old like their description of SAS will be very antiquated it'll it'll might even talk about like CDs and things of this nature and so From the beginning, I think it was a fair assumption.

22:13My opinion was that we should be using a dense embedding, right, that semantically embed these passages. And so that is what we started with. I think what we found and when we brought sparse back into it was there are times, especially with certain when it comes to citations and pulling out certain terms from passages that come from the dense embeddings where you also want to search sparse, where you want to do, you know, a full text search of certain words and certain terms and pull those in as well so that you can. and then compare the two of them. And what we saw was a pretty good increase in accuracy on the citation side.

23:00So we have some evals that we run on the retrieval part that we have a baseline of like these citations, right? These passages are the ones that should be retrieved for these queries. And as we kind of layered sparse back into that, we saw another, we saw an increase in accuracy. and so that's kind of what we stuck with.

23:22Sam Charrington:When you refer to dense and sparse, it sounds like you're talking about embeddings versus full-text search as opposed to like two tiers of embeddings or something like that. Yeah, that's right. So yeah, dense is definitely embeddings, like yeah, semantic embeddings that we use OpenAI's embedding models for and then yeah, when I say sparse, I'm referring to, in our case, we use pinecone to essentially create a sparse representation. So we've loaded a vocabulary and then each passage is fed through and it keeps an index like full text search of the different terms and their different usages across passages.

24:05So it's not quite, you know, elastic search or Apache Lucene, but it's sparse in like a TF-IDF type implementation.

24:15Sam Charrington:Got it. So you've got a predefined vocabulary. And as you pass these documents in the pine cone, it's just flagging which documents talk about which of these terms. Yeah, which passages are talking about which terms. So then you can search. Yeah. So then you can search over them and find, you know, if this one uses this particular term that's not frequently used across the corpus, it'll be a high result. And where are the search terms coming from? Like, do you, you know, is it, I'm kind of getting ahead of your answer, but I'm imagining like document comes in and the first pass is to see if it's at all relevant to the task at hand.

24:59Sam Charrington:And so you're just searching for a bunch of terms to screen the document. Is that the idea? Not quite, but I think, yeah, it's a great question. So the query comes from something slightly upstream, which we also use tRAM for. And that is kind of the first step for us to support a particular product is for us to create what we call a taxonomy of that product. So what that means is we create kind of like a tree structure of for this particular product type, what are the different characteristics across the world that affect its taxability? So an example might be for clothing. Clothing that is made for children versus made for adults can have different taxabilities.

25:52and you can imagine more questions like this where you know maybe pants have different taxability than shirts something like that and we build this big tree and what the query that eventually gets fed into t-ram is essentially a well it's a couple of things but one thing is a description of that particular type of product so in our clothing example maybe it's adult pants. And so we have a description of adult pants. And so that is the main query that we put into the system to then pull out relevant passages. And we'll use filtering, of course. We're doing determinations for Florida. We'll filter to only the passages that come from Florida's