How Specialized Models Drive Developer Productivity | Tabnine’s Brandon Jung

0:00Even if the technology is there, I think the bigger question is trust. right and trust comes from transparency full stop and i continue to see a dramatic lack of transparency across the board with the way uh many companies handle what data goes into those models and we see that again and again with ctos at open ai not even knowing what into the model so at the point we can't say what goes into it is not going to engender people comfortable in putting more trust into more and more important things into these models. I think it's just critical that we push for transparency and trust as it is in, I don't know, things like government and organizations.

0:41Like these are not new principles and there are ones that will be true no matter where you apply them. 13 % of all poll requests are bot created today and they're creating a unique impact on your SDLC. Linear B's upcoming research exposes the effects bots are having on your team's developer experience and productivity and engineering orgs who created a system for managing bot-generated PRs are able to reduce their entire review load by over 6%, while also making drastic improvements in their security and compliance posture. You want to learn how your team can manage bot-generated PRs and get early access to Linear B's data report?

1:14Head to the show notes to register for our upcoming workshop on September 24th or 25th. Hey, everybody. Welcome back to Dev Interrupted. I'm your host, Conor Bronson, and today I'm delighted to be joined by Brandon Young, Vice President of Ecosystem at Tab9. Welcome to the show, Brandon. Connor, thanks so much for having us and look forward to it. Yeah, it's great to have you on the show. This year, we've been getting different perspectives from engineering leaders like you on AI. Is AGI going to take over? Are our specialized models the way to go? And with more than 1 million users leveraging Tab9 for AI-assisted coding, It seems, Brandon, that you're firmly on the side of specialized models.

1:56Why is that? Well, I think there's a whole bunch of stuff that plays out into it. I think we'll maybe get a few of it. I think, first off, knowing what's in the model is super important. This is, while we are in generative AI, it's really the key piece is AI. So we've always known that AI is good data in, good data out, bad data in, bad data out. So from that aspect, that's not really changed just because it's generative. You put generative instead of AI. Those basic principles still apply. So I think the data is going to continue to be a primary reason. And I think clearly cost is going to be something over time that as people learn and use these in different facets, areas that are much more specialized, the very, very large models become less and less useful.

2:46And so I think both of those are going to play into both the knowledge of the data and the cost of running the models will be two that switch it towards specialized models, small specialized models versus very large. Let's drill into both of those, starting with that data transparency and data accuracy challenge that you mentioned. What problems do LLMs have when it comes to data integrity and transparency? Sure. So LLMs, first off, they just want lots of data. And that's just fundamentally the way that they're set up is, generally speaking, the rule of thumb is more data is better. And there's a high correlation between the size of the model and the amount of data you need to train it.

3:31And at some point, as you're hitting the extraordinarily large models we're hitting now, there's just not even data to train them. And so now we're even getting a lot of ideas around synthetic data to feed into these really large models. So there's that aspect, right? The secondary aspect is in terms of what they output. Now, a generative AI model is by definition not going to give the exact same answer every single time. And it will occasionally have hallucinations. When it says otherwise, it's not how it currently works. And if someone magically solves that problem, well, good for them. But I would not see that coming.

4:09So if that's the case, then what data goes into the model is really important to what comes out. And that varies from different places. So we've seen this from an image standpoint. What images you train on is what images you get out. And that's played out both early releases of a number of the models early on. What if it had a bias? And then they have another bias because they put a filter on it. There's a process you're always working on on that. But as far as it applies to code, I think the real questions that are coming through in our industry is questions of copyrighted data, questions of proprietary data.

4:49Is that in the model? And again, for some customers, not a problem. A good number of startups, probably not an issue. Large banks, government agencies, high security companies, probably pretty important that you know what might come out of that model and that you have some level of understanding. So as always, I guess the answer is it depends. I think there's legislation we can get into that might drive that even more towards in the importance of knowing what's in those models. But time we'll see over the next 12 months, it's going to be interesting. So I still want to talk about this cost element as well, but let's keep going on data transparency and the actual origination and accuracy of data.

5:34You mentioned legislation, which is something we don't talk a lot about on the show. What are you seeing happen on the regulatory side that could be impacting this? Sure. So it's not surprisingly, we all are now functionally think of GDPR as a standard that's required. GDPR naturally came out of the EU. And from a regulation standpoint, the EU tends to be much more forward, first to market, whatever you want to look at just tends to be. And the aspect I think most people probably didn't pick up was mid-March, there was a EU passed, what is the Artificial Intelligence Act or AI Act. And in it, there are some very important aspects that talk specifically about having, you cannot have any copyrighted data in your model.

6:25That's going to prove to be super difficult for any of the large models because every one of the large models is going to, already has that in place. So, um, I think as we go forward from that, that's first off one, we're just going to see, uh, driven. And I think Europe's going to drive that, uh, for right or wrong, that's going to show up. And then we're going to see that in kind of 20, that's, I think required that will be implemented in 2025. So Europe is going to have a different stance. And usually that's kind of plays downstream into the, uh, us and the rest of the world problem. It makes total sense that Europe is driving this, as you pointed out, typically do around these data and privacy regulations.

7:08Do you see specialized models then having a large advantage around tracking data origination, copyright versus these LLMs? yeah absolutely i mean it's it's an aspect that we we started at like tab nine we have a model we offer a bunch of models so if you want to use a very large model and you're okay with that right go for it you can use uh anything from the gpt stuff to the clod to to cohere to llama to gemini but all of those we don't know what's in there by definition they're black boxes and then And Tab9 early on said, hey, based on automatic customers, we've got a model that is based on only fully permissive open source and that you can audit, right?

7:53So that you can see everything that was in it. That's a non-trivial effort. And I think, to be honest with you, I think we've just been all been lazy around the data. If we want to be specific on the data, we've all been lazy in doing what we've known we need to do for data quality for a decade, two decades. this has always been the case but I think what happened was the return for having good data, so said another way having really good data is yes important, but if you're not using it, people are like, okay we won't solve it, but when all of this feeds into a generative AI model all of a sudden we've kind of found the magic application that I think is going to force a lot of that data quality and cleanliness to really amp so oddly it's kind of the pull mechanism to get us all much better about our data quality makes sense there's so much more visibility on it now every time this hallucination people are seeing it and they want to understand why it's happening they want to understand oh you know how did this get in here correct and if we can track that back or understand data sourcing better and origination it can both improve transparency around where models can improve while also dealing with some of these regulatory concerns that I think a lot of these models are going to be impacted by, as you point out.

9:16I might just double-click. I don't think it's a hallucination per se because everything will hallucinate. So if you hallucinate, you're really creating something that doesn't so much exist. The bigger issue is going to be large chunks, in this case for us, of code that is coming from a copyrighted source, right? It's good. I just want to validate just because everyone's like, hey, you're going to have hallucinations. There's other ways. It's an important clarification. clear inaccuracies or actually having copyright issues, to your point. And it brings up another thing, which is these other trade-offs that are being experienced by LLMs.

9:52When they take this large, try to get all of the use cases into a single model, push for more and more training data, there are risks that come with that. And you mentioned cost as one of them. Can you expand on that a bit. Sure. So the size of the model, and particularly if you start going into models that are say multimodal. So to be clear, multimodal is the notion of like, I can do coding and then I can do pictures and I can do a lot of other things. And that size of model that you're pushing that through is enormous. What you found if you get to that size of model, the very large models is the first concern I think we should just all ask is there's a huge amount of money going into training this, both acquiring the data and building the models.

10:40Alignment of incentives or just understand incentives is the first question. So why would these very large companies be building these very large models? And the short answer is super straightforward. It runs on their cloud infrastructure, right? And you can see this very clearly from the investments going into whether it's all the OpenAI and Microsoft's investment there, the multiple investments into Anthropic and into Cohere, Google's having, of course, Gemini. The goal on these, of course, is to drive underlying consumption. That's the motivation, or I should say, it's why they're so interested in it.

11:18I'm not saying they would do it just because of that, but it drives a lot of consumption. And so the natural question then is, who has control of those and where can you take them and use them? what goes into them, obviously big question, and then we haven't seen the cost over, so an API call right now probably looks to be dramatically subsidized by all the providers in the gold rush of trying to get developers and others to use their APIs, and we've been down this path Uber, Lyft we've seen how this works when a huge amount of VC and capital goes into building something, it's a gold rush to figure out who's who's models get used.

12:02And then once they get used, but naturally you have to pay for this. And these are continually updated. They are not a one and done option. So I think we just have to be, you know, real clear that those very large models, at some point, someone will have to pay for them. And that either comes in one format, either you're putting a bunch of consumption onto the cloud, and it's a lock into the cloud of whichever was you choose their model, or it becomes something that you have to pay that cost for. So hence, those big models, eventually there's a cost. Eventually it has to be paid for. And I think that's when, as that becomes something that people are saying, okay, well now you got to pay for it or now it has a cost associated with it.

12:44I think then you're going to start seeing people moving towards a specialized models. Specialized models are better. I think the other aspect that we're seeing why this is, you know, the big models are selected is when I don't know what I'm going to use it for and I haven't actually defined my use cases. So in the initial gold rush of, I want to use it for, you know, today I would like to write Shakespeare in the style of Cardi B, or maybe it's Cardi B in the style of Shakespeare, whichever you want. And then I want to go write some really good Python code. These are two very different worlds.

13:16and either you have to create models and we have to get better about selecting models for use cases, which you're starting to see. But generally speaking, people are still selecting between what big model do I ever use, not what focused use case I want. And I think, you know, clearly the place that we're already seeing this is a hugging face, right? You're seeing hugging face get phenomenal amount of traction. That's because that's where this will shift towards is individual models for individual use cases. I think you'll see some proponents push back and say, well, this arms race is all because we want to be the first ones to get to AGI and artificial general intelligence is going to solve this problem with LLMs.

13:58There won't be these tradeoffs. That's what we really need here. It's not about creating specialized models right now. The AGI will do that for us. What do you say to those folks? The way these function, I think we're just a long ways. We're very long ways off AGI. I'm aware that the initial interaction with a genre of AI has a tendency to feel like you are interacting with a human. So it does have some of those aspects to it. So first off, I just think we're a very long ways off from actually getting to that case. I also think it's also a question of, it's reasonable to say that the first person, the first one to AGI, it's a gold rush and whoever gets there is a big one.

14:38And let's be sort of super clear also, the first person to even get generative AI out at scale, OpenAI, has clearly grabbed the lion's share of that work. So this is not, I see the alignment and the view that that's useful. It doesn't suggest to me the way these operate, unless we get a stepwise function in something dramatically different than the way that these models currently function, that at least for a lot of use cases that you shouldn't just be using a smaller generative AI model. So even at that case, the cost of running it, the bet is the cost comes way down. I'm sure the cost will come down somewhat, but we're talking many orders of magnitude.

15:23And perhaps even if the technology is there, I think the bigger question is trust, right? and trust comes from transparency, full stop. And I continue to see a dramatic lack of transparency across the board with the way many companies handle what data goes into those models. And we see that again and again with CTOs at OpenAI not even knowing what into the model. So at the point we can't say what goes into it is not going to engender people comfortable in putting more trust into more and more important things into these models. I think it's just critical that we push for transparency and trust as it is in, I don't know, things like government and organizations.

16:06Like, these are not new principles, and there are ones that will be true no matter where you apply them. Yeah, to your point, we're kind of at this max hype moment where AI has reached, it's on the scene now, people are using it regularly, it's broken containment. but we also have this massive capital expenditure happening and companies that as you pointed out our cloud providers can justify that they're like great you know yeah it cost us a bunch up front but we're going to get to run this in our cloud for years and years it's all going to be worth in the end but so many folks are kind of keeping up with the joneses here and there are going to be a lot of these models that don't necessarily work out and so it's important to start figuring out the right approach for your company on here.

16:53And as we talked about at the start of the show, I mean, Tab9 is obviously taking a very tailored approach, helping drive AI-assisted code. And I'd love to kind of dive in a bit about these tailored use cases and how software engineering and dev leaders should be thinking about leveraging specific models or tools. with, I think, I'll say debunked this broader case for the moment. You know, maybe things will change. Maybe we'll see AGI here in a couple of years and great. We can all just go use that. But for now, finding the right specialized model and understanding the use case that you have for it is really important.

17:33How do you think software engineering leaders, you know, the audience listens to the show and other developers should be thinking about that challenge of one, figuring out their use case and two, finding the right model. Bringing direct principle right back to is know your data. I mean, after hundreds and hundreds of conversations and hundreds of customers, almost every customer we talk to around the data or around the general AI for code use case is very, very interested, very quickly gets the notion of, well, I do this in a specialized way. And that could be as extreme as things like I've got COBOL or FORTRAN, like some language that you know, just the rest of the world hasn't seen.

18:12or it could be like, look, I like old crufty Java because we do it this way, but we're six versions back from Java. You know, this is how we operate. And many of the kind of use cases in between. So the first piece that everyone should be doing, should have been doing, has always been supposed to doing is like, what is your good data? Like, what are your good repositories? Think of the idea of a golden repo. This is our best practices. You cannot replicate what you have not defined. Okay? You cannot replicate what you have not defined. So these models, I can customize a model a bunch of different ways, right?

18:50It can go anywhere from on the far end fine-tuning a model. That's something that you're going to do right now is the best practice for things in languages like a Fortran or a COBOL. But you have to have quite a bit of data to do that. Now, COBOL, there's more than enough COBOL in most companies that are using it that that's actually doable. but then there's a lot of much more lightweight ways of doing that with RAG, with vector databases, with graph databases that allow for those technologies allow things like what we would kind of think of as coaching, expert guidance, such that you get a dev, the first thing they get is, oh this is how we implement and do it at our company right, and when we look at what they need, that's the Biggest difference we're going to see between a general model trained on, you know, general GitHub code and a model that is trained that is specifically for your use case or your company.

19:51And a good amount of the initial adoption has been super general front end web development, which, again, is going to look very standard across what you see on general GitHub. That's obviously the first place you're going to see it. But, you know, up until five years ago, the largest number of lines of code written every year was COBOL. There's a lot of code there that we need to either modernize, maintain, update, etc. And there is just no shortcut to knowing your data. So step one, like we said, know your data. from there then the question is hey let's then you want to build a model and or customize a model or wrap a rag around those models um all reasonable um options and then the other part that need to look at is how do you get the right suggestion to the right user at the right time in the right place and currently there's a couple different ways of thinking about this right a very large model the chat style models are very good for a question and a long answer and a response that comes in, you know, three, four seconds.

20:55We're used to it. It's like, oh, ask a question. Oh, that's fantastic. That's a good part of it. But what we've actually found when we look at the way a developer writes, they kind of write in two options. Discovery mode, I don't know what I want to write. And, you know, I'm hammering out code. I'm in the zone. If you want something to really help you down that path, it's got to be operating in, you know, 150 milliseconds of time. A large model can't do that. Now, maybe someone magically figures out how to do that, but the cost of doing a model. So let me just sort of draw this out. Most of the way people interact with models today are very large models and actually not a ton of tokens relatively going back and forth.

21:36This is what drives the cost. If you're trying to run this at very high speed, think every keystroke you add is sending out a whole new set of tokens and need them all back. So you're going at a velocity that's dramatically higher. And that needs a different, very specialized model that can operate at a much, much, much higher speed down to some things like right now we mix those models to get you the fastest suggestion at the right time and the right place. So there's a lot more nuance, at least, and this will kind of apply across other use cases, but coding in particular takes multiple models to make that sort of work based off different characteristics.

22:13All that being said, know your data, know your use case, know your risk tolerance. So if you have high risk tolerance, hey, if you have high risk tolerance, let Microsoft subsidize the heck out of what you're doing, because that's what they're doing, and go for it. I think that's a great idea. If that's alternatively, that's not something that your company's is comfortable or you need to alter it, then there's other options and we might be a place at Tap9 to fit. That's a great description. And I think it's interesting to think through both that risk factor piece and also, you know, what is your use case here?

22:47It really does depend. Are you doing front end web development or are you trying to do something that's, you know, more specialized or are you just saying, hey, I just need to write a bunch of SQL code because then maybe you can just use ChatGPT, right? Sure. It really depends what you are trying to get out of the model and how you're building your organization for the long term. And so I love that you're kind of speaking to the specialization piece and also how much tailoring there is to be done here. Because as you highlighted with, you mentioned graph and databases and vector databases and the way we have to think about actually managing our data.

23:27It's so clear that if you just look broadly in media, I think you'd get a very small sliver of what you need to do to successfully run an AI program. but as we've explored on the show this last year there are so many pieces of doing this right and it's not just finding the right tool it's you know getting your data in the level it needs to be so that you can tailor that model to where you need to be it's having the right partner in your product org that there's buy-in within the org about what you're actually going to get done with it it's knowing your use case and getting the whatever buy-in you need at the executive level to actually spend the money you need to spend things up.

24:03And then it's understanding that ROI may not be immediate. You may take time to tailor the model to what you need it to be. And you're looking at like this long-term operational benefit. And it really depends on the organization. And I think to me, like talking to folks like yourself has really convinced me that at least in the next couple of years, tailored models are where things are going because there is so much work to be to make sure that you are getting things right. And if you are a major enterprise, you can't afford to have the wrong data in there. You can't afford to have code that is being set up wrong and just pass forward.

24:43So understanding like how the workflows around this work and how you're actually passing code back and forth that's either been AI assisted or fully AI generated by agents. There's so much that goes into this and it's really easy, I think, for some hype-focused folks to say, oh, well, like AI is going to improve efficiency 30%, you know, go, go, go, go, go, and not think about the work that has to be done to make this effective and actually sustainable long-term. What other fun question we usually get is, does it write secure code? I swear that question comes up like almost every time. Like, it will write good code.

25:21I cannot guarantee it is secure, to which usually my first question is, Do you have a good full pipeline in place with all your security and everything? Are you following the best practices, Dora metrics? If you're not familiar, the Dora metrics are, in my opinion, super good to really benchmark as a starting point for an engineering team. Adding Genative AI at least into the coding space without having those in place is a recipe for complete disaster. Because now you've just taken, you just put more in the front end of your process and it's not going to go well. So I see a lot of people adopting this before getting your software development all in place.

26:03And there's plenty of really good ways to go about doing that. But until that's operationally in place, high security, high velocity, high functioning, I think actually these tools in some cases will dramatically slow down an organization. and somewhat obscure developers' skills if you don't really know. So clearly it uplevels everyone. It is something that every developer is going to need to learn and adjust and use. So I think those are very clear. But it also may make it more difficult to tell which developers deeply understand nuances of perhaps your most important differentiated code. And so there's going to be many of those best practices always been there of pair programming, etc.

26:53Those are going to be critical to building up capabilities and building up team. Yeah, I've heard these models described as a great initial pair programmer for you to work with. But there's so much value to doing more within your team around that and how you construct your team. And I would be remiss if I didn't mention here that if you are listening to this, and you need to get basic visibility into your engineering org, you want to get those door metrics, go to linear b.io and sign up for free to get your door metrics for your organization of any size. And you can start getting that visibility layer so that you can be more effective in your AI initiative.

27:32What are other ways that you think teams need to adjust and maybe construct their internal infrastructure or processes differently to effectively work towards having AI that is actually helping them code and delivering more value faster? So from an adoption standpoint, that that's adopting it across same tools kind of across the board. So a lot of what the way these tools were originally adopted in fairness, still for a number of companies are they're adopted by individual developers. so an individual developer is like I'm going to use this tool now depending on what that tool that they're using that tool may be actually exfiltrating your code into someone else's mob which is not what you're shooting for so there's that aspect but I think it's also important for people like what's the best practices and have that and transparent yes we're going to adopt this tool yes here's how we're going to use it it's going to change the way you go about development I think it's going to change what work gets done.

28:39I think we've seen some teams, for example, move towards what traditionally would have a testing team. So you do development and then you have a whole testing team. Testing is actually one of the fastest ways, fastest and best use cases, because if someone throws, here's, you understand when you wrote the code, what are the tests that the corner cases you might need, but you're going to miss some. But generally, if you went through and say tab nine or another tool throws, here's seven unit tests for you, you probably thought of three and you can quickly look at other four and be like, yep, that's great.