#204 Peter Guagenti: How AI Will Change the Way Developers Work (Tabnine’s Vision Explained)

0:00There's a spectrum of capabilities inside of all these tools. And it's not just code generation, it's break fix, it's other generation, like things like documentation, it's fully autonomous testing, it's all of these things. And we think about the spectrum as running from an AI assistant at one end to a fully autonomous AI software engineer on the other. I think the most important thing to note, if you want to understand how this stuff works, is that the model is only the foundation of the house. The actual interaction with those models is where all of the actual success is. Hi, I'm Peter Guagenti.

0:29I'm president and Chief Marketing Officer at Tab9. I've got a long history in enterprise software, including having built and scaled companies like NGINX and Cockroach Labs. But actually, I spent most of my career as a web developer and engineering lead myself, actually, in management consulting and in digital agencies. And I worked on over 100 different company projects over 15 years doing what we now call digital transformation. And tell us about Tab9. You know, I've heard a lot of people refer to Tab9 before this call. I did a little bit of research, but I'm not entirely clear about the company's history and its direction.

1:12So can you tell us about Tab9's origins and what it does and what it's looking to do in the future? Absolutely. So Tab9 is actually the originator of the AI code assistant category. So the company was founded by two very good friends, Iran Yahav and Jorv Weiss. Iran was actually a professor at the Technion, which is a computer science program in Israel. And they've actually been focused on how to leverage AI and machine learning to simplify software development. And this was actually, they started this work almost a decade ago and had been making very good advancements. And then LLMs emerged. And back in 2018, the team actually released the very first LLM-based code assistant for Java, and that was Tab9.

2:02And so fast forward to today, the company spent in that time actually building out the capabilities, ironically, before CIOs or engineering leads really appreciated it. So it was mostly just selling directly to developers. There's now over a million users of Tab9 on a frequent basis. Millions have tried it and used it over the years. And about a year or so ago, the company really pivoted and started supporting large engineering teams. As a significant player in the category, we had engineering managers coming to the company and saying, okay, we want this capability. And they were drawn, unlike some of the cloud provider tools, they were drawn to a few things for us.

2:44One is we've always done fully private deployments. So there's no code sharing. There's no user data sharing. it all either runs in your private self-hosted environment, either VPC or on-prem, or in a fully private SaaS environment, which we operate, where you have completely private individual environments. And they also were drawn to our model flexibility. So we created from the very beginning what we call the Tab9 protected model. So we have our own LLM that sits under both our code completions and our chat agents that's actually trained only on permissively licensed code. We do support other LLMs.

3:22We actually have a great model that we developed in partnership with Mistral. We work with all the large model providers, including Anthropic and OpenAI. But the protected model gives highly regulated or highly sensitive organizations an option to only use a model that they know they have rights to use the code that are underneath it, which is pretty critical for many of these folks. And then you asked the question around sort of where are we today and where we're going. We've really been focusing on innovation around AI code assistants and other software development AI agents that are highly personalized individual engineering teams.

4:01We have a thesis, right, which is that generative AI is great. But, you know, if you look at, say, the code assistant category and use generative AI, it's like hiring a seasoned software engineer off the street, right? You're going to end up with someone who has very strong knowledge and really understands the principles. But if you ask them how to do something, it'll do it in a way that's academically sound and correct, but not necessarily a fit for you. And we think that this is one of the things that's holding back wide scale adoption of generative AI is fit to purpose. And so we've really focused in the last couple of years and continue to double down on leveraging context, RAG, even customization to models and other connections to other systems to really inform the AI platform.

4:45So instead of behaving like a software engineer off the street, it behaves like somebody who's operated in the company for the last five, 10 years, right? A fully onboarded, fully experienced software engineer. And we think that's the future, right? We think the more context and personalization we can provide around these AI agents where they really do fit you and how you work and your team's behaviors and standards and guidelines, then the more value that it's going to deliver. Yeah, you know, everybody has been following, and I've done a number of episodes on automated code generation. A lot of the things that have been coming out from AlphaCode to, you know, certainly Copilot and CodeWhisperer.

5:27And most recently, I talked to the founder of Julius AI, and, you know, everyone's waiting for Devin to be for their general release. The question that's being asked about all of these is, at what point are they fully autonomous? At what point can they generate code that you can trust without having them operating as a pair programmer and without having them make suggestions? uh where it is i know you guys came out with this atlasian jira i don't know if it's a plugin but integration uh that i think is is fairly autonomous can you talk about that and then what what's the process to get to that full autonomy uh julius of ai who i had on fairly recently is focused on tabular data, on creating an autonomous agent that you can work with to manipulate tabular data.

6:41Is that where this is going, that there will be these narrow use cases that are refined enough that they're autonomous? or at what point does it break out into more general coding? Yeah. Well, look, the way we think about this is there's a spectrum of capabilities inside of all these tools, right? And it's not just code generation. It's break-fix. It's other generation, like things like documentation. It's fully autonomous testing. It's all of these things. And we think about the spectrum as running from an AI assistant at one end to a fully autonomous AI software engineer on the other end. And I think I would be careful about the hype that you see around some of the tools that are out there that are showing really fancy demos and things like that.

7:33And I can make a really fancy demo as well, right? I can make something that looks like it's, you know, going to completely replace a software engineer, but it's not realistic, right? It's not realistic. And I think to answer your question, this is going to sound slightly conflicted, but it's important to say it is, yes, there are things we can do fully autonomously today. So we released around the Atlassian Team 24 event. We're an Atlassian-backed company through Atlassian Ventures. We're also a huge Atlassian partner. And we want to get people excited in showing them a straight Jira ticket to code tool, which actually does straight app dev.

8:09It's not a, you know, here's some code and maybe you can go fine-tune it. It was taking a ticket and actually creating a full node application. And that is possible today, right? It's just the question is, where are the constraints, right? And where we're very focused is we're trying to take as many of these discrete tasks as possible that a software engineer is working through today and take it to as full completion as possible. So an example of this, obviously, the first one was documentation. We do complete documentation generation across multiple touch points. That's basically fully autonomous now.

8:48You don't have to worry about that. Yes, we believe that we should always have a human in the loop on these things. We think that there is significant value in having a human in the loop. So even with that full autonomy, we want human review and validation, just like we have code review and things like that from architects and managers today. That's what a software engineer should be doing as AI works. and a lot of that capability exists pretty deep in the product. It's for break fix. We're seeing it. We have almost fully autonomous testing now available. That's very strong. We've got a lot of this capability fully developed.

9:23I think to the point of where we get to where there's effectively autonomous app dev, I think we are not far away. And I think the way we're going to get there, however, is probably through an iterative process, right? And a slightly different user experience than we have today with these AI platforms, where we actually have AI working in conjunction with a software engineer to ask further questions and ask for clarification and to propose options and get feedback, which is actually how we work today, right? It's how we work as humans today. And so, you know, we're probably not very far off. We actually have in the lab, you know, pretty advanced versions of a lot of this stuff already.

10:05But I think what's going to happen is we're going to start building up that layer cake a piece at a time by creating fully automated versions and autonomous capabilities for pieces of it. Now, I want to say one really critical thing in this, because I think there's this, the people who are here getting most excited about this are people who are not software engineers. And frankly, the people who get most excited are people who don't usually really understand how software engineering works. So the idea that these AI bots are going to come and we're not going to have software engineers anymore is foolishness.

10:36Anybody who's ever worked in a microservices, data-intensive, modern application will tell you that the level of complexity in a modern application is such that it is highly unlikely AI is going to completely replace humans anytime in the foreseeable future. And what I suspect will happen is as we automate more and more of these tasks and more and more of these processes, the apps themselves are going to get more complex. It's going to unlock the ability for us to do even more interesting things. And you're going to see an up leveling of the software engineer themselves to get out of the doldrums, to get out of the sort of rote, repeated tasks that tend to eat up a lot of our time and more into design and evolution and really advanced thinking.

11:21and I look forward to that day, right? I mean, if you go back 20, 30 years ago, we used to do things like memory management and you just have to figure out how to write to disk and all these other crazy things that we did when we did app dev. As we automated more and more of those things, think about how advanced the applications got. I can't even imagine what the applications are going to look like 20 years from today. Yeah, although if there is this iterative process where you're talking to an LLM or to a generative AI system and refining your intent to the point that it can code something that's executable or compilable, that does open up development to non-coders like myself, who may not be creating extremely complicated apps, but can create apps that the code assistants aren't able to now.

12:23And do you think that's important or do you think that's kind of ancillary to the main direction of where all of this is heading, which is to enable engineers to write increasingly complex apps? I think it's an interesting question because I think there is a question here on what is a software engineer and who makes applications. That's really what you're saying, right? Because the only thing that the AI assistant really replaces is the need for us to handwrite code, right? Because at the end of the day, you think about a modern microservices-based application with lots of nodes and inflows and outflows of data and specific experiences that you're creating.

13:13You do have to know how to architect and engineer that in order to do that well, even if someone else is writing the code. I started my career as a developer and designer. Started out doing both user experience, designing the system, and then actually hand coding it. Very quickly got out of the code. and then eventually even got out of having to do all of the flows, right? And however, we're still architecting the actual product and the actual application. And, you know, just we kept moving up the stack, right? And all those things. And I think to imagine a world where someone who's a complete layman and doesn't understand how the system works to actually go and quote-unquote write an application is probably unrealistic, right?

13:52Because you actually still have to understand the interplay and interconnectedness of the data and the systems and the components around you. I'd imagine though that this starts to look 10, 20 years from now more like the way an architect builds a skyscraper, right? Where they don't have to go and plan for every piece of material and they don't have to plan for every component that goes in, but they have to have a vision. They have to understand how these systems work and interact with each other, at least if you want them to perform well and to be successful, right? I think you're seeing it already, probably sneak previews of it today, even without AI.

14:26functions as a service, drag and drop componentry. I spent a significant portion of my career working in open source frameworks like Drupal and WordPress and Magento. And at the time you could sort of Lego snap together applications, but you still really did have to understand how these things worked. And I'd imagine that, and this is pure speculation, obviously, but I'd imagine if you fast forward 10, 20 years from today, the people who are still going to be most successful building applications are the ones who understand the total system and how these systems work and pull together the right componentry, I just imagine a lot of it is going to be generated automatically by AI for us.

15:01And Tab9, you started as a Java tool. Are you in other languages now? Yeah, no. The very first LLM we built was basically just trained on Java exclusively, right? And that was, you think about, this is back in 2018. So this is now six years ago, seven years ago when we first started doing this. The LLMs were very limited. The context windows were very small. We had to be very focused. You fast forward to today, however, and we support over 80 languages and frameworks. We support anything that really any software engineer is typically going to work with. And then on top of that, Tab9, unlike most of our competitors, we actually do fine-tuned models.

15:46So we work with mostly very large engineering teams who are our enterprise customers. We have sort of individual developers and small teams who use our SaaS product, but our enterprise customers tend to be big banks. They tend to be defense organizations. They tend to be pharmaceutical companies or software companies or even hardware manufacturers. And so with them, a lot of times we're actually going and doing fine-tuned versions of our models that are trained on their code. And that starts to get really interesting when you do that, because not only does it make the behavior of the model stronger for your specific use cases, but actually it allows us to close the gap on obscure languages and frameworks that tend to be not a lot of great public training data for.

16:28You think about machine code for very specific hardware companies or obscure software languages that are used in financial services. You see a lot of that. And so that 80 doesn't even cover it. I feel like most of the time these days, when someone asks us for language coverage, we just immediately go in and start testing against against their use case and start immediately seeing that it's actually super performant. The powerful thing about these LLMs is these LLMs that basically know how to write software. So as long as they have the visibility to the structure of a programming language, they're incredibly successful with that language.

17:04Yeah. But then exactly that point that they know how to write software, That would argue that eventually they would be able to architect a piece of software, or at least guide a user in how to architect a piece of software beyond the coding. Can you talk about developing the fully autonomous JIRA tool? How did you train it and what its capabilities are? And then talk about how do you then take that process and broaden it to something else? So I think the most important thing to note, if you want to understand how this stuff works, is that the model is only the foundation of the house. The actual interaction with those models is where all of the actual success is.

17:57And a perfect example of this, go and use, you know, something like ChatGPT directly and ask it software development questions. And then go use tab nine connected to GPT 4.0. and ask the same exact question. And you're going to get a radically more evolved answer. And that's actually because the LLMs themselves are really just a font of knowledge. That's what they really are. And there's this whole other layer of work we do around context and what information actually gets sent into the LLM when you ask it a question in order to give it appropriate context and understanding, plus the prompt engineering itself, plus even just the UI UX.

18:37like, you know, things we do around helping to structure the question or ask follow on questions and things like that. And the way I think about this is and this is important context to explain the autonomous generation. If, you know, you and I met on the street and you asked me, you know, I'm looking to buy a car. What car would I get? Right. And I knew all about cars. I would give you 100 answers that are all potentially relevant to you. But if you're my best friend and I know you, you know, you've always wanted a convertible and you live in warm weather and you are, you know, you just had a windfall of cash and you really want something fancy and you want to be known.

19:15Like, I'm going to start directing you in a very specific case versus, you know, somebody who's new family. They've got three children. They, you know, they do the school run and they need something that's going to be safe and practical for that. You know, it's a totally different context. And that understanding is really key to be able to operate autonomously, because it's never enough to be able just to answer the question. You have to answer the question appropriately for the use case, for the context of the organization it's serving, for the users it's serving, for, you know, 100 parameters.

19:50And so what we do today natively in tab nine is we use local code-based awareness, global code-based awareness, and then integrations to non-code sources of information on top of all of our prompt engineering, all the work we do in order to make sure that the recommendations come back really strong. So local code-based awareness is literally looking at every open file. It's looking at all this, the things that are accessible from there, you know, things like the errors and other things visible from within the IDE. global code-based awareness is just that. It's a Git connection into whatever your repos are.

20:23And then non-code is things like Jira and Confluent or whatever. So why is that relevant to the autonomous? Because when you start giving it design parameters, it's not enough to have all the design parameters. It has to understand what else it's interacting with and how else it works, right? And so the Jira autonomous agent that we built is actually very simplistic, right? It's really just looking and saying, okay, this is the design specification. Now let's actually go and generate the application. It makes a bunch of assumptions around language. It makes assumptions around data storage. It makes a bunch of assumptions that are built into that.

21:02That if you have full context, then those aren't assumptions anymore. Now they're actually parameters that are being fed in. And so if you think about where this has to go from here, and we had this even in, if you watch the video of the Jira agent we built, it actually does ask clarifying questions, right? And we think that that's the future of this is taking that spec and then identifying what it doesn't know. And that's a really, really challenging problem, right? Because AI is not self-aware, right? So how do we figure out what it doesn't know and and what it needs to know in order to solve that problem.

21:42And that's really where this next wave of innovation is going to come from, is being able to resolve that, usually through a combination of what additional context we feed it automatically, what additional context we ask for, and then what clarifying questions we need to resolve. And I think that's going to get us to a place of where these agents are behaving more like Jarvis in Iron Man and less like your chatbot at the call center where you're just trying to get tracking information for your package, right? And that context, the global context you were talking about, how is that fed in to the LLM?

22:21I mean, do you have, do you load up a vector database with all of the, with the code base around the problem at hand, or have you just fine-tuned the model on a specific? In our case, it's actually a vector database, right? We're using retrieval augmented generation. What we have done, however, is we have a number of proprietary things that we've done around what specifically we're using as context and in what way. Because that's really the art, right? The art of getting these things to respond the way you want them to is all of the effort you do around that, exactly. And so what we have is basically a vector database that gets built.

23:03It gets preloaded with the parameters that we believe are appropriate. it. And that's both based on our local code base awareness. So if like you're one of our credit card based customers, it's whatever we can access with your permission from in the IDE. If you're an enterprise customer, then it's what an administrator gives us access to in their Git repo. That vector database for those customers actually does live in their environment, not in ours. And so does the model, actually, if they use one of our private models, it also lives in their environment. And so that's all self-contained. And we've basically have been testing and tuning this for years now around what is relevant and what is not relevant when we think about the prompt engineering around this and then what context goes in.

23:43LLMs themselves are not smart things, right? I think this is where we find them remarkable because of this idea that we can ask these sort of freeform questions. But if you actually start using them for very specific generative use cases, you think about there's a great article recently in Fortune about the rise of generative AI in law, right? What you're discovering is you go and ask chat GPT legal questions and it hallucinates like you wouldn't imagine, right? And it'll even tell you things that it'll tell you citations that aren't real, right? And it's already caught a bunch of lawyers out in cases live, right?

24:21And, you know, but then you go and look at what LexisNexis is doing and these folks who really understand it, they're doing some really advanced things once again to use their knowledge to check the knowledge. and eventually, and we're already starting to do this with some of our code review agents, which we'll roll out later this summer, we have models checking the models. So we have models sort of holding each other accountable, right, in order to eliminate any sort of hallucination or identify any issues. And I think that's the kind of effort that's going to be required for generative AI to actually be useful.

24:53Yeah, and on the vector database, presumably you have that workflow automated. that you plug into? Yeah. Recently, I was talking to people at Boston Consulting Group who are way ahead of the other groups on AI and generative AI. And they have been experimenting with loading, instead of using a vector database, just loading, you know, as these context windows get larger and larger loading uh you know a couple hundred pages of uh content of text or code or whatever it is into the prompt or into the i'm not sure if it's it's not really a system prompt well maybe it is a system prompt but the and then when you ask a question the inference comes out of that i mean it's like rag but instead of having uh to first create a vector database you just put all the stuff front loaded in the prompt.

25:59And understandably, that doesn't really work for a lot of use cases because it creates a lot of latency issues. But would that be an easier way to do it? I mean, this show is sponsored by BetterHelp. When your schedule is packed with kids' activities, big work projects, and more, it's easy to let your priorities slip. Even when you know what makes us happy, it's hard to make time for it. But when you feel like you have no time for yourself, non-negotiables like therapy are more important than ever. Therapy is no longer something to be hidden. I've benefited from therapy. Many of my friends have.

26:41If you're thinking of starting therapy, give BetterHelp a try. It's entirely online, designed to be convenient, flexible, and suited to your schedule. Just fill out a brief questionnaire to get matched with a licensed therapist and switch therapists anytime you like at no extra charge. Never skip therapy day with BetterHelp. Visit betterhelp.com slash IonAI today to get 10 % off your first month of therapy. That's betterhelp, B-E-T-T-E-R-H-E-L-P.com slash IonAI. IonAI all run together, E-Y-E-O-N-A-I. Go to.com slash IonAI for 10 % off your first month of therapy. You'll enjoy it. No, I mean, it's easier from an engineering perspective, but it's actually like the brute force method.

27:39I mean, I would argue that whoever is doing that doesn't actually understand how context works, right? Because if you're just loading that in, And you could do, an individual can do that with ChatGPT today and just copy and paste an entire catalog of information and ask it questions about that information. I think the thing that they're missing in that is some of that information is relevant and some is not. And it's not just about latency. Yes, it'll dramatically increase your latency if you're feeding in that much information for it to process in every single query that you're putting out. But also it might give you irrelevant information.