What an AI Exec Wants to Hear in an Interview | with Paul Wachtler, SVP of AI at Exiger

0:00Shamil Malachiyev: Hi, everyone, and welcome to this week's episode of the AI Floor podcast. My today's guest is a Senior Vice President of AI Data and Strategy at Exeger, the supply chain and third-party risk platform used by almost a third of Fortune 500 companies. Please welcome to the studio, Paul Wachtler. It's really critical to have some AI literacy and I'm looking for people that have used AI successfully in their organizations. I don't even care so much about which tools you're using, as long as you could talk to me about how you're using them, what issues and obstacles you're running into. You don't need to teach the technology, then you're just teaching the business.

0:36So not just having used, you know, ChatGPT before, but having examples of where you've leveraged it for positive outcomes and being able to explain that. Get a free ChatGPT account, get a free Claude account. You don't need to feel like you have to use the latest tool all the time. Use what's free at first. If you hit the limits, maybe go to the$20 a month plan. I do recommend staying on monthly plans versus annual because these models do change. Most people still don't need agents for using AI. I think just to be able to set up projects in ChatGPT or Claude, keep your contacts together, have a shared set of instructions and sources and just work from that, it's probably more than enough for most people.

1:18You have the system ask you questions and it's going to ask you and clarify things that you never thought of. If you say you saved hours without any measurement, it just kind of goes in the air. If you're motivated and interested and learn just a little bit more, put yourself in there, do a little bit more with it, you put yourself ahead of 75 % of the average people.

1:37Shamil Malachiyev: Hi, everyone, and welcome to this week's episode of the AI Floor podcast. My today's guest is a Senior Vice President of AI Data and Strategy at Exeger, the supply chain and third-party risk platform used by almost a third of Fortune 500 companies and over 50 government agencies, including the U.S. Army and the Department of Defense. Paul spent 15 years in strategy and analytics at SaaS companies like Zandas before joining Exeger, where he's gone from running business intelligence to leading how AI gets applied to real business decisions. Please welcome to the studio, Paul Wachtler. Hi, everyone.

2:13Great to be here today.

2:14Shamil Malachiyev: Thank you so much for having me. So good to have you here. Could you tell us a bit about some of your background? Sure. And the thing that excites you the most about the way the technology is going currently? Yeah, absolutely. I've spent most of my career in data and business intelligence, leveraging different tools to present information to executives and drive decisions and strategy from that. Seeing a lot of different iterations of tools over the years. And just recently, especially this year in the last nine months or so, The pace of development of recent AI tools, AI-enabled coding, and other things that have come out have really changed the shape of what is possible.

2:56So I've gone from being a data and business intelligence leader to now focusing really on not only how to do data and business intelligence leveraging AI, but all of the pieces of AI that come along with running the organization. how to successfully implement AI, measure ROI from it, get positive results. I think there has been so much growth and all companies are trying to make sure their people are leveraging it properly. And so I'm excited about the opportunities that that brings and the work that I've been doing around that.

3:33Shamil Malachiyev: I think quite a lot of companies right now are getting the FOMO, especially when you're reading that almost every company is becoming full of builders. like everybody in the company is a builder now and it's sometimes funny when like some of the linkedin posts that i'm reading they say like oh whilst you are rebuilding a saz platform that you're using to do internally your customers are doing the same thing to your software and so on so it's kind of like a funny funny time in the industry yeah yeah no definitely you know funny as well and you can't compare yourself to linkedin or twitter because anything you read there is going to make it look like you're light years behind.

4:12So you're always going to feel like you're missing out if you're looking there. But realistically, I think it does allow a lot of people to build that previously didn't have that capability that weren't software engineers or developers. And they're able to take these tools and get to a working prototype of something without having to wait on a backlog or engineering or wait a few weeks until something turns around. And not to say that everything that everyone's creating is great, but I think it also lets you get to failure a lot faster. If you're looking at something tangible, then it's a lot easier to say, okay, this is not what I want.

4:45Let's figure out how we get to what I do want. I also think there's different levels that people are naturally going to be at as builders. Some are going to be a lot more advanced, use some of the more developer focused, you know, cloud code, codex, tools like that. And then you're going to have people that are, you know, vibe coding, doing AI enabled coding in Replit, Lovable, or other tools. And then you're also just gonna have people who, you know, use ChatGPT like Google, just ask it questions and things like that. So I think there's a lot of different levels. It definitely enables people that didn't have that ability before, but you're going to see that stratification of, you know, how people use it based on their comfort level and what they're motivated to do.

5:28Shamil Malachiyev: And now that, you know, building has become a lot simpler, I think the main question becomes, what do we build? Like making the decision, like what will actually be useful? Like what to focus our time efforts? Because building includes a lot of polishing for use cases, making sure that it's really like valuable to the relevant party that it has the guardrails and everything. And I guess having the business intelligence behind it makes it easier for you to justify the decision of what to build. Sure. The business intelligence piece definitely can inform it. I mean, you're using data to inform your business decisions.

6:03But I think as with a lot of things that we're seeing that, you know, AI impacting business, a lot of these things have been around in business since well before AI. So before AI, you had to determine, you know, what you're going to work on, what the right path was and how to measure that and, you know, determine that successful implementation. And so I think now you could get there a lot faster. It's a lot less cost, at least initially, to get something out there and then iterate from there. So I think it's more important than ever to prioritize based on your business objectives, what the company's working for, you know, translate those into what you have now and what you want the outcome to be.

6:41Make sure it's measurable so that you could see that you're actually working towards progress or maybe you're not. Maybe it's not working and that's important to measure those things, too. So I think AI magnifies a lot of these business problems. It makes them a lot more obvious, a lot larger than maybe they used to be. But yeah, same business principles as always apply.

7:03Shamil Malachiyev: And working with business intelligence, like in Exeter, having access to so many other different companies, seeing what they're struggling with, what they're building. What would you advise to somebody from Fortune 500 down to a founder of a small company? Sure. What would you advise to be one of the first AI projects to build internally and to focus on? Yeah. And I don't want to be cliche and say it depends, but I think the most important thing and definitely in business, but also just for personal AI development, I think find something that you know about, something that you're an expert about, that you have a general idea of how things work and whether there's rules around it or what the constraints might be, and try to automate something along those lines.

7:55So if you're a musician, maybe you do something with music. If you're an athlete, maybe you do something with sports at work. I mean, if you're a tech company, maybe you just try automating your weekly reporting or something like that. Just take something, and especially at work, everybody has tasks that are manual that they spend a lot of time doing every week, every month. Take one of those tasks, automate it, see what it takes to get there and be mindful because you're going to have to spend a lot of time reviewing the output, making sure that it looks just as good. The data is just as good as when you were doing it before.

8:34But hopefully over time, if you're measuring, you should be able to reduce the amount of time there and then take that time and do something more important with it. I think that's also an important area for businesses is it's not enough just to save that time from AI, you have to be really mindful and measure where that time is going. Are you doing more work? Are you saving money somehow? Are you, you know, putting it towards more development or whatever it may be? Just be able to measure that because if you say you saved hours without any measurement, it just kind of goes in the air. It doesn't really lead to anything.

9:12And how about actually like, well, in order to do data intelligence, I guess you need

9:17Shamil Malachiyev: to also have a lot of data which is structured well because i know a lot of people just like when i ask them like oh what's your data situation it's like oh i just use an mcp connector to google drive and it just finds whatever it needs and then creates whatever it needs yeah so it will be the correct approach to kind of to a more like production ready preparation yeah no absolutely and i think this is definitely one of the areas that ai magnifies i think previously with business intelligence, there was always talk about semantic and context layers, having your data well modeled, understanding the different business scenarios where certain things apply.

9:54I think for a lot of companies, especially smaller companies, a lot of that context and semantic work lived inside the analyst's head. They knew who to go to and what people wanted to see and when to format in certain ways and whatever it may be. So you didn't need to necessarily formalize that process. AI, on the other hand, is taking whatever you give it. So if you give it unstructured data, it's going to give you something that sounds good. And it's not going to be really it's going to be for the wrong business context, be for the wrong time frame, whatever it may be. It's not going to give you what you're actually looking for.

10:27And so with AI, you need to be a lot more explicit about defining what all your data means, how it joins together, the context for all of that. And one thing I've been getting into more recently is just having to manage that context over time because businesses change. So it's not enough to define it once. You need to make sure that as your business is growing and moving and different decisions and different things are happening externally, that your context is staying updated. And that in itself could take a lot of work. You might need a team just to maintain context. So, yeah, absolutely. It's more important than ever to have that well-defined when using AI.

11:06And yeah, if you just point an LLM at your data, especially if it's unstructured, you're not going to get anything valuable. And you're going to end up getting yourself in trouble by presenting something that is just not related to the actual data.

11:20Shamil Malachiyev: And how can somebody understand whether their data is structured or not? Does it just mean like having everything live in a post-grade databases? No, and there's different levels to this too. I think it is probably best practice to have your data laid out well in a data warehouse and some type of organized, I want to say documented structure, but even documentation gets stale pretty quickly. But honestly, even if you had a spreadsheet and that was your data source, not that I recommend that, but for a small company, if that's what you were using and updating it, it's more about working with your AI so it understands what you're asking, when and where.

12:04And so taking time to have it ask you questions about what the data means in different situations, how it might come in differently. And then you need testing and evals over time. You need to see if the responses are within the range that are acceptable for you. I mean, is it responding well? Does it degrade over time? And yeah, just back to the maintenance point, I think even if you set it up perfectly well in a month, it may not be the same situation.

12:33Shamil Malachiyev: and uh for our listeners paul is like throughout the years you've been doing quite a lot of presentations on on the very practical aspects of uh building with ai especially over the last nine months as a technology has been advancing in terms of building maybe you can speak to their practicalities or like how would and every person who's heard about ai is on linkedin they see everyone talking about building your own product like how do they actually start yeah what is the correct way to get their feet wet yeah absolutely um so this is i led a workshop earlier this year at the chatbot summit berlin um 90 minutes and basically showing business leaders how to take an idea and get it to a prototype quickly do some iteration between systems and then by the end of that 90 minutes you have something tangible that you could then present with just to get show them that process.

13:31So I think you don't need to start with paid tools, especially if you're just doing this on your own outside of your organization. You just want to start getting into it and seeing how it works. Get a free ChatGPT account, get a free Clot account. You could work on Replitz, free model. A lot of these tools have free models. And if you start using them, you'll quickly run into the limits, but it at least lets you see what's possible and how they work. What I have found incredibly powerful, at least with AI enabled development has been working between an LLM and a AI enabled application like Replit or Lovable or something like that, or Claude Code, perhaps.

14:11Basically, the idea is you work with the LLM and I don't care which one you use, but just to define your idea upfront. So you're talking to it about what you want to create, all of the business aspects behind it, anything that you could think of. And then the most important part after that is you have the system ask you questions. So after you do your brain dump, you get everything out. You say, okay, now ask me clarifying questions about how we can implement this. And it's going to ask you and clarify things that you never thought of. And at the end of that, what you want to get, you want to ask your system for a build spec that you could then put into your tool.

14:48So you don't need to think through every single technical piece of it. You get this build space. You could put that into the tool. Basically, after the application's built, you iterate between the LLM and the tool. And after a round or two of doing that, you have something that's pretty good at not ready for production, but it's at least ready to go into a meeting with and potentially drive a decision, potentially show people what's possible to do. So I think, you know, even for the average person that hasn't started. If you're looking to build something with AI, if you're looking to get to something that you could use in your organization, start with an LLM, get comfortable just working back and forth with that.

15:28But then adding that AI enabled coding piece, whichever tool you decide to use, that's really going to accelerate what you're able to do.

15:36Shamil Malachiyev: And how much of the time should people be spending on creating a SPAC document before actually? yeah um honestly it doesn't have to take too long it it's up to the individual you could have it ask you 40 questions you could have it ask you five questions um in the workshop because i was mindful of time i was having it only ask people five questions at a time i didn't want to you know go into huge detail um if it's a passion project and you're trying to build out something that you've had in your head forever and you want to build it out maybe you have it take longer and you really get into all the detail and everything.

16:10If you're trying to iterate on something for work and you have a pretty good idea, but maybe you don't know all the ins and outs of how it's going to work, or maybe there are still decisions that need to be made on how it's going to work. You don't need as much. You could really spend maybe 15 minutes, maybe less just to get, especially the initial build spec. You're getting everything out there that you want. You're having to ask you questions. It's going to give you that build spec and you just put it into your tool, let it run and see whatever you get out of that. Now, depending on which tool you run, you need to worry about maybe hosting the website and all the other production pieces that come with it.

16:46And that is where a replet or something like that makes it a bit easier because they have hosting themselves. But yeah, just being able to iterate faster and get to that point where you could show something is just, it's really incredible.

17:00Shamil Malachiyev: When working internally with my teams, I keep telling people to first create visual mockups HTML mockups that anyone can look at, click through it's kind of like one picture could replace a thousand words and I think one working prototype could replace a lot of pictures it kind of gives you the full perception of what the product is going to look like and what is in your opinion the main kind of point of having a prototype and then having it be ready for the actual production use? Sure. I think there's a whole process that you need when looking to take something from a demo into production. And there's the technical side, which every company has dealt with on their own of how they push things into production and the security, the authentication, the authorization, everything that goes into making sure that it's productionized on the technical side.

17:59I think just as important, especially with AI is the business side of it to make sure that you're not just putting this tool into production because you built it quickly and you think it looks nice, that it's actually going to fit the business process. So you really need to make sure that it fits what the organization needs, the people that are going to use it, the use case, that it could scale to beyond what it's originally supporting. And sometimes applications don't need to go into production. If you have a pilot application supporting one or two small teams, it doesn't need to then go on to support the whole organization as long as it's well managed, as long as it's well reviewed and everything's working out well, but you don't have to take every tool to production.

18:43So making that business decision explicitly on what you're going to bring to production, why, and what you're testing. And maybe you do a test where you only put it into production for a month or two and you measure, you see how that goes, and then you can always take it out later. But But just being mindful of making those decisions explicitly is absolutely required when you're putting an AI project into production.

19:07Shamil Malachiyev: And I guess you have experience just as much of like building internally for Exeter, as well as participating in consulting other companies, government agencies in terms of like AI implementations. And you are in crossroads of like heavily regulated industries. We have not just the ISOs and SOC 2s, but also like FADRAM and ISO 41. What is it like building in heavily regulated places? Because at the same time, you want to go faster, but you have to be extremely careful. And there is a lot of red tape that you have to go. Yeah, yeah. No, especially in my work at Exeger, you know, heavily tied with our security teams and our legal teams, making sure that everything we do is within the regulations that we need to follow for our customers, for the different countries we operate in and whatever else it may be.

20:00So, I mean, we had a very tightly coupled organization for that previously, but obviously with the rise of AI, the rise of us using AI and different tooling and solutions, we've had to be mindful of how we implement all of that and track that and what we're able to be audited for and whatever else may come along with that. So it is even more important, again, an AI magnifier where you need to make sure you're following every regulation. And it's not you're only giving the tool the permission that it needs within specified guidelines and specified boundaries because you don't want it to go outside of that.

20:39So I think, you know, just being extra mindful of it, being proactive about it, too, you know, understanding the environment that you're operating in, what you need to follow, what you might need to follow in the future if you're going after different customers or other things, and being able to implement that into your processes and, you know, be successful, not have it, you know, totally take down your capabilities and what you're able to do.

21:05Shamil Malachiyev: because internally it was pretty helpful for us like for example going for ISO 2701 certification there were a lot of security questions that we've never thought of that we had to think through and implement and then we had SOC 2 and then closer to like the end of the review period when we started asking about AI questions the answer we got was well SOC 2 doesn't have specific explicit kind of rules regarding AI so it's up to you you create your own ways of like how do you want to regulate Like, how do you want to prove to everyone that you're regulating it well? Are there any, like, standards or tips that you could give to companies?

21:42Shamil Malachiyev: Like, these are the, like, 90 % chance questions that you'll be getting from, like, vendors or buyers that you should be thinking about. Sure. Maybe, like, don't use DeepSync free for your API key. No, and it's very dependent on where you are in the world and how your organization operates. Companies in the EU have different feelings about which companies to use versus companies in the US, how we might feel about it. So, again, I think having, especially inside your organization, just being able to work with your legal and IT security teams and being mindful of what you need and what you don't need because you're not going to need to fulfill everything.

22:24There's a lot of country specific regulation now, especially in Europe. I mean, the European Union has been pretty proactive in not allowing certain AI use and things like that. So if you want to do any business in Europe, you have to be mindful of that. I mean, the US government as well has put in different things that you need to track and be able to report on and things like that. So yeah, no, you need to be proactive about that. Now, this doesn't apply if you're just trying to put together your first AI application. Like you could just dabble and you don't need to worry about legal repercussions.

23:00I think it's when you try to either sell what you're doing or you're organizationally, you're trying to, you know, sell something with AI capability. That's when you need to be mindful of it.

23:11Shamil Malachiyev: And I guess it's like the more heavily regulated of the industry you're trying to sell into, the more questions that you'll get. yeah like um if you're working for a financial institution they're probably going to have a lot more restriction on where data could live and where it's able to go versus you know a commerce company or something like that and are those companies within like financial industries i can imagine that you know if you're looking at the cutting edge of ai technology all the companies adopting that and then you look at the regulated industry they must be moving a lot slower Like how much slower are they moving in regards to the?

23:51I think it's just different. I mean, so you have organizations where they're building the frontier models like OpenAI and Anthropic, and they have stronger models internally than the rest of us can see. So that's a different experience versus a traditional large financial company that, you know, they have huge technology teams. They know of the technology. They know how to use it. I think, especially for them, they've gotten very used to bringing models and technology in-house. So they're probably much more likely to deploy in-house LLMs, in-house agents that can't get out of their systems. And so I think, you know, they're probably, especially any financial traders, they've been doing things like this for years and years.

24:38They've had, you know, the inside advantage and the quant analysts and all of that able to pull things apart. So I think, you know, they're not allowing broad access to large open models, I'm sure. They're very mindful about their data getting out. But internally, they're probably likely to be setting up their own models, maybe even creating their own financial LLM-focused models to run with it. You know, they're always making record profits. I'm sure they're not too worried about it.

25:09Shamil Malachiyev: when it comes to question of like choosing the right the correct llm model because i remember the times like half a year ago like march around like february march 1026 and all of us were like waiting for the next model to see like how much more capable it's going to be and i think starting from around maybe july once we had astra fable i don't i don't know like if everybody's waiting for these new models because the old ones are pretty kick-ass like do you really need higher like more sophisticated models yeah i know that that's a really good point and i think the capabilities are amazing in what they're able to do but for the average person they don't need to be running on astra or fable for a simple question or for planning something simple and so i i think for the individual you probably run into it if you hit your you know weekly monthly credit limits in one of the organizations and then it asks you to upgrade and that's how you'll realize like maybe I shouldn't be running these higher models.

26:13I think more so for organizations, this is becoming a lot more of a factor now, especially as people were pushing or organizations were pushing their people to use AI more, just get it integrated into your process, weren't so focused on where that spend was going. But now they need to be mindful of who's running which models for what, who has access to which models. You may not need to give your team access to all of the latest things. And so it's become much more of an operational issue on how you deploy the models organizationally, who's allowed to access it for what, and switching between the right models depending on the task.

26:55So that is a lot more in focus now than it was, especially a year ago and it's just a new um you know wrinkle in how you manage ai and if we extrapolate into

27:07Shamil Malachiyev: like three years forward models are getting more and more capable that capability becomes cheaper and cheaper sure eventually companies will be able to run like pretty much most of them on prime fairly good models maybe like opposite five kind of level and do you think that is how we're going to end up with each company running some sort of their own model being protective of their own data perhaps i i mean i think with ai in general a lot of its power does come from its ability to pull in additional sources of data and you know put your data against external sources now again like we were talking about regulated industries they'll definitely focus more internally i think for the average company, the average tech company, it's a lot more work to manage on-prem servers and backup and recovery and electricity and whatever else you may need to run in that than to pay somebody a subscription to have them run it for you.

28:09So I think Anthropic and OpenAI will get a lot more integrated into enterprises. They'll find ways to assuage some of the security concerns that may be out there. They'll make it cheaper. They're going to integrate into more of your systems. So it's not just going to be plugging in AI. It's going to be your AI CRM and your AI finance tool and all of your other different tools sponsored by OpenAI or Anthropic or whatever else it may be. So they take over your enterprise implementation. And so, yeah, I mean, just in the last nine months, the progress has been incredible. So in three years, I really, it's so hard to say where it might be, but I think it'll probably still for most non-regulated or heavily regulated companies, you're still going to be working directly with these frontier model companies.

29:03Now, it's not to say that there's another company out there we haven't heard about that's going to take them all over. It's not to say that, I don't know, maybe Meta will have the best model at that point, or maybe Google will take it over. And we've seen that just with ChatGPT and Anthropic over this past year, them going back and forth for that top spot. So the possibilities really are open. But yeah, right now is the worst AI that we're going to see going forward. So if that's the baseline, then it's really hard to imagine where it could potentially go.

29:36Shamil Malachiyev: and you probably follow some of the news on like the new decision-making AI yeah the Jav and I guess when it comes to like AI decision-making like business intelligence how much of an impact does that play I think especially when you're running LLMs internally on your processes and you're using it as a decision maker as an orchestrator to you know put different tasks to different places, you don't need an output in prose, you don't need sentences. And so these models like JEV that are able to not only be incredibly faster, but a lot cheaper per token than the larger models, I think that plays a huge part in internal operation of your system.

30:25So, you know, there's concerns about security in those models and all that. I think that's with any model that you may use, but definitely just the evolution of going to cheaper, faster models that give you better outputs that work better within your systems because your systems are not humans. They don't need that output that we may want to review. And so it's been huge lately. I think we're just going to see more and more integration of those different types of models. With that said, you're still going to want the LLMs that can give you that output that you can integrate with that you can talk to and so again it's just really figuring out that right balance of which tools you use for which situations where and how you manage the cost and contracts and

31:12Shamil Malachiyev: everything that come along with that i've seen a lot and i'm hearing a lot lately with like countries trying to be more protective of the basically intelligence as the main kind of resource that they are now producing. Sure. And when we look at on-prem solutions, they're mostly used like open source AI models and the most capable ones seem to be coming from China. And yes, they might be distilled from the, you know, the U.S. models that we use. Do you think there will be some good alternatives for U.S. companies? So this definitely goes back to what I was talking about before with the differences in which company or which countries you're based in.

31:56I know in Europe, the models from Asia are very well received and are more in use versus in the US traditionally, there's security concerns about any software originating in China or Russia or anywhere like that. And so not that they're not in use, I think especially smaller organizations have less concern about regulation and things like that. I think it's just when you get to a certain size and you're selling to certain customers, they don't want Chinese software involved in the process. And so that becomes difficult, but no, some of those models, DeepSeek and others are incredible. They run a lot faster and for a lot less cost.

32:41So again, just managing all the infrastructure pieces, including the regulation and what you're actually able to do with your company.

32:50Shamil Malachiyev: another thing that i wanted to get your opinion on is like one we're all accelerating and you know every month the more ai adoption we have the more output we are ready to provide how shall leaders of like small teams or like say 10 20 people think about the new kind of paradigm of human efficiency because okay like we have the capacity to do 10 times more we start doing two times more and that's pretty much enough to like fix the bottleneck that used to happen there but now we have some 80 percent of capabilities or 60 percent of capabilities that and we don't know where to spend do you have an answer to i know it's it's a complex question No, no, this is great.

33:39I think, you know, depending on where you are in the organization, it's going to be easier to implement than others. And not that it's the right measurement, but let's say you're a software engineer, just a development team. You could see how much code you were putting out before, how many merges were accepted, and you can measure that over time. Not necessarily meaning that, you know, the code's getting better that AI is creating. You could just creating more. But there are more traditional measurements available versus if you're on a legal team or a finance team or, you know, an internal support team there.

34:15It's going to be harder to figure out the exact measurements. And so what you need to do is you need to think about the goals that you have for your team. and you probably spoke to your boss about them last year, what you're working towards, what your objectives are, and break those down into steps that you can improve with AI. So like I said before, if you have those manual processes you're doing over and over, that's a perfect use case for it. If you're creating a lot of the same materials for different board reviews or sales reviews or whatever it may be internally, that's another place you could look.

34:51So I think it's really being mindful of where you're spending, I want to say lower quality time where it doesn't really take as much of your creative input. And it's just a lot more busy work and try to automate that so that you can focus on the larger picture and maybe more strategy, more of how your organization is going to help drive or how your team is going to help drive the organization. There's a lot you could implement there. So it does depend on which team you operate in. But as long as you're mindful about the goals you're trying to achieve and being able to measure them, you should be able to have some success.

35:28Shamil Malachiyev: And if we look at Paul, you know, before joining Exeger to the Paul we see now, what are the main differences in you and your view on AI and technology? Yeah, I joined Exeger in January of 23. So very early on in AI, you know, I was there when ChatGPT just started picking up and we're slacking each other and like, look at this tool and what is it doing? I think there's been an evolution in organizations responding to AI. I think initially it was intrigue and, you know, just trying to figure out how it works and how we could use it. There was a lot of security concern up front. Some companies just didn't want it in their systems or in their data or connected to anything.

36:19I think we've come a long way where there are security agreements you could have with these LLM companies. You could have enterprise agreements. You could firewall different things and control access to different areas. And like we spoke about before, just the importance of getting a handle on how it's being used in your organization, by whom, where, for which circumstances is so important. Because before it was kind of the Wild West, like, hey, everybody use AI, use whatever you want. Like, let's just figure out how to use it. now, especially as costs of AI implementation are rising. And, you know, there are all of these AI costs coming up on everyone's P &L that weren't there a year ago.

36:58They're very mindful of, hey, not only do we need to get a handle on this, but we need to get ahead of it so that we could be smart with our AI investment. Where should we be spending that money? Where do we want to double down and really, you know, use those larger models in certain areas. And then in other areas, you know, just level up our teams to a certain level of proficiency. So I think that's important to just understanding how you want AI used in your organization at different levels. Again, not everybody's going to be at the same level, not everybody needs to be, but there needs to be some baseline level of proficiency, so that they could operate in an AI organization.

37:34Do you ever get burnt out? Because of like, how much faster you, you work with the help of AI, just the shooting decisions that you have to make i i definitely do have a lot more screens up at once with agents and different tasks running and so i feel like for better for worse i've gotten better at context switching between different tasks like i can let one run go to another one see what's going on over there i don't necessarily recommend that um i think you could get burnt out with technology no matter what. But yeah, I think as long as you know what you're working on and what you're trying to accomplish, and you still need to set some boundaries around your capability.

38:18So yes, you will be able to deliver faster with AI, but that doesn't mean that you should take on 10 times the work and just expect to operate as effectively. Especially a lot of these AI deliverables and things still need your review, still need you to look at, not only look at, but be responsible for the output and what comes out. And so you're going to be the bottleneck in all of that. So as long as you're able to operate quickly, then it'll work well for you. I think you need to find the level that you work at.

38:48Shamil Malachiyev: I think that's always been true. AI just allows you to do a lot more. So you need to be mindful, especially people pleasers. I'm notoriously a people pleaser. I don't like saying no to different teams. I like to help people out. You need to just be mindful of what's organizationally the most important thing to work on. And use AI for that too. Set up agents to help you manage your workload and all the inputs you have coming in and what you should be working on. And if something new comes in, have it help you balance against your current workload. The tools aren't just for development, they're really for making your life at least more organized.

39:25I don't want to say easier, but at least to have some more organization around what you're doing.

39:30Shamil Malachiyev: And maybe we can talk a bit about, because doing a project with an AI and creating an agent are slightly different things. And how would you advise, like, what is the smallest kind of agent that anybody can go out there and build for themselves? Yeah. And what are the things they should be thinking about? Sure. So for a long time, I did not think that the average person really needed agents. per se, I thought most of them could get along with prompting into an LLM, working back and forth and getting what they needed. Recently, there's been releases, ChatGPT released Dots, Meta released Muse, Grok has GrokBot of these almost personalized AI agents.

40:17And there are some really incredible things they're able to do depending on the level of information you want to give it. So it's always what information you want to give these companies And for meta, they've probably had your information for 20 years anyway. So it's not that much more. But being able to integrate not just into your calendar and email, but like to have it book things for you or to schedule things or do other things. And so it's just all the level of comfort that you have. I don't think the average person wants to do that. But I think that, you know, especially everyone has that chore that they're putting off like, oh, I need to reschedule that appointment or I need to call the handy person to come over, whatever it may be.

40:55If you could have an agent do that, not that you don't care about how the communication looks, you could review it and all that. But you don't need to be the one to send it yourself. That's a nice little thing to be able to do. I think typically, though, most people still don't need agents for using AI. I think just to be able to set up projects in ChatGPT or Claude, keep your contacts together, have a shared set of instructions and sources and just work from that. It's probably more than enough for most people. But these rise of personal agents have certainly made things interesting recently.

41:31That's separate from organizationally. I think agents definitely have a place internal to organizations where they could help with a lot of automation running different processes. But again, you have to be extremely mindful of the boundaries that they operate in, what they're able to access. I always skeptical about giving agents write access versus just read access. So they can't write things back and overwrite something and mess something up. So just need to be mindful about that. But yeah, no, agents are going to be, they're starting to be pretty fun personally. And I think a lot more is going to come with that.

42:05Shamil Malachiyev: Because I think a lot of people are starting to trust agents with their cards to make purchases. And the question in my mind is how long until people start trusting access to their 1Password accounts so they can access anything to perform on your behalf. And I think everybody starts in this, like, oh, we're not going to give you the right permission. Only create drafts for me. Then you work for a couple of months and you're like, I can't be bothered anymore. Okay. Well, it's where do you, where do you want to be the bottleneck? Where do you want to spend your time? I mean, if you need to review everything that only scales so far versus if you want to scale beyond that, then I think where you are is the bottleneck changes.

42:50So instead of human review on every output, you then set up tests and evals where you're measuring the output, you're measuring the quality of the output and how that fluctuates over time. And then if it breaks a certain barrier or trips a certain measurement, then you can jump in and review that. And you're kind of managing the fleet of agents instead of just output on an individual level. But again, it depends what it's doing. It depends where it's going. I mean, probably don't want it to generate a summary or a presentation for your CEO and have it send it without you looking at it. So there's a lot of just consideration about how you set it up.

43:30Shamil Malachiyev: I had a couple of stories. like when i just started i think the first thing was some like paper clips um something that i started playing around with which was similar to like open claw and i gave it some tasks to work on over the night and then i woke up to see a lot of drafts to a lot of uh clients of my company with like re-evaluating the the business proposals and i'm like oh my god so no none of that was didn't go out yep no that was that was the date that i switched it off because i i got scared by like how easily it could have been sent out as well yeah oh absolutely yeah that was one setting right there that if you had just left it unchecked or said something and a lot of this is also up to how the systems interpret your instructions so if you had said it slightly differently it might have interpreted it as oh go ahead send it out and so it could have been a very different story when you woke up that morning and i think like cloud code owners are also saying that like with every couple of months for us to review the initial like prompts uh the instructions the guardrails because as models are becoming more capable they also get better at avoiding prompt injections or understanding what you're trying to do without you having to be like oh imagine you are a software engineer it's like it's not 2025 anymore you know so that that is true and they're definitely better where you you don't need to tell it to imagine to be certain people or influencers or whatever you may want to do i still find um you know with claude especially it likes to tell me i'm done before i think i'm done so it likes to be like okay we already spoke about this you're you don't need to revise this i'm like no i i that's why i'm talking to you i want to revise this i want to go back and forth.

45:24So sometimes even though the models are better, that still takes some prompting or coaxing versus ChatGPT, which is usually like your best friends, all gung-ho, like everything you do is amazing. Every idea you have is the best and like is super supportive. And you're like, no, I need the pushback. I need you to be more critical. So I think the models have different, I don't want to say personality traits, but just different ways of dealing with them, even as they have gotten better. And maybe you remember the past two months,

45:54Shamil Malachiyev: there was a lot of, I don't know how with other models, but with Claude, it used to like, whenever you have a session at around like 10 p.m., it would send you to sleep. It'd be like, okay, good night. See you tomorrow, go to sleep. Right when you were in the middle of something that you were trying to get done for the next day or something important. Yeah, well, it's like when you're on a PC and just decides to start updating and restart on you in the middle of your Zoom conversation or something like that. Yeah, I think companies are getting a little bit better about making sure people can work when they want.

46:27But again, if you have a personal plan, you're gonna run into credit limits. So you might be cranking away on something and all of a sudden you hit your weekly limit and do you wanna pay$100 more to keep going or do you wanna just wait for it to reset? And so I think at first those limits weren't there And so people could just plug away all day long and, you know, really was unlimited. And now that they want to use the features and they're gated and you have to use tokens, it stifles the innovation a little. It's harder to, you know, just be as open and push forward when you're worried about running out of credits.

47:07Shamil Malachiyev: And what I've noticed is a lot of people who are starting to vibe code. like first thing they do is they start just develop a lot of different stuff with a lot of features with everything yeah without understanding like that you still need to take responsibility polish things remove 98 every time but at the same time a lot of them are not realizing how how credible their software could be and i guess a lot of people are facing with have some sort of like imposter syndrome like ah i'm not a software developer what do i know and like this cannot work in production and i guess you have that with people on your team like people who are not software developers who you're trying to get to use ai um what is your usual approach to get people kind of calmer on the topic yeah so well so like i said before not everything that you create needs to go into production.

48:05And I think that right there, that should take away the concern that people have about, oh, I can't get something to production. It's like, create something and then worry later if it should go to production. You're going to create 10 apps really quickly and maybe one of them, maybe none of them are worth pushing forward. I also see a lot of people and everyone has had ideas and I think AI has allowed people to take some of these ideas that they've had in their heads and try to develop on it. Then they get to a point where it actually needs to turn into a business or they need to sell it or actually go out there with somebody and they fall apart.

48:41They just stop. And so I think the ideas and that initial iteration are easy. I think it's the execution beyond that. So yes, if you want to put an application out for the public, you need to make sure it's secure. And And I mean, Claw, Anthropic, OpenAI, Replit, they all have guides on securing your applications in production. Like you should be looking at that stuff. You should be running the prompts to make sure that there aren't different issues, that your site can't be taken down, that people won't be able to steal all the information off your site, whatever it may be. But, you know, determine if it should get there in the first place.

49:16Like, don't be worried about that end state when you don't even have an application that people want in the first place. You know, so I think that's the hardest thing. It's like, not every idea is good. You need people to want to buy your idea or buy into it at least for it to really then need to go to the next step. So, you know, do it in steps, start out, see where you go. And if you find yourself in a place where people want your idea, then that's a different conversation. And then you can move to that next step.

49:44Shamil Malachiyev: And what would you say are the main things that people usually get wrong when it comes to building a product? um well and that i think it's not just for building ai i think just building anything in general it's what do people want to purchase and use and so so many times people think they have an incredible idea and it might be something relevant for them but if there's not a market for it if there's not a use case that people will buy into it doesn't matter how incredible your tool is or what it's capable of doing. And you could spend months and develop something that's incredible and no one could use it.

50:24Or what we're seeing a lot currently is people develop pretty amazing AI tools and then Anthropic or OpenAI release their own version and then you don't need the smaller company's version anymore. So that's something to be mindful of on a larger scale, I think. But yeah, for most people, just be mindful about what you're building and how and just try to start. I think it is easy to have things be broken in your application. And so that's definitely something, especially when people start out, they may publish and take down their entire application. They might delete their database behind it. They're going to make some mistakes.

51:08There's going to be some growing pains for sure. So just work through that like you're working through any other tool. You know, put as much security and conditions around what you're doing, start small and build out from there. And as you're working with your tools, be mindful of different security performance functionality concerns. Probably up front, not as much as the security, as much as the performance and functionality, actually having it able to do what it's supposed to, when it's supposed to be responsive and all of those pieces. so yeah just be mindful that you're developing something you want other people to use if it's just for you if you're developing something for yourself who cares what it looks like or how it works or how it operates do whatever you want like but as long as you know it's within the realm of legal boundaries like do you know just code up something and use it for yourself it's it's when you want it to share it with other people that's when you have to bring these

52:01Shamil Malachiyev: decisions then and in order to make sure that the product has the highest chance of moving through the legal compliance security procurement what are some of the important bits to pay attention to and prepare for those conversations yeah yeah um so i think the legal and compliance part comes later if you're trying to sell an absolutely important like we were talking about you need to build those into your processes i think up front these security pieces So who could access your site, how they can access it, what are they able to do on it? If you have administrative functions, make sure that only you're allowed to get in there.

52:39If you have a site with a lot of data behind it, is that data just open to everybody? Is it locked down? Making sure that they can't do SQL injection or anything else to mess up the data behind it or return things that you don't want. So it's really just as software development always had with production releases, going through those processes. And so I think it's not new. It's new in relation to AI, but the process itself isn't new. So there's plenty of existing great literature and videos and other things on how to productionize an app. And so with AI, you could work with the systems to help you implement these different safeguards and conditions and everything else and test it, try to break your app.

53:21You know, you could do things that it won't expect to see how it handles it, enter different things into different fields it doesn't expect and see what happens. and that's just user acceptance testing that's been around forever. So I think there's a lot of material to help you get to those next steps. And I think this has been the case the last 15, 20 years. People need to be able to leverage the resources out there to make themselves better. So AI lowers the boundary to entry for sure. But if you want to get better with your use of AI, look at the resources out there, see who's doing it, how they're doing it.

53:57You could ask people questions. Just the more you use something, the more comfortable you get with it, the better outcomes you're going to get from it.

54:05Shamil Malachiyev: And when picking people to join your team, for example, what are the things that you're looking for in those people which weren't there for like a year ago, for example? Yeah, no, I mean, it's really critical to have some AI literacy. And I really, at my level, I'm looking for people that have used AI successfully in their organizations. And it hasn't been out long. So that means that they need some, you know, current experience with that. But I think just, and I don't even care so much about which tools you're using, as long as you could talk to me about how you're using them, what issues and obstacles you're running into, and, you know, demonstrate that to me, I think being able to come in and, you know, you join a new company, there's always going to be new tools you have to learn and pick up.

54:51But if you understand how the tools operate, how an LLM will work with a coding tool, will work with your decision support model or whatever else you have in place, then it's more you don't need to teach the technology. Then you're just teaching the business. So then it's learning just more about the ins and outs of the business and how we operate and who our customers are and all of that, which takes time as it is. but it's so much better than having to also teach the technology because that's going to take a lot of your personal time to level up somebody else on that while you're trying to move your team forward.

55:25So I think, you know, especially as people are looking for roles in technology organizations, not just having used, you know, ChatGPT before, but having examples of where you've leveraged it for positive outcomes and being able to explain that is going to be ever

55:42Shamil Malachiyev: more important and are you more bearish or bullish on the ai future i yeah i i take the stay on the bearish side i try to look for the positives and i i try not to think about terminator and cybernet and uh whatever else but no i i think um it really gives people a lot of opportunity me that wasn't there before. I think it is going to shift some lower level skill work and knowledge work, you know, down the funnel a little bit. I think people, some people may have to adjust how they operate, but I think for the average worker, it really, if you use it intentionally and use it well and you measure it, it really could magnify and really change how and what you get done.

56:35And so we always talk about everything getting faster, but it's just the new speed of business. It's just everyone's going to have to be at that speed. And so it's who could stay on top of the craziness, who could, you know, oversee all of the different pieces and really, you know, see the forest for the trees, not just get focused on each individual piece, but how the organization's moving and how it's leveraging AI, where you're putting your investments and how you grow with that that's just going to be ever so more important. So the technology is only going to get better. I think the responsibility on leaders and organizations, especially a tech organization, but I think even non-tech organizations really have to be mindful about how they're using this technology to improve their organization, move it forward, reach your objectives, and revisiting that, revising and replanning every so often as these tools do improve and there's different opportunities that become available.

57:33Shamil Malachiyev: And is there anything that you would like for the viewers to get a glimpse into that maybe we haven't mentioned yet? Yeah. So I do think that there are a lot of people out there who are skeptical about using AI, not because they don't think they could use it. Maybe they've heard some bad news coverage. Maybe they don't know what they would use it for, I think everyone should just try it. Try it on something that you know, like I said, try it on something you're an expert in so you could push it and tell it it's not working right. But as, you know, with computer literacy in the 80s and, you know, internet web literacy in the 2000s, I mean, it's just important to know how to use certain things.

58:20And so as these tools are ingrained in society and they're not going away and, you know, they're going to be in more things. It is important for people just to generally know how they work and how to use them. And then for the average person, it really can accelerate what you're able to do. And so just be mindful. Don't start creating everything. Be smart. And again, once you create something, see if it's something that you want to share with others that people actually want to use. But there are so many use cases. So definitely start using it. With that said, it doesn't matter which tool you use.

58:55You're always going to see new tools coming out, new models, new features, new agents coming out all the time. And you don't need to feel like you have to use the latest tool all the time. Use what's free at first. If you hit the limits, maybe go to the$20 a month plan. I do recommend staying on monthly plans versus annual because these models do change. You don't want to be locked in for a year on any one of them, but just see how it works for you. and then you could see where you could go from there. I mean, I know people using it to help with organizing like their kids' sports schedules and just, you know, what they need to do around the house and things and just having a daily rundown of everything.

59:34And so there's a lot of personal use case. There's a lot of business use case. AI is not going anywhere. And so, you know, dip your toe in the water, see what you could do with it. And then at least be comfortable with the level that you're at. You don't need to go beyond that. You don't need to develop agents and get all this orchestration and everything else going, but be comfortable with it, understand it. And also that helps you understand, you know, what's really happening in the world and what's not and what you should, what's coming up and what you should and shouldn't be concerned about personally and things like that.

1:00:10There's a lot of apocalyptic views about AI that have. What's your view on all of that? Well, I think it's the same views that have been around with every major piece of technology. I was reading the other day about how in the 50s when the calculator came out, you know, people were sure that it was going to destroy everyone's math skills and no one was able to do any higher level math with this calculator. Obviously, we know it's now integrated into schools and everything we do. And it's just, you know, when the web came out, I know when online banking came out, initially, a lot of people were scared about doing any banking online.

1:00:46And now everybody's banking online. So I think it's natural up front to be a lot of concern. I think the AI companies, for a lot of different internal reasons, are stoking that fire. I think they want to get the government to do certain regulation. They want certain things to happen in certain ways. So they put out these crazy stories of, oh, this AI broke out of our system and got into some other system. It's like, did you give it access to the internet? Did you not give it instructions on what to do? That's going to happen. But that's, you know, so I think, you know, just being mindful of how things work and, you know, what companies are saying and why they might be saying is important.

1:01:26But yeah, I don't think, especially in the near term future, that any major issues are going to happen. I do think for certain people or most people, it's going to change how they do their job. For some people, it may mean they need to get a different type of job. But, you know, just as soon as you could get on top of that and learn the systems and how they work, that gives you a level up. And that's another thing I've seen is a lot of people are at that starting level still where, you know, earlier this year, I started seeing all of these different evolutions and products. And a lot of people did start moving along with that.

1:02:03You have people exactly where they were back in January, still using the same LLM, doing the same things and not seeing where it could go. And so I think if you're motivated and interested and learn just a little bit more, put yourself in there, do a little bit more with it, you put yourself ahead of 75 % of the average people because they're not doing that.

1:02:25Shamil Malachiyev: I think there is a lot of space for all those like educational kind of initiatives to take place. And I think another thing is it's hard not to embrace the learning with AI tools because one, they don't judge. They're never going to be like, oh, come on, seriously, you don't know that. They're always willing to help. Unless you ask it about your finances. If you connect your finances, you might say, what are you doing? Why does it look like this? What is this every single month? Different story. That's so true. And it also can teach you anything in the way that is convenient for you to learn. Yes.

1:03:01Shamil Malachiyev: Do you want a flashcard game? There you have it. Do you want to have a conversation? There you have it. And it is incredible for learning. I have chat GPT teach me different new AI topics about what's going on and summarize things and just the latest news. I have a few different agents, you know, just aggregating the latest news for me so I could see what's coming out, what's important. I think it's also important to kind of demystify a lot of that kind of technology. Because everybody was talking about MCPs, agents. And when you look under the hood, it's just a very long prompt with a couple of tools that are granted to the agent, right?

1:03:40Shamil Malachiyev: so i think for people to realize that it's not really that scary like it no no yeah no i mean just dip your foot in the water and see see what you could do um you know there there's levels of the information you could give it so be mindful of that but also at the same time be aware of all the information you've given all the companies out there already to this point so are you really protecting your information that much more something to be mindful of but But at the same time, you know, ChatGPT has plugins for finance and for health and other things where you could literally give it your whole life's worth of information if you want to.

1:04:18And I don't see how, unless you just don't use these tools, that you don't give out your information. So that's something that unfortunately, for better, for worse, people have had to become comfortable with. But yeah, no, I mean, just if you're comfortable with doing that, see what it could do for you. Have it send an email for you. have it cancel something for you or reschedule. And you'll see that it's pretty amazing what it's able to go off and do.

1:04:43Shamil Malachiyev: First time is always scary, you know, but it gets better. With any technology, right? The first time you use it, you don't know what you're doing. You don't know where to look. You feel like there's a million features you don't know how to use, but with more use over time, you get more comfortable and then they change things anyway. Then you look at it one day and you're like, this interface is totally different. Where do I go and how do I use it? And so I think companies are always, you know, updating the look and feel of things and what's in there. And especially OpenAI and Anthropic are always putting in new features and functionality and things.

1:05:13And so you don't feel like you're missing out if you're not using whatever you've heard about is the latest and greatest. I mean, last year, the ability to build GPTs in chat GPT was really huge. Everyone was like, oh, we need GPTs for all our things. And now it's been replaced with skills. Now it's been replaced with agents and who knows where it's going to go from there. And so that's the thing too. If you build a lot of infrastructure, it's very likely you might have to rebuild it in a few months. If things change, if you don't need things set up in the way that you had it previously. So it's a lot to stay on top of, um, leverage AI to stay on top of it the best you can.

1:05:47And, um, you know, everybody's just doing the best they can with it. I think that's the most important thing. Just do what you can do.

1:05:53Shamil Malachiyev: And if you need some inspiration, motivation, message, Paul. Yeah. Absolutely. Thanks so much for the conversation today. It's been a pleasure. Thanks so much, Jamil. It's been great to be here. Thank you so much.