Goldman Sachs CIO on How the Bank Is Actually Using AI

27:21Every data needs to be protected. Therefore, we use those safeties all throughout, regardless. But then for the super, super, super secret stuff, you know, we like to do it in our own environment. Since you're talking about building your own environment, and this is something we've talked a lot about on the podcast, hardware constraints, energy constraints, things like that. How does that manifest in your world, some of these physical, real-world constraints to building out the compute platform at Goldman Sachs? Well, initially we thought maybe we can host those GPUs in our own data centers. And then immediately you round into considerations such as, A, first of all, they develop a lot of heat.

28:02Secondly, they consume a lot of power. Three, there is a decent chance that they might fail because of all those considerations if they're not properly addressed. And then D, they need very special, for example, interconnects and high speed bandwidth between them. And so the decision, what we ended up doing is actually to have them hosted into some of the hyperscalers that we use, but use them in their own virtual private cloud. So those racks are basically only ours. And if you're asking me the more general question, which is, hey, where is the world going with regards of that? Okay. So right now there are two really rapidly competing forces.

28:42One is pushing towards more and more consumption. and one is pushing for more and more optimization. Okay. And I can talk about that for a couple of minutes. For the more consumption, I mean, really the two dimensions for scaling a model is one of the most important one is obviously the size of the prompt or the context. Okay. And there is pretty good evidence that the larger the context, which is really like the memory of those models, and the more you can get out in terms of the ability to reason on your data. data has already gone up from thousands to tens of thousands to now millions. And there is a prediction.

29:17You heard some very prominent people saying that there could be the trillion prompt. And the power scales quadratically with the prompt. And so that points to a consumption of energy and GPU power, which is going to continue to raise exponentially. At the same time, we've seen great results with optimization techniques, such as quantization, reducing from 16-bit to 8-bit to 4-bit precision, having even smaller models using what's called windowed attention, which means that you know that you can only pay more attention to some of the parts of the context instead of all of it. And so maybe you need a smaller one.

29:54And so I'm seeing those two kind of going into two opposite directions. It's going to be very interesting to see how that evolves. I would say for the short term, I see that definitely that trend is going to continue to go up. And one of the things that fascinates me the most is that from one version to another, the most striking difference is the ability to reason and the ability to actually come up with logical step-by-step instructions or step-by-step chains of thought of what the output is going to be. So we decided, okay, first of all, we need to get access to the most powerful GPUs. Secondly, we need to host them into an environment that actually allows for the most optimal functioning in terms of bandwidth, in terms of power consumption, et cetera.

30:41And then at the same time, we've been focusing a lot on optimizing the algorithm so that we could really get the most out of that. Just to press you on this point, what are the conversations actually like with cloud providers at the moment when you're trying to get more compute or more space, more racks, whatever? Is it maybe different for you because you were at AWS? Maybe you can just call someone up there and be like, we would like some more space, some more servers. Or have you found yourselves at times maybe limited in what you can do by the amount of power available to you? Well, I wish that would be the case, but I cannot just pick up the phone and get whatever I want.

31:20I think so far, I mean, obviously because we are a really good client of those companies in general, but also because we've been very selective in the use cases that we put in production. I have to say, like I said before, think about that. If you look at the consumption of resources today, those who consume more resources are people that actually do the training of their own models. Okay. And initially everybody was trying to do full training from scratch, which was taking like the absolute, if that's 100, we do fine tuning, which is adaptation of existing models. That could be a one to 100 or less in terms of consumption or resources.

31:58So because of the techniques they were using and because of the fact that we decided to really focus on fine tuning or rag versus full training, we haven't really hit any caps. And also I have to be honest, We bought our GPUs pretty well early, so probably there wasn't as much craziness as there is today. So that's turned out probably to be a good idea.

32:40some recurring crypto buys, maybe even a few strategic option plays on the side. The point is, you're engaged with your investments, and Public gets that. That's why they built an investing platform for those who take it seriously. On Public, you can put together a multi-asset portfolio for the long haul. Stocks, bonds, options, crypto, it's all there. Plus an industry-leading 3.6 % APY, high-yield cash account. Switch to the platform built for those who take investing seriously. Go to public.com slash market and earn an uncapped 1 % bonus when you transfer your portfolio. That's public.com slash market.

33:15Paid for by Public Investing. All investing involves the risk of loss, including loss of principal. Brokered services for U.S.-listed registered securities, options, and bonds in a self-directed account are offered by Public Investing, Inc., member FINRA, and SIPC. Crypto trading provided by ZeroHash. Complete disclosures available at public.com slash disclosures. In business, a gift does more than say thank you. It reinforces relationships, celebrates milestones, and reflects what your brand stands for. With 4imprint, that message lands with certainty. 4imprint offers thousands of high-quality, customizable products from apparel and drinkware, bags to carry your logo, smart tech, and thoughtfully chosen office items.

33:54So your brand shows up with the kind of care and professionalism people remember. You can personalize every detail, your logo, your message, your presentation. And with thousands without a setup fee, it's easy to create impact at scale without stretching your budget. 4imprint's team also provides expert support and dependable service, backed by their 360-degree guarantee, so you can be 4imprint certain your gift arrives on time, exactly as expected, and fully on brand. Because when the gesture is intentional and the impression matters, the right gift makes your message clear and leaves a lasting impact.

34:27Explore gifting with confidence at 4imprint.com. You know, NVIDIA is huge. Everyone would like to have some of NVIDIA's market cap be their market cap. I have offering some cheaper product. We interviewed some guys who have a semiconductor startup that's just going to be LLM-focused startups. We know that Google, for example, has TPUs, their own chips. Can you envision as a roadmap some alternative where GPUs are not the dominant hardware for AI? Well, that's literally like the trillion dollar question. Yeah, well, that's what I'm asking you. Yeah, but I'm not an analyst and I'm just a technology.

35:06Remember, I'm the guy that makes sure that the print is going on. I would say you're probably a better person to ask than an analyst because you're actually the one who's going to be making buying decisions. Okay, so then I'm going to answer to you. So you have to distinguish between, there are actually two dimensions that we need to consider. One is training and the other one is inference. Okay, that's the first dichotomy. For training, at the moment, there's most likely nothing better than GPUs, okay? Okay. Because when you train a model, the software or PyTorch or whatever framework needs to see all your GPUs as one, as a cluster.

35:40And it's not just the GPU itself, but what NVIDIA has been doing a great job at is actually to make them work in unison with the virtualization software called CUDA, which runs on NVIDIA GPUs, which is a pretty extraordinary piece of software. And it became the standard for that. And also because, you know, the performance premium that you have on those GPUs, when you're trying to train those incredibly large models, is something that you really, really want. And so the training part, I'm pretty sure that is going to be dominated by GPUs for a while. But then, you know, as those models get used, obviously the pendulum swings towards inference, which is the actual, now you have a model, which is a bunch of weights and you just need to calculate a bunch of matrix multiplications.

36:25On that, I think accelerators and specialized chips are actually going to have a really big role to play. So you may imagine that you go from a world where everybody builds the cars and not too many people drive the cars to a world where most people are going to drive cars. And then there is another two dimensions, which is models that are hosted by the client and models that are hosted by a hyperscaler. So as you know today, I can take a model like Lama, I can put it in my own environment. You can run it on a MacBook. I can run it on a MacBook or I can run it in my own data center and with my own GPUs.

37:02And given that I'm used to GPUs, given that those are the ones that we can buy, given that CUDA is what developers know, et cetera, I'm most likely going to use that. That's a good part for NVIDIA for that. But then there is another way to use those models, which is to have someone host them for me and I just access them through an API. That's what services like Amazon Bedrock does. You basically choose your own model and then you serve it through them. When you do that, you don't really know what's underneath. You don't know if it's a GPU or if it is an accelerator, if it is Amazon's own chips or Google's own chips, et cetera.

37:37And so now the real question, that's why the trillion dollar question is, are most people going to use those models through hosted environments where the hyperscaler will have a lot of freedom with regards to what they use underneath? And most likely they will vertically integrate, or are they going to use them themselves in a more like in a self-service way? And in that case, it's less likely that those accelerators are going to dominate. We currently are in sort of a balanced way because we have our own that we use, like I described, and also we use the hosted models. And so where is this going to go?

38:15It's hard to say because I think it depends on the evolution of the models and it depends which models are going to be made available as an open source that you can actually host yourself. And I think right now, one of the greatest questions is, are the open source models are going to be an absolutely on par alternative to the hosted model, to the foundational proprietary models. And that, given LAMA 3.1, that answer seems to be more likely a yes. I had a question about this, actually, which is, do you think Wall Street's attitudes towards open source have changed over time? And the reason I ask is because nowadays, it seems like a fact of life.

38:54Everyone uses open source, whether you're a Goldman or somewhere else. But I remember, you know, like back in as recently as like 2012, I remember Deutsche Bank had like this open source project called the Lodestone Foundation where they were like, oh, we should all stop wasting our own resources, developing our own code and our own software. We should all pool our resources together and do open source. And they had to actually lobby. It was unsuccessful, ultimately, but they were trying to get all the banks on Wall Street to work together for open source. Nowadays, it seems like there's been this significant cultural shift.

39:31It's not even a question. So in general, my direction, my guidance to my team is don't build anything unless you have to. Don't think that just because you're a smart person, you can build software better than anybody else. Maybe you can, but it's a good thing that we focus on building things that are actually differentiating for us. And then I think the use of open source software, which we very much endorse, is also a really good hedge with regards to which vendors to use because it really heavily reduces the vendor lock-in. Of course, open source software, as you know, is a tremendous long tail.

40:11There's millions of that. And so I think there are best practices around the use of open source. And those best practices are, you know, like, you know, you need to run reviews on open source or tech risk reviews or security reviews or anything as if almost you built it yourself. And then secondly, tending to concentrate on the larger, very well supported by the community type of open source. And so my philosophy is yes to open source, but then you need to own it in the truest way because you are actually going to be generally the one that actually needs to support that. And so really building knowledge around that.

40:47But now you can ask AI to run the code for you and check it for bugs. Yeah. So, okay. That, of course, leads to probably what, if you ask everybody, where did you get so far the biggest bang for the buck for AI, most CIOs are going to tell you on developer productivity. And I think it's something that for us was the first project that we actually expanded at scale. I have to say that today, virtually every developer in Goldman Sachs is equipped with generative AI coding tools. and we have 12 ,000 of them. So we didn't enable yet the ones that are using our own proprietary language called slang, but everybody else has an AI tool.

41:23And the results have been pretty extraordinary. How do you measure that? What are some numbers or how would you describe the results? So we measure it according to a number of metrics, such as the time that it takes from, let's say when you start the sprint to when you actually commit the code or when you complete your task. We measure it by number of commits, meaning how many times you actually put code into production. We measure it by number of defects, which in this case is like, for example, deployment related errors. So there are more like velocity and quality metrics at the same time. We have seen a wide range ranging from 10 to 40 % productivity increase.

42:02I would say that today we are probably on average seeing 20%. Now, developers don't spend 100 % of their time coding. They maybe spend 50 % of their time coding. So your question is, what are they doing with half of their time? Where? There is a lot of other activities such as documenting code, such as doing deployment, doing deployment scripts, doing a bunch of tests, et cetera, et cetera. So what's called generally the software development life cycle. And so we see net of 10%, but then the cool thing is that those AIs and the things that we're building around that are starting to go beyond coding.

42:40They're starting to help you write the right tests, write the right documentation. They are even figuring out algorithms or even, for example, reducing or minimizing the likelihood of deployment issues, writing deployment scripts for you. So as that expands, we're going to be closer to 100%, and therefore we're going to be closer probably to 20%, which for an organization of our side is a pretty massive efficiency plan. Can I ask a question about hiring developers? So I've probably read 100 articles over the years about Wall Street competing with tech companies to hire developers like, oh, they got to have ping pong tables.

43:12Lloyd Blankfein used to say they're a technology company. Yeah, you got to have your ping pong tables and your free lunches and let people wear sneakers and all that stuff. But now it seems with AI, there is a number of people interested in it who are truly believing that within a few years, they might build the digital god that's 10 ,000 times smarter than any human and that they approach the task with messianic fervor. And I imagine if you're at Goldman and you're trying to help a banker answer a question to a client about something in the chemical industry, maybe that's not the thing that gets you out of bed, the way sort of metaphysical realms about what is the nature of consciousness and things like that, that AI people talk.

43:54Does that present any challenges or anything when trying to hire talented AI developers? I think developers love to solve real problems. And one of the things also that attracted me in the first place, not that it matters, but I'm saying, you know, I tell you my own personal experience is that working in a technology company is absolutely fantastic, but you're always like one step removed from the business or from the application. So I have to, you know, let's say you are the bank and I'm the technology company. I need to sell you a tool that then you're going to use to run your business or improve your business.

44:29We are kind of one degree of separation less. I were right there in a digital business that is fast, huge amounts of data, huge amounts of flows, immediate results, and that's kind of addictive. And so developers, especially when AIs are starting to do all those magical things that we're talking about, they can see the impact on the business right away. And that, I think, is kind of attracting a lot of people. In fact, there is more and more people that are moving into the industries, oil and gas, transportation, chemical, medical, finance, because this is new and there's nothing more exciting than seeing it in action.

45:08And so there is so much action going on that I think is actually really, really interesting. I think another question that maybe you haven't asked me, but it's kind of part of this question is what kind of developers and how is the profession of being a developer is actually changed? Oh, wait, I had a related question. It's not quite that question, but you can certainly answer that too. But okay, to my knowledge, Goldman Sachs doesn't have a job title specifically with the words prompt engineer in it. So looking at the impact of AI on your business overall, is AI a net hiring positive or a net hiring negative for Goldman's employees overall?

45:49Well, meaning are we going to hire more or less developers? Yeah, does it lead to more jobs because you're doing more things and productivity increases? Or does it lead to fewer jobs because now you can automate a bunch of stuff? Well, listen, there is so many things that we would like to do if we had more resources that I think this is going to be leading to more things that we can do. You know, some people tell me sometimes, so you're going to maybe hire less or have less developers. I don't know. I've been in IT, quote unquote, for like literally almost 40 years. And I've never, ever seen that go down.

46:21But I've seen inflection points where you can actually get developers to do way more and worry about way less that is not related to a business outcome. And so I think it's more like how the profession is going to change. In my opinion, we're going to be less low level and more, hey, I need to really understand the business problem. Hey, I really need to think outcome driven. Hey, I need to have a crisp mental model and I need to be able to describe it in words. So the profession is going to change. And there are tasks that I think are so repetitive that the automation of those is actually going to help developers really kind of feeling really, really connected with the business and with the strategy.

47:06And that will attract people that are generally curious, that are generally interested in understanding what we actually do. So the focus kind of shifts from the how to the what and to the why, which is really kind of at the heart, I think of this evolution of technology over the years from the back office of IT, which doesn't even know what you're doing, but as long as your monitor is actually working to, hey, I'm actually able to take a business problem and break it down into pieces that then even an AI can write code for. So to your specific question, I think this might maybe potentially for some companies are going to try to realize some of those efficiencies by curbing the growth or even sometimes reducing it.

47:46for companies like us that are extremely competitive, for companies that have lots of ambition, this is a race at the end of the day. And I think we're going to go for, you know, trying to get even more out of our developers and actually like, you know, trying to turn them more into something that makes them feel super, super connected to the business. What about non-developer roles, non-tech roles? And, you know, again, I guess a company like Goldman doesn't have, you know, probably a lot of like low-level customer support things or in a window is like, oh, I need to change my plane ticket, et cetera.

48:18But you know, a lot of modern work is essentially just answering somebody's basic question, are there roles within a bank that are going to either fundamentally change or go away due to sort of agentic or generative AI? I think a lot of the work that is about content production or content summarization will actually be streamlined quite a bit. Like, for example, taking an earnings report and making it into 10 different sources in order for different channels of distribution. Here's the one for internal people. Here's the one for the client. Here's the one for the website, et cetera, et cetera.

48:54Imagine the creation of pitch books for clients where you take templates, you put a bunch of data, you go out and do research, you take logos, you take this, you take that. There is a lot of that machinery and factory, which we have thousands of people doing that. I'm sure there's a lot of junior analysts who would be maybe glad to hear that some of making pitch books is going to be automated away. No, but I think that's a good thing. It takes away some of the toil. And so I think at the end of the day, listen, right now, have you noticed that everything is kind of converging to words and concepts?

49:25No matter if you're a developer, if you're a knowledge worker, those jobs are kind of colliding. And I'm absolutely – developers have seen that first. Why? Well, because it's a low-hanging fruit. the developers deal with the vocabulary that is not 50 ,000 words. It's like two, 300 keywords for a language. And so of course that works extremely well. And of course that's the first thing to go. But I think eventually the knowledge worker is going to be the one that is really benefiting no matter if you are a developer or if you are working on a pitch book or if you're working on a summarization of a meeting or the action items, or you're working on a strategy, et cetera, et cetera.

50:01And I think overall, this will elevate the quality of the work, which then everybody says a happy worker or a happy developer is a productive developer. I think you're happy when you're actually doing something that allows you to do your best work. And I'm hoping that if AI allows all of us to do more of our best work, I think it's going to be probably the biggest effect that we can have. I know we just have a couple more minutes. So one very quick question. What makes a good prompt? Well, believe it or not, empathy. You need to be empathic and you need to be gentle and you need to be kind and you need to kind of, you know, take.

50:38Just looking at me like I'm not empathetic in my prompts. I always say please and thank you. No, Tracy makes fun of me for how empathetic I am. No, I've said it's very sweet that you say please and thank you. You need to take the AI literally by the hand and take it where you want to go. And I tell you that, you know, one of my interesting, more interesting experience with prompts is the following. You know how hard it is to get an AI to say, I don't know. It's almost impossible. You're always going to get an answer. And so one time I decided I want to get it to the point. And so I had to navigate the prompt and the AI to understand that it was safe and okay to say, I don't know.

51:18And so then at the end, I prompted and said, what's the capital of the United States? Washington, D.C. Okay. And then I said, what's the weather going to be tomorrow? And I got an answer. And then I said, what's the weather going to be in a year? And it's simply, I don't know. And then at one point, I even decided what to say. It's like, is there a role for humans in a world of AI? I don't want to know the answer. Dot, dot, dot. Oh, God. Okay. Okay, well, everyone's going to be off on ChatGPT now trying to get it to say I don't know. Marco Argenti from Goldman Sachs, thank you so much. That was fantastic.

51:58Thank you, Joe. Thank you so much. Thank you so much.

52:12Joe, that was a lot of fun. And I have to say, I do not make fun of you for saying please and thank you to ChatGPT. I'm going to repeat it. I've said it's endearing. It's very sweet. And I've tried to follow your example. And I don't say thank you because I usually move on to the next question, but I do say please. I've heard this, though. It's funny that he said that because I actually have heard this, that there does seem to be quantitative evidence that words like please and thank you, etc., do actually improve. Really? Yeah. Mad Busegan, who we've known on Twitter forever, has posted about this.

52:47So So there's a good reason to do it besides just the habit, all entities you talk to, you should be in the habit of politely. Oh, yeah. That was your argument, right? Yeah, yeah. Yeah. Okay. Well, I thought that was fascinating. We've been talking a lot about AI and the sort of potential use cases and the chips that are driving the technology and things like that. But it was nice to hear from someone who's actually making the purchasing decisions and implementing them at a large institution. Absolutely. That was probably one of my favorite AI conversations we had for precisely that reason, because it was interesting hearing him talk about this idea that right now, like these open source models, particularly like the latest version of Llama, is getting really close to sort of the core proprietary models.

53:31That was striking. The fact that he sees perhaps particularly on the inference side of model usage, an opportunity for greater use of different types of hardware, also very interesting. That's right. And we're so used to talking about the massive amounts of power and energy that AI will consume. And we, you and I, have had a lot of conversations about how we're going to power all these servers and things. But what's gotten far less attention is just optimizing the way you use AI such that you don't need to consume as much power. So maybe doing less training, leaving training to the big like hyperscalers or whatever, and then just doing the inference.

54:11In the end, it's going to be both, right? Because in the end, like there's both is going to happen. People are going to find algorithmic techniques. And Marco described some of them to lessen the sort of pressure and stress that you're putting on your hardware. But of course, that's just going to mean you're going to use it more. And then also people are going to have to solve the power consumption side, kind of like all of economic history in general, in which we're always finding new ways to get more out of the same, you know, gigajoule of energy. energy, but also using more energy at the same time.

54:44Yeah, absolutely. Well, shall we leave it there? Let's leave it there. This has been another episode of the Odd Thoughts podcast. I'm Tracey Alloway. You can follow me at Tracey Alloway. And I'm Jill Weisenthal. You can follow me at The Stalwart. Follow our producers, Carmen Rodriguez at Carmen Erman, Dasho Bennett at Dashbot, and Kel Brooks at Kel Brooks. Thank you to our producer, Moses Andam. And for more Odd Lots content, go to bloomberg.com slash oddlots, where we have transcripts, a blog, and a newsletter. And you can chat about all of these topics in our Discord, where we even have an AI channel.

55:17Great stuff in there, discord.gg slash oddlots. And if you enjoy Odd Lots, if you like our continuing series of AI conversations, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is connect your Bloomberg account with Apple Podcasts. In order to do that, just find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

56:15We buy insurance for peace of mind, but every year millions of claims are denied. Not because people did anything wrong, but because their policies quietly excluded what happened. Insurers know every detail. Policyholders rarely do. That's why My Policy Advocate exists. For just 27 cents a day, their platform reads your policies and explains where you are vulnerable. They don't sell insurance. They deliver transparency. Before you trust your policy to protect you, let My Policy Advocate tell you what it really says. Go to MyPolicyAdvocate.com. In business, a thoughtful gift does more than say thank you.

56:50It recognizes achievement, builds loyalty, and shows someone they are genuinely valued. With 4imprint, you can choose from thousands of high-quality products, apparel, drinkware, tech, and more, designed to leave a lasting impression. And with expert support, dependable service, and their 360-degree guarantee, your gift will arrive exactly as intended, on time and on brand. Explore gifting with purpose at 4imprint.com. 4imprint, for certain.