How to make AI more responsible, with Navrina Singh

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards. Pioneers of AI is made possible with support from Inflection AI. It's not just enterprise AI. It's your enterprise AI.

1:00I've always been a renegade. I always was challenging the status quo, always at the front end of innovation. And when I saw this big hairy problem of responsible AI, I think it was this moment of reckoning. If there was a legacy to be left and if there was a moment, that moment is now. That moment was back in 2020. By then, Navrina Singh had already clocked over two decades as a computer engineer in big tech. at companies like Qualcomm and Microsoft. She had front seats to AI development, and she saw the writing on the wall about AI. Before a lot of us did, she knew that AI would transform our world.

1:43She also knew that in order for this transformation to be a positive one, we would need to build responsible and safe AI. There's too much at stake. Nobody was solving this problem. This was on the back of my mind for past 10 years prior to that. And I was like, if not now, we are going to be in a very messy world where not only our democracy, our society, our planet are going to be beholden to this very powerful technological transformation. And we as builders need to really own our responsibility. And you don't have to be really great to get started, but you have to start to be great. So Navrina did what she always does.

2:25She took action. She founded a company called Credo AI that's at the forefront of AI governance. They help their clients achieve responsible AI practices by doing things like ensuring compliance and adhering to AI regulation. But what does AI governance actually mean and why do we need it? Navrina Singh has the answers. On this episode, we talk about how we can build responsible AI through company and governmental policies. And if you work at a company that's using third-party AI software for research or hiring or really anything, this is a must-listen. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:24Hi, Navrina. Thanks for coming on the show. So good to see you. How are you doing? I'm fantastic. Well, thank you so much for having me. So today we're going to talk about AI governance. And the way I think about AI governance is basically it's a set of practices, frameworks, policies, tools that help us develop and deploy AI responsibly. But I wanted to hear kind of how do you think about AI governance and why do we need it in the first place? Absolutely, Rana. What a critical topic for the time that we are living through right now, by the way. So I think about AI governance as really the competitive advantage that companies need to not only build trust in artificial intelligence, but also to win in artificial intelligence.

4:09And I think your definition is spot on because it's not one thing. It is really putting the right structures, whether it is accountability structures, policies, frameworks, tooling to make sure that your AI investments are not only guarded with the right values, but you're extracting the maximum ROI from these investments. So, you know, I love giving this example. Think about artificial intelligence as one of your favorite fast cars, Bugatti or Ferrari, right? But AI governance is truly the brakes and also the pit crew that makes these fast cars not only operationalize and move faster, but also making sure that you're actually winning this race.

4:54So that's what AI governance is right now. A lot of the time I see companies think about governance, compliance, ethics after the fact. So you've built the product, you're about to ship it, and then somebody says, wait, did we think about the ethics of all of this? Or is it safe? And it's after the fact. And I guess one should really think about this from the get-go. Absolutely, Rana. You know, the companies and individuals who think about governance as an afterthought have honestly already lost in artificial intelligence because you can't bring the same software building mentality to AI. And the reason for that is not only the scale and impact, but also how quickly that impact can go wrong with unintended consequences if gone unchecked.

5:41And that's why governance really needs to be front and center to your AI strategy, especially as you start thinking about agentic AI. You're going to have all these agents that are floating around representing you and what you need to be doing and the objectives you need to be accomplishing. if they're not aligned to the right values and making sure they're not governed. Imagine what can go wrong. Yeah. I want to come back to the AI agents angle. But before we go there, you founded Credo in 2020. That was really early in the kind of AI governance landscapes. Nobody was talking about this beyond blogs.

6:17So what was the market and especially the investor response like? So first and foremost, there was no market in AI governance, you know. So it was we actually at Credo AI created and coined the term AI governance, made sure that was socialized at the highest levels, not only in the private sector, but also in the policy sector. I currently sit on President Biden's National AI Advisory Commission. And so really getting that opportunity to bring AI governance to policy and government ecosystem and then ensuring the analysts were brought into this category was, I would say, a big part of our focus in the early days.

6:54analysts, specifically business analysts, who use data to strategically understand a business, including its processes, products, and services. What do you mean by analysts being bought into this? When you have a technological transformation, many a times we start taking examples from the history to apply to the current future. And what ends up happening is when artificial intelligence started to take off back in 2018, 2019, there was a thesis that, oh, governance is not going to be important. That was first. And then the second was, if it is going to be important, we can use the same methodologies that we've used for other software technologies.

7:40And the challenge with that understanding and thinking, Rana, is artificial intelligence is a very dynamic technology. The static ways to govern cannot be applied to it. In addition, the speed at which AI transformation is happening, and we've seen examples of that in the past 18 months, there are new emergent properties, especially with large language models as well as multimodal systems that we have not seen before. So you can't really apply patterns from the past into the current AI technology. So we had to educate the analyst because of the dynamic nature and the unknown problems in artificial intelligence.

8:21So there was a lot of education needed to create this category. And then the past four and a half years have been really focused on not just talking about it, but actually operationalizing that through tools and products. And that's what we are building and delivering to the world right now. Credo AI customers have access to their AI governance platform. Think of it as a dashboard of all things responsible AI. Customers can use this platform to get risk assessments on things like bias, discrimination, or even cybersecurity threats in their AI tools. They can also use the platform to reach policy compliance and generate bespoke reports.

9:01So I love what you said about how AI is constantly evolving and developing, and whether it's the algorithms or multimodality, which essentially is going beyond text to including computer vision and maybe voice and other sensors and other kinds of data. and even AI agents, like you were saying, like these technologies that are going to act on our behalf, take on roles that traditionally were done by humans and now go off and execute them on our behalf. We need a new framework and a new approach to governing these systems. Can you go one level deeper? Like what does that actually mean? Agency is a very powerful capability and I would say one of the core characteristics humans have that we can reason and use our agency to our benefit.

9:46And one of the biggest things of making sure that that agency is guided in the right way is first step through alignment. So in governance, the first step is really aligning on what does good look like. And as you can imagine, that definition of good varies by individuals, by enterprises, by countries. So we are extensively focused on aligning an enterprise to what good looks like for them, aligning their technical and business stakeholders around what that good looks like to them is, and then making sure that it is operationalized. And as you can imagine right now, in terms of alignment of AI systems, you can align to standards like NIST risk management framework.

10:30You can align to standards like ISO. So the ISO standard basically defines what should a company do in terms of making sure it has the right oversight. Do you have a responsible AI policy or a code of responsible AI policy within your company? Do you have a chief ethics officer that is responsible for all of responsible AI within the company, etc.? You can align to regulations like EU AI Act, or you can align to your own guardrails for facial recognition or speech recognition systems.

11:05The NIST Risk Management Framework and ISO are government and non-government standards that help set guardrails for all kinds of industries. The EU AI Act is a wide-sweeping policy that will place guardrails on companies making and deploying AI. And this doesn't just affect European companies, but also companies, say, ones based in the U.S., that provide services to Europe. Like Navrina said, these guardrails can look like having an ethics team in your company, or they can look like data transparency. So customers know how these models are being trained. But for now, what you need to know is that these are some of the standards that Credo AI helps their customers meet.

11:50So there's different mechanisms to align, but I think the most important thing to know is one size does not fit all. and as a result of which really aligning on what your AI systems should be doing and serving to your consumers is a very important step of governance. Now, once you've aligned on that, you know, it's really important to go and check whether your AI systems and your processes are doing what you've decided in that alignment. And this is where a lot of interrogation and testing and evaluations of your data sets, models, and use cases come into picture. Okay, so companies need to be thinking about how they're aligning with industry standards.

12:31They also need to be establishing their own internal guidelines. On the ground, this translates into practical decisions, like choosing less biased AI hiring tools or evaluating how a generative AI model manages private company data. We'll get to that and more in a minute after a short break. Stay with us.

12:58If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's Teamwork Collection. It has planning in JIRA, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at atlassian.com slash teamchanger. That's A-T-L-A-S-S-I-A-N dot com slash teamchanger.

13:47before the break we defined ai governance and discussed some of the guardrails companies need to be thinking about when it comes to ai now let's dive into some concrete examples of what this actually looks like so i want to give a few examples so at the production company that produces this podcast, we actually have a listener's bill of rights in the age of AI. And one aspect of this bill of rights is that anytime we use AI as a voice, or even like if you're hearing or seeing an AI, like a clone of myself, for example, we are basically going to disclose that this is an AI versus a real person.

14:29So that's one example. And then at Affectiva, the company I co-founded, we were developing all these computer vision algorithms, say a smile detector or a drowsiness classifier. And we were just really committed to building this ethically and mitigating any data or algorithmic bias. So these are kind of two examples in my universe. Can you give an example of what AI governance and building this responsibly and safely would look like at, say, a bank? One classic example, which irrespective of the industry, is everyone's hiring, right? And when you're hiring, a lot of companies are dependent on third-party tools.

15:06And some of these third-party tools can come from, you know, Eightfold AI, Workday, name it. For something as classic as that, you can imagine that you as an organization need to decide how are you going to make sure that these systems are actually accepting diverse set of candidates that aligns with your company policies and might not be unintentionally leaving out a certain demographic just because they have not been trained on those demographics. So if you are an organization based in New York, New York actually passed a law about a year and a half back, which is called New York City Law No.

15:44144, that requires organizations that are buying third-party AI systems to do a fairness audit. So very classic example that we are seeing across all enterprises, especially if you're buying a third party HR system, which is based on machine learning, is you have to do a fairness audit. And so this is where governance really comes into place because you want a standardized way that all your teams are actually doing that analysis at scale. So say, you know, I'm a big organization and I'm using one of these AI tools for hiring. So would I then engage with Credo AI to help ensure that these tools are fair and kind of meet these regulations?

16:29Yes. You know, since we are in this interesting world of generative AI, a lot of our enterprise customers are making massively big bets in AI. And as you can imagine, us included, we are seeing really amazing productivity gains from using generative AI tools, whether it is for coding, whether it is for marketing, customer support. And as you can imagine, as a startup, that is really valuable. But one of the challenges that happens when you are bringing in these generative AI systems into your organization is, again, unintentionally, you are not thinking about the risks. So if there is a Gen AI system and you are providing them your own proprietary information, how are you making sure that they're not training that on your data that you are providing?

17:17Or second, you can think about data retention policies. I as a company might have a data retention policy that might be zero days, but maybe some of these Gen AI companies have 30 days, 60 days, et cetera. When an organization is buying third party Gen AI systems, we do a couple of things. right out of the box. We provide you risk profiles of many of the systems that our customers are using, and you can actually publicly see them on our website. And two is ongoing governance really depends upon the context of use. When you're using these large language models or multimodal systems for marketing versus search versus coding, the context of use matters.

18:00And the context is what we use to define the guardrails. And so we show you within that context how your systems are performing and whether they're not compliant. So I'm a member of the Young Presidents Organization. It's a YPO. It's a global network of about 40 ,000 CEOs globally. And just over the past year and a half, everybody's freaking out. All these CEOs are freaking out, trying to figure out how to incorporate AI into their businesses and do that in a safe way. What's your advice for these CEOs? How should they think about governance? Yeah. You know, first, before I go into the advice, I think this is, I'm glad they're freaking out, but I also would advise that they should not freak out because this is such a great opportunity.

18:46We are living through such an amazing time in artificial intelligence that we have one of the most powerful technology at our hands that we should be using to figure out how we are going to show up as leaders in this age of AI. So a couple of key things. The first and foremost is, if you ask these enterprise leaders, where is true artificial intelligence and gen AI actually being used in your organization? Most of them won't know. So taking stock of where AI and machine learning initiatives are within your organization is literally the first step. And also knowing where your data is, right? Kind of taking stock of your data.

19:26Most companies have no idea, but it's disparate. It's like all over the place. And you know, Rana, that's actually in governance. One of the shifts that we are seeing is absolutely governance requires you to have good data hygiene because good data means good AI. But a lot of, I would say, business impact is dependent on the actual application. The context matters, as I've been mentioning. So really understanding where AI is used. So you might have data sitting anywhere, but if you're not putting that to work, you honestly don't have much risk. But when you start putting that data to work through AI applications, that's what you should be paying a lot more attention to.

20:06So the first thing is really taking stock of where are you actively using AI applications within your organization, whether that is internal, let's say for hiring, for marketing, benefits, et cetera, or whether it is for external product creation. Once you've taken stock, the second thing is really aligning across your leadership on what does that good look like? Because AI governance is not a grassroots initiative. It needs to come from the top. And then the third thing is once you've aligned on what good looks like, that's when you start operationalizing and enforcing the standardized way to have that accountability across all your AI implementation.

20:45I love that you kind of said how this is, it's so important that this starts from the top. I always say the chief executive officer, the CEO is also the chief ethics officer. It has to be kind of a part of the values of a company. I'm curious, what are some of the biggest challenges you faced convincing companies to invest in AI governance? I would say it's the, I'll share the challenges, but it's the same across even the investor base. And I would say it's, you know, so it's a very interesting, everyone will raise their hand for innovation. But when you start talking about who wants to do it responsibly, everyone just lowers their hands.

21:24And then when you ask them that who is actually going to be held accountable for what you're putting out in the world through AI, everyone disappears, right? So I would say that we are going through a very similar motion in artificial intelligence, where the excitement around innovation is all time high. But the actual implementation of doing it responsibly and safely and understanding the risk is not there. So some of the, I would say, resistance first is they're like, it's not a priority for me right now. There's no regulations. And I think that's what we need to change very quickly because you don't do AI governance to meet a checkbox, which is regulation.

22:02You are doing AI governance to build that competitive advantage and to lead with trust because you're leading through transparency with your customers. And then the second thing that we hear quite a lot is it does not apply to me. And I think that's where it's like it does apply to you. Let's start taking stock of where artificial intelligence is being used with or without you knowing. Because in big enterprises, we have a lot of shadow AI use cases that employees, you know, they all are using AI tools. And I would say that if you're not using AI tools, you're already behind. So we encourage everyone to use these AI tools.

22:39But again, how do you do so with education and with responsibility and with governance tools like Credo AI? If you work at a company that uses or even makes AI tools, the gears are probably turning right now. If you haven't yet started your AI governance journey, now is a great time to do so. But AI goes beyond individual company responsibility. What is our government's role when it comes to AI regulation? We'll get to that after a short break.

23:15Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.

23:46It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger in our neighborhood. So we had to find a place. Moving from a home operation into a storefront was a huge next step. But Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak. As a small business, finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.

24:22You know, it just gave us that runway to be able to breathe a little bit. Then you get to focus on the cooking of the food and making the experience great. To learn more, go to CapitalOne.com slash business cards. So let's switch gears to the regulatory landscape. There's so much happening on this front, and you help companies engage with how to use AI safely, but do you also kind of hold their hands and help them navigate this regulatory landscape? And if so, how do you do that? Absolutely, Rana. You know, just like artificial intelligence is a new technology, the policy ecosystem is trying to evolve as fast as they can with this technology as well.

25:02We take on the burden for you to keep track of all the emerging regulation standards and best practices and then guide you through making sure that you can very responsibly apply that to your AI applications. We've been very focused from day one on not only providing the tools that operationalize the policies and regulations and standards, but also bring AI expertise to policymakers so that whatever they are proposing actually makes sense in these operations pipelines. So I spent last fall, I was in Belgium for a few weeks on my Eisenhower fellowship, and I met with several of the European Union AI Act legislators, which was really fascinating.

25:46What should we all know about the AI Act? And can you kind of repeat what's the timeline for implementing this regulation? Yeah, you know, so a little bit historical context, European Commission started working on this almost like five years ago. So there was a lot of... Yeah, which people don't realize that, right? Exactly. So, you know, I really say kudos to them for having put the amount of thought and effort into bringing multiple stakeholders to create EU AI Act as a first piece of legislation. I think what has been really exciting to see with the EU AI Act is they've brought in a very comprehensive risk-based approach and right-based approach.

26:27So rights are human rights. So they are really thinking about when these AI applications are going to be used in Europe, how are they going to ensure that it actually works for the European citizens? So marrying the rights-based approach with risk-based approach has been really exciting to see for EU AI Act. So just core things that everyone should know is EU AI Act has a certain risk categorizations, depending upon whether you are a developer of artificial intelligence system or you're a provider of artificial intelligence systems. And then based on where you fall in the risk profile, there are certain separate requirements that apply to you in terms of quality management, risk management requirements for those applications.