Can agents keep a secret? We asked 1Password’s CTO Nancy Wang

23:31Nancy Wang:And then finally audit. Right. And but as you can see here, right, it's no longer as clean as, hey, we like the platform owner, you know, owns these different substrates of storage, compute, networking. And, you know, the customer owns the data is really about like who owns sort of the underlying platform and who owns a runtime. Right. That I see as sort of the separation of responsibilities. Yeah. And when you start drawing the lines like that, too, it really becomes like understanding like the guardrails that you need to have in place as an individual developer, but then also the golden path systems you have to make to make sure that those movements between those stakeholders, the vendors that you use to build your product and your engineers that are building your product and delivering it, that there's this, you know, coexistence in terms of like your data policy, your security, and also making these decisions like in an instant, like you're talking about scaling and having huge amounts of interactions, inbound activity that you have to scan for.

24:31You have to like screen against suspicious activity while you're simultaneously trying to authenticate. And then there's the whole matter of like trying to actually like load balance this whole thing across the whole globe, right? So it's a large engineering challenge. But what makes it even harder is that you'd like make a almost like a snap decision. Like in a moment, you have to decide, does this agent delegated this person for this scoped task get this and you have to have a system that can have an instant response so it doesn't block things like oracle red bull who are trying to go fast and build things

25:00Nancy Wang:you know really build a wind tunnel you don't want to be blocked you don't want to be blocking the wind tunnel time optimization and so like because of that like it goes back to what you said at the beginning about making the golden path the easiest path making the golden path makes everything faster. But two, it just makes me think of like these gates that decide in a moment. How do you think about them in a determinism aspect in terms of a probabilistic aspect? Is this like an agent making a decision in real time? Probably not. It's probably a whole bunch of mechanical guardrails. Like how do you fall back on that idea?

25:34Nancy Wang:Yeah, I mean, that's absolutely right. In fact, one of the upcoming reports that our security team is going to publish is really like a harness that we've also built internally that, you know, our product security team was able to distill, I think, over 300 different, you know, security rules that we've used since the beginning of 1Password into an AI-assisted review program, or we call it our security harness, right? And why that's really important for like us on the engineering team is we're not sort of then waiting on tickets from the AppSec team that review code. Like, we're able to actually build that furnace into our code review process.

26:11Nancy Wang:And so, again, reduces that time to value. And that's really what we're focusing on. Wow. Okay. So security is a loop now and it has a harness and there's a flow that you can go through and there's steps that all those stakeholders can do at every stage of that circular flow to be good inputs for the next. And it becomes a virtuous system. It's kind of interesting how - Yeah, and self-learning as well. It's self-learning, self-healing. And then that perfectly becomes adaptive to these things like software factories that you mentioned a moment ago that it's a very trendy topic right now. Everyone is very obsessed with trying to figure out how do we create that kind of end-to-end loop.

26:47And, you know, I think understanding the determinism gates along the way and being able to split things into lanes and measure all of that activity is going to be the most, like, effective way to get, like, a grasp on it. Also just having good log practices. It goes back to just, like, good fundamentals. fundamentals we talk about that a lot on dev interrupted that like right now oh how can i just get really ahead in an agentic world well you know eat your wheaties like do your eat your breakfast do your homework like you need that you need to have clear communication process because like building software is a contact sport we got to communicate with each other and if you don't have the processes in place to facilitate that then you're not going to go much faster or much safer

27:25Nancy Wang:with agents exactly and i mean that's that's where you don't throw the old textbooks out the window Right now is exactly the right time to do zero trust. Now is exactly the right time to do identity verification. Right now is the right time to go make sure that you have golden copies of your sensitive data stored in like Fort Knox somewhere. So when you're ransom, you have, you know, golden copies to recover your business data from. So those definitely still exist. And which is why now maybe speaking from a business perspective is if you look at, for example, what companies have done really well, especially in this AI transformation age is actually still the companies like the infra companies, right?

28:07Nancy Wang:The middleware companies that we continue using like Twilio, for example. I mean, they've went, you know, crazy in terms of growth, right? The storage companies, because look, agents access file systems. I mean, as an infra nerd and builder, it makes me really happy that I guess I'm not super irrelevant in this day and age. A new hire's first mistake used to be doing something wrong with Git. Now it's an AI slot pull request merged straight into main from an untamed agent. But the accountability should be the same. You should own the code that you ship, whether you wrote it or generated it. As AI writes more of your code, it's more important than ever to have strong checks and balances in place.

28:46Without them, security risks and spec mismatches slip straight into production. LinearB provides policy-driven AI code review that catches risks and enforces your standards before human review even begins. Govern your AI workflows without slowing your team down. Learn more at LinearB.io. Yeah, agents want to use security tools. They want to be secure. People want to build secure workflows. workflows. And so it's only just going to become more fuel for more things that get built as people become more agentic. And, you know, speaking of everyone becoming more agentic and everyone's right now is figuring out how they can lead their engineering teams into that agentic engineering world.

29:25I want to take a moment also to revisit something like the builder hat you've been putting on lately. And it sounds really exciting. I know, too, that in the developer world of one password, you all just had so much tooling and releases that have come out very recently and you had a whole month hackathon as well where folks built a whole bunch of stuff on top of your platform. You know, I'm just curious from your perspective as like a CTO in that role, how do you focus on moving quickly, but also then preserving the brand around we are deep on privacy and trust. And yes, it's safe to build an experiment and be agentic, but you can be secure at the same time.

30:01I think leading with that message is it can be, you know, it can be really hard for folks to strike the right balance.

30:06Nancy Wang:Yeah, for sure. In fact, actually, you know, real data points is when we released our blog, and this was probably a few months ago, around using cursor agents to help us refactor the monolith. Well, that was really important because, you know, agents actually helped us probably refactor 50 to 60 percent faster by being able to actually crawl the entire code base much faster than a human could. And it actually made pretty good suggestions on which endpoints we could pull out first, right? And which sort of data models were kind of so entangled with those endpoints that as we were factored and were able to pull out separate microservices, you know, in what order should we pull out those microservices?

30:48Nancy Wang:And because, you know, their recommendations were actually pretty good because it was able to crawl the entire code base, you know, we've since now pulled out two or maybe three services now, endpoints completely out of the monolith, which has, you know, helped obviously free up headroom because as we get more customers, right, the number and the volume of queries per second also goes up. So we want to be able to serve that traffic in a good way, right, going back to, again, resiliency, availability, uptime signals. And so that obviously led to some conversation, of course, in the community of, well, is 1Password just letting our agents then or letting your agents see our credentials, right?

31:26Nancy Wang:And so that's where, you know, we had to do some explaining to do, which is, look, agents can't see your credentials. Why? Because we can't see your credentials. That's why it's zero knowledge, right? Your credentials, frankly, if, first of all, I can't log into your vault because it needs, it's an asymmetric key encryption, right? Only you have your private key. I might have your public key, but if I don't have your private key, it's still no op. And let's say, even if, right, we were able to get in or someone used your private key because they were on your device, what we see is actually encrypted blobs, right?

31:58Nancy Wang:And so that's the level a security guarantee that we do not compromise on no matter what feature we build, whether it's with humans manually writing code or with agents doing, you know, coding loops, right? So that's things that we will just not compromise on. Also things like cryptography, right? A lot of our cryptography is still very manual because again, this goes way above my knowledge here, right? But in terms of how we design the protocols, how we guarantee safety and security, right? We never want to compromise that. And so this also becomes a conversation around, well, where do you enable full agent loops?

32:35Nancy Wang:And so those naturally are going to be largely front-end heavy features, for example, because agents are a lot faster in building UI, right? Or things that require heavy security architecture, security models, many of that or most of that actually still require a human in the loop, right? So this is where we just have to be very deliberate around when and when not to use it. Right. So does that come down to like measuring and under or like rather like labeling and delineating that work? You mentioned like security reviews. And so having a system by which like the work that moves through our engineering org, we can identify it, maybe even tag it and then do different types of work depending on the level of human attention or mediated help that it needs.

33:20Is that kind of what you're working towards?

33:22Nancy Wang:Yeah. Right. So similar to how you would have like, you know, story points, right, for this is a harder to implement versus easier to implement. You know, there's also tagging to your point, classification involved. And largely, I think this is how I think about agents is, you know, they're best when you use them to automate deterministic toil, right? So things like, you know, where you can easily test success and you can easily do rollbacks, right? Those are great candidates. But, you know, when it comes to, for example, ambiguous judgment, things like building this policy engine, right, that does just-in-time determinations, keep the human in the loop, right?

33:56Nancy Wang:Anything that touches production systems, we definitely have to keep the human in the loop. We require a human operator to review and sign off on any changes that go into, for example, our core repo, right? Also understanding things like, you know, customer trust boundaries, because that's also really important for us. And anything that, you know, using Amazon terminology is a one-way door, Right. Once you go through, you can't easily come back. So it's really this, you know, kind of sliding scale of the more sensitive the decision or the system that you're taking the action on. Right. You have to just be way more explicit with like the spec and also the escalation path for a human.

34:34Yeah, this becomes a really powerful lens, too, for the playbook you gave us earlier, because by doing this labeling and understanding the level of attention that you need, then you're able to then a better route that work. But then also, too, when you look at your AI uplift, like we go back to those charts like, oh, we're having X number of more PRs. We're writing the X number more lines of code or whatever the case may be. And you see that lift, that trend in the graph. If you can map some of that to those kinds of recurring story point ideas that are small or very modular, easy to break apart and make repeatable.

35:08And you can trace that. Then that becomes like a really powerful way of being like, yes, this is a durable practice that we're building. Because like you just identified some really great candidates there. You know, we've talked a lot about on the show, too, about people transforming like really old architecture or doing mass migrations of like run times and stuff, especially on like really large platforms. And this is largely something that, like you said, can be heavily tested and rolled back and can be very systematically done. It's about creating just the system that can that can do it. It just needs someone to put in some initial investment.

35:40Right. And so creating those systems is what gives you that durable, long term lift. I think that's really, really insightful. And the opposite side of that coin, too, I'm curious about is like something that's really trendy right now, too. You talked about like software factories and stuff. Also companies going with like the idea of like token maxing, like just use as many tokens as possible, consume as much as through your API as possible. Like what is your thought on token maxing? Obviously, that runs antithesis to some of the things you've covered today. So like how do you prevent that kind of mentality within like an engineering org?

36:14Nancy Wang:Well, so first off, we're not putting caps yet on token spend. And I'm laughing because we have a Slack channel called AI Guild, where all the developers that are using our tools hang out. And I think one person just said, hey, am I allowed to use Fable now? I know it's very expensive. And I just thought to respond, but I'm like, you know, let the community maybe take this one. I also I don't want to be, you know, that person that comes in and, you know, turns the lights on at a disco party. Right. So it's OK. So I'll kind of let them figure it out. But you're welcome. AI board that I pulled her away right before she she dropped the message in and dropped in here.

36:53You know, exactly.

36:55Nancy Wang:But look, it comes back to, you know, the same thesis around you can't just measure productivity by how many PRs you generate. Right. Maybe you're generating like five two line PRs, maybe because it's a bunch of bug fixes or maybe you're generating like one, you know, half a million lines of code PR. Right. Because you're building this massive feature. So just like not every PR is created equal. Right. Not every token is being spent on equally useful things. And so this is where, you know, we do have a leaderboard internally of who are the top spenders. And so far, I mean, you know, I will manually pick out anomalies, but they also align with the engineers who are, you know, delivering the most work, right?

37:36Nancy Wang:Most features, most products, most platform changes. And so rough and tumble, it kind of matches out. Now, I think what's going to be really interesting, and this is something we're also building, is if you're able to tie, you know, spend, right, for these various AI tools to productivity, right? Because that's eventually not just what CTOs need to know, because we have budgets that we need to measure against or manage against, but also your CFO, right? So that they understand like, hey, where is all this money going, right? Because not only do you now have fixed costs in terms of human headcount, which is usually, was actually the biggest spend target, right?

38:15Nancy Wang:Now you also have variable spend or operating costs from the usage of AI tooling. And what's netting out, right? Are we in the positive? Are we not in the positive? Yeah, this is a huge part of all of the things we've been talking about today and as part of like what people would even use a tool like ours to try to figure out, right? Because you do have these variable and fixed costs. They have different levels of accuracy and throughput through your security, through your like your entire SDLC, right? And so because of those two factors, it's really complex to get a grapple on it. You need a place that centralizes like this is all of our API costs through cloud code.

38:50This is our subscription costs. These are our seats on cursor. And then for those same users, being able to understand like, oh, and then these are the PRs they're doing. Like you go back to your token maxing board and you talk about like the high performers and like, sure, they're at the top of the list, but they're also having huge impact. And so it's amazing to put them there and to identify what's working right. And I think that comes to to to understanding probably like if there is that uplift of AI adoption and it's durable and you're getting results and you're able to attribute those, you know, those story points we talked about earlier to to those durable downstream things.

39:27Then identifying, oh, did you create a system that did this at scale? Did you migrate something? What you find is that like, oh, that person on that leaderboard, they represent these large scale projects or automations or things that they've managed to architect. And so now the learning opportunity becomes, how do I share this expertise with the whole organization? How do I turn this thousand X developer into a 10X everybody is what we've been saying around here. And so distributing those gains. I'm curious to like how you think about using those signals to drive the change.

40:01Nancy Wang:Yeah, I mean, this goes into actually some of the updates that we've made in our interview loop, for example, what we usually test for. we no longer do coding tests is actually agent builder test, actually, where I ask to see your prompts, your test cases, and I progressively ask for harder test cases. Right. And this goes back to, you know, really good engineers. Right. The best ones that I know are really systems thinkers. And this is something that was really drilled into us, whether and I actually worked at Google as well prior to AWS, also building infrastructure systems. Right. This is where that comes back into the picture in tenfold, right?

40:41Nancy Wang:Because things like state management, right? State machines, be able to retry, retry logic, failbacks, right? Agents are just workflows, right? That reason, right? In between different steps. So they don't take always the next step after, you know, it's not a sort of dependable, you know, four-while loop, right? So this is where, you know, you have to have strong specs, right? You have to have strong tests and, you know, be able to be literate around evals. I find that's probably the still, you know, maybe the hardest hurdle for many engineers who are in that sort of AI transformation process is understanding what is an eval, first of all, and also how do I write good evals so that my agents that I'm actually running in production, I know if they're working or not working.

41:27I love this interview format, this idea of flipping almost like the coding exam and realizing that like what we used to stare at and focus at is that the code creation during those whatever time period and evaluate how it was done, how they thought through it. Now it's like, you cut that out. What's around it? How do they think about the planning and then actually lining up the systems to be done? How do they define what was good? What were the constraints? And actually understand what would success versus failure look like and eliminate those things before even starting to build. And then being able to meet the test that we put to the fire a little bit of like, how well do you understand this system that you put together?

42:09Something you're proposing, which by the way, within an engineering org, when you're building and proposing systems within a product, like there's a lot at stakes. You need to understand the stakeholders and the downstream impacts and it's the stakes are much higher. So being able to screen for that level of not only taste and execution on the systems thinking level, but also just like the impact thinking as an engineer, because I think engineers right now, it's like the biggest challenge is how do I get as close as possible to the customer problem and how do I get literate in what my customers or what my users are most deeply concerned about?

42:42And traditionally, all of the barriers between them and that conversation were so, so much. You had the CSMs and the account execs and the product marketing team and the product team and then everyone else before you would start to think like the engineers and getting them close to that problem. But now all of the people in between have some element of engineering and engineering has extended all the way into all of those organizations. The engineer actually has the ability to walk all the way up to the customer, look the customer right in the face, be part of the conversation. And I think that's so exciting for builders and being able to screen for the builders that are not only not only see that opportunity and want to walk all the way up to that conversation and be a be a contributor and sit at the table.

43:20But then also think, well, how do I build things that are going to make an amazing experience for this user? I just want to protect them and keep them safe. Right.

43:28Nancy Wang:And that's actually, you know, maybe a great segue to share one of actually the more exciting experiments that we're running this quarter is actually who is a builder, right, within one password. And sure, you know, the usual suspects of engineering and you have some, you know, technical PMs and designers who are also builders. But now for the first time, we're actually expanding that designation of essentially who's able to write code and then check in code to customer support reps. And the reason why that is so important is, you know, as a we also have a fairly large consumer business where our consumers, you know, they're using our browser extension fairly daily, our desktop app.

44:07Nancy Wang:And for many consumer users are also developers or hobbyist developers. So they're using our CLI, our SDK, and they're the first ones to report if something is flaky or buggy or, you know, the modal might be off on the UI or whatnot. And so when CSMs get that, typically it gets routed into engineering. Engineering has to look at the backlog and say, okay, we can maybe take care of it in this sprint or in this other sprint. And that's a pretty long time, right? Versus if we empower the CSMs with coding agents, right, like the ones we just talked about, and also build them a safe factory in which they can run their own tests, set up their own, you know, dev environments, right, to run their code through, right?

44:47Nancy Wang:And we can enable them to ship code, right? That's going to be the holy grail of unlock for all of the customer support tickets that we get in. Yeah, that's a huge leverage. Actually, that's such an amazing point. Very recently, we had a new segment on the show with one of our friends of the show, Kelly Vonder. And she wrote an article recently about that exact thing, about how you can create this safe environment that can prevent a backlog from even happening. And that too, it can empower these CSMs, these folks that are close to these customer pain points that are always having problems hearing things directly from the customer, want to be able to directly enable and fix them to be able to actually make that change without having to fight for space on a roadmap.

45:29It's like fundamentally orthogonal to what they're trying to solve. So now they can finally get to the heart of it. I think that's like one of the many unlocks that engineering leaders can really find within their org now. And, you know, I got to say, Nancy, you've given us like a lot of, I've said, playbooks, recipes, menus, like all like routes, like all sorts of ways to think about navigating this world. And I just really want to thank you so much for joining us and sharing your security minded perspective. And I think it's a really helpful reminder for all of us that you can not compromise on your values and what makes your product durable and safe for others, but you can still be agentic and you can still be on the cutting edge.

46:05And there's so much that's coming out of One Password right now. just as we wrap up, where can our audience go to learn more about like the latest and what's coming next?

46:13Nancy Wang:Yeah, check us out on the website. And especially 1password.dev is our new dev website. So if you want to understand, you know, what's new and hot, right, coming from 1password as a builder, as a developer, you know, check us out. We're also going to be doing a lot more sort of demo style videos from our technical marketing team. So you can see, for example, one, you know, that I just reviewed is how do we discover some of the secrets that are lying on disk, right, through the app and be able to vault them or protect them with one password vaults. And so that's an example of something that, you know, it's going to be, I'm really excited for it to come out and be truly game changing.

46:52Yeah, that's awesome. So we'll make sure those get in our show notes. Our listeners can go and follow those. And to those listening, you know, if you've enjoyed our conversation, please come and find Nancy and I on LinkedIn or Substack. where this newsletter accompanying this podcast is also published. And if you made it this far, you clearly liked it. So give us a like, maybe subscribe, listen to the next one as well, give us a review. But more importantly, come and find us and join the conversation. I think we're all better builders right now if we can share best practices and build more openly.

47:20So thanks again for joining us. And Nancy, I really appreciate, again, joining us to chat on the show. It was a ton of fun and I can't wait to have you back sometime.

47:28Nancy Wang:Thanks so much for having me, Andrew. bio