Sponsored: Robo-Burp is coming for your web apps

0:03James Wilson: Hey everyone, I'm James Wilson and welcome to this Risky Business sponsored interview with Port Swigga's Head of Marketing, Kieron Hughes, and Product Marketing Manager, Andrzej Matykiewicz. In this interview, we're going to be talking about Burp AT. That is the agentic way to drive the well-known and very popular Burp Suite pen testing tool. Burp AT was in beta for a while, it's now out of beta and it was released at Black Hat but you know agenda can mean lots of different things and straight away I had so many questions is this an MCP? Everyone's got an MCP, is it a harness? Can I connect it to other things?

0:44James Wilson: Do I bring my own LLMs? So we started off there, we started off by getting a grounding in what exactly use Burp AT, how does it work, what is it, what is it not. And then Kieran and Anjay talk me through some of the things that they learned while it was in beta, some of the interesting stories that they've had from customers about how they've been able to use Burp AT. But then we zoom out also to ask, so what? What does it mean to have an autonomous mode for such a well-established pen testing tool? What does this mean for the industry? So I'll drop you here into the interview where Kieran starts off by giving us an introduction to exactly what Burp AT is and how it sits on top of Burp Suite.

1:25James Wilson: Enjoy. I like to think of it as like we've taken the testing tools that Burp users have trusted now for well over 20 years and given them an agentic core. So you can now give Burp and its agents an outcome, an objective that you're seeking. It can use the full context of the project and the engagement, and it can then leverage Burp's tools to then go and take the actions. So the benefit there is that it can really benefit from the methods and the skills that we've worked with our ports with a research team on and obviously James Kettle is a part of that. And you retain full kind of scope and permission control and it all acts within Burp's boundaries.

2:06So yeah, very exciting for us to get that live at Black Hat and it's been great to see the reaction so far.

2:12James Wilson: Even in the way you frame that there, I've got questions already in terms of every man and his dog these days has an MCP that just wraps their existing product. But what you just said there sounds a little bit different insofar as you talked about Burp AT's agents, you talked about them having their own goals and context. So maybe just for someone who is high-level familiar with what Burp Suite is and what Burp Suite does, and obviously I'm very familiar with working with coding agents and security, you know, Von Research-focused harnesses, where do you guys sit on that spectrum of are you an MCP server Are you an integrated harness?

2:46James Wilson: Are you a bit of a combination of both? How does this work? So Burp AT is a fully dedicated product, and it can be, depending on how you brand it, a control plane, a harness for the agents and the technology that we built within. It is different from MCP access. We have seen a lot of experimentation from our users with the Burp MCP that is available separately and has been for a little while now. but an MCP and model access isn't the same as a professional grade security testing tool. A few reasons for that. One is the obvious one is governance and control. It's the boundaries but also the auditability and the confidence and the trust that it can act in the way that you hope it can act without any hallucinations or going outside of those bounds.

3:32But the other side of it is maximizing access, native access to the tooling within as well of which maybe Anjay can share a bit more about.

3:41James Wilson: Well, and before you jump in there, Anjay, the next question for me there is, it sounds like what you've created is almost like a batteries included, ready to go solution that leverages all things, burp suite tests, but in an agentic sort of environment. But I really want to understand, like, what does this product look like for the user? Like, am I launching a web app? Am I integrating with this as my primary surface? What models are behind it? Do I bring my own models? can I integrate with other things? Maybe let's bring it down to that next level of technical detail, Andrzej. Yeah, so I think to pick up slightly on something Kieran said, I think the difference with an MCP integration is this has been sort of built from the ground up to be optimised for agentic use in a way that maybe building your own harness via the MCP server isn't.

4:29I think a lot of MCP servers, including our own, are sort of a translation wrapper around an API of some kind, which wasn't designed for agentic use. So our API was designed to help you build plugins, and then we've wrapped it in a sort of MCP layer. But with Burp AT, it's way above and beyond that. So it's kind of natively integrated in ways that are not exposed via the API, because we've been able to do much more fancy things with it behind the scenes.

4:58James Wilson: Right, because there's only so much you can really get with a MCP as a, I guess, layer of abstraction. And as you say, not all MCP servers are created equally. Some are just wrapping APIs, some are doing a little bit extra, but it sounds like you've intentionally created a very, very opinionated product, right? You've decided which models should be used, the way the harness should work, the way the tools should integrate it, all to make sure that you're getting the absolute best out of Burp here. Is that the right way to look at this product? Yeah, exactly. So to clarify, there isn't sort of one model behind the scenes that's, you know, one model to rule them all we're using uh multiple different models uh i think the product manager talks about it as you know the right model for the right tool at the right time so we found that for certain uh tasks that an agent might perform maybe some of the older uh sort of dumber models actually perform better against our benchmarks but then in other cases um you know you need the sort of latest and greatest with the all of its uh reasoning power to to get the best results so So we're sort of giving you what we found will give you the best results.

5:59James Wilson: Yeah. It's banter's included. It's opinionated. It's everything we'd expect from Portswigger and it just works out of the box. But now let's maybe pivot into Kieran. What are you actually seeing from customers now that this has been in their hands? I guess during the beta program, what did you learn from that? And now that it's actually out there, what's sort of the interesting things you've seen done with this? Yeah, I mean, it's been great to see a lot of experimentation, right? So I think that the constant thing we're seeing where people are getting the most value is where you can give it a real objective to pursue.

6:31So it's essentially built in a way that's got different operating modes within. So if you want to stay particularly close to how the agents are working, as some testers do, either through skepticism or interest, you can do that and you can work really closely with it on an engagement. So, for example, you can delegate part of an investigation, like you've seen some suspicious behavior, and you're looking to confirm whether a real vulnerability actually exists behind it, and it can go forth and do that. But then on the other end, you can be more ambitious. And we're seeing users do that where they're shooting for more ambitious objectives.

7:03They're giving it maybe full app access and allowing it to work essentially more autonomously in a multi-step approach to obviously generate the findings and the evidence from maybe what might be akin to an end-to-end test. You essentially stay as close to it as you like. And we're essentially just seeing those modes coexist in a lot of cases. So on some tasks, a user will want to be there, down and dirty in the detail, whereas others, they want to just set something off in the background and allow it to go. And yeah, a lot of great examples so far of it. And we've got some really good use cases brewing.

7:38James Wilson: And talk me through a little bit around the case study you mentioned that one of your customers had brought to light for you. Yeah, so that was actually one of our beta users from Orange Cyber Defense, who are a large MSSP. So he, on an engagement, managed to find a vulnerability that disclosed supposedly anonymous whistleblower reports from the company's whistleblower platform. Ouch. Yeah. He said it was an awkward conversation with the client, was how he put it. So it's one of those things he was explaining, you know, during a pen test, he normally has a, they call it a four plus one, where he's spending four days testing and one day writing up his findings.

8:14And he says, you know, you're making hundreds, if not thousands of judgment calls throughout that about what do I test? How deep do I go? When do I change tack? So there's all kinds of vulnerabilities out there that even on an app like this, that's tested year after year, sat there undiscovered, partially due to them just being inherently time consuming to test for. So in this particular case, it was, I think he said there was 66 ,000 lines of minified JavaScript that he managed to find. and he's kind of like well there's probably something in there but i've got four days and i can't spend a day and a half trying to figure that out so pop it on the to-do list for if there's time at the end of the test which obviously there never is um i might be able to get around to that but then uh because he was taking part in the beta trial he sort of said actually do you know what burp at take a look at this see what you can find uh while he sort of carried on with with his sort of normal testing workflow working through his list um he sort of came back a few minutes later and it's saying oh this is uh this is the logic for a whistleblower reporting platform and interesting finding uh all of the reports are only encrypted with a six-digit alphanumeric code which we can probably brute force uh so so then he's he's kind of said oh well that's interesting um all right go ahead and do that then um and then an advantage of it being sort of natively integrated with burp is that it then um it then kind of uh spins up an intruder attack, configures it all, and then brute forces all of the codes and is giving back all the plain text, Whistlebrother reports.

9:46So yeah, quite a nice finding, really. And that was pretty much autonomously, he just kind of gave it things to look at, and it came back with the results.

9:53James Wilson: That's pretty cool. I mean, the whole idea of it being a force multiplier and allowing you to actually explore those things that would otherwise just fall off the back of the prioritization cycle is cool. But curious about the feedback, though. is that kind of an outlier in terms of, yeah, we threw Burp AT at this thing and ha ha, look, it found something. Or is that, you know, quite a common thing or are customers, you know, more finding that they really want to be watching what it's doing, guiding what it's doing, being very direct and deliberate about how it's sort of being deployed. And I'm curious as to the answer to that, but then where I want to really take this is we're all hearing about these sandbox escapes.

10:32James Wilson: And so how do you guys feel about, you know, when BoBIT is left to its own devices? Do you worry about it running away? Well, I'd say to take the first part, I think we've seen a bit of both. You know, there's naturally people who are slightly skeptical of it and are using it at the kind of lower autonomy end of the scale where, you know, they're banging their head against a wall trying to bypass a WAF or something. And then they run out of ideas. They're like, oh, I don't know. Why don't you try to see how you get on? But then all the way to things like this, where people are just giving it complete tasks to do end to end and it's coming back.

11:00And actually, I'd say the majority of people are using it more in the latter case. And people are generally surprised by quite how well it does in that respect.

11:11James Wilson: Surprised on what dimension, though? I'm curious as to what you mean there. Well, I think there's just an inherent scepticism, and there has been. And that's gradually kind of waning over the past, I'd say over the past 12 months, really. People are just getting more and more comfortable with the idea that AI, agentic AI use is going to be part of their workflow in some capacity. and so I think some people start out giving it you know smaller tasks or helping them finish off something they're doing but then as they're getting results in that small in that sort of assisted capacity they're then kind of trusting it more and more and sort of ramping up that autonomy dial and what we're actually finding is people are very very quickly moving up on that spectrum so starting out small and then you know give it a bit more then they'll screw it go and just do the whole task yourself.

11:57James Wilson: Straight on the YOLO mode, you know, bypass all permissions. Basically, yeah. So we sort of built it in a way that we wanted it to be, you know, give you as much control as possible so that you could do that and, you know, ramp the autonomy kind of up or down depending on the specific task you're doing, depending on the engagement, the client's kind of risk appetite for want of a better word. So we're really seeing that. people are really finding value in that. And out of curiosity, how sort of dynamic is that ability to dial up the full autonomy versus diving in, right? Like if I tell it, hey, go have a look at this website.

12:35James Wilson: I just, you know, just see what you can find. And then it comes back and says, oh, you know, I found this, this interesting thing. And do I have the ability then to say, well, hold on, stop now. Let's bring this right down a notch. And I want to know every single step and I'll guide you from here. How does that sort of dynamism play into this? Yeah, 100%. So we've got, so on the one hand, you can be sort of fully manual because it's still just the same VerpSuite you've always been using. But then we've got sort of three high level autonomy modes, let's call them. So on the one hand, you can have a sort of a more manual autonomy mode where you can configure tool call by tool call, whether it always asks you before doing it, whether it just acts without asking you and you control it on a tool by tool basis.

13:15At the other end of the spectrum, we have a sort of autonomous mode, which is it'll largely just work uninterrupted, except for a couple of key exceptions. So it can't add anything to the target scope without asking you for permission, even on the autonomous mode for fairly obvious reasons. And also it's able to script up its own, like where it doesn't have a tool to do a specific task, it can obviously code up its own tools. So before it executes any code that is written, it will always, always ask you. And then in the middle of those two, we have what we're calling smart modes, which essentially works like the autonomous mode, where it'll kind of run all the routine tasks without interrupting you.

13:56But then we then kind of bring in the model in certain scenarios where it'll kind of analyze based on the kind of the context of what it's doing within your project file and kind of think the way I think about it is it's sort of like saying, you know, I know you've told me to just go and do everything autonomously, but you might want to just double check this one. And so it sort of intelligently decides when to escalate things up to you. So you've got that kind of happy medium there.

14:22James Wilson: Yeah. And then before we sort of zoom out and think about what's the so what behind all of this, it's a little bit of a rabbit hole, but bear with me. How hard have you pushed this to really make it try to escape its boundaries? Like you mentioned there that, you know, it can't add scope to or a new target. That makes sense. But surely given all the things that are going on with models escaping the sandbox at the moment, have you actually sat down with it and actually tried to see what does it take to say to Burp AT, look, just do everything you possibly can. Stop at nothing, just keep going, be relentless.

14:56James Wilson: I'm curious, have you tried to get it to exceed its own boundaries? I don't know specifically about the tests we've done on the model itself, but what I can say is that the way we've architected it It has been very deliberate in the way we've isolated the model from the tooling. So all of the permissions and scope enforcement and things like that are enforced on a tooling layer. And because the model can sort of think about what it wants to do, but it can't actually act in its own right, what it does is it calls out to Burps tools and then all of the actions it performs, it does so via Burp Suite in the same way as a human would, clicking around the UI kind of.

15:35So because it's using just deterministic tools to act, we're then able to enforce deterministic guardrails on that. We're not relying on kind of coercing the model to please don't do this or please don't do that.

15:47James Wilson: So you've got sort of a bounded context there. It's operating only within the existing tool set of Burp. Now, one could argue that that's already quite a powerful tool set. So I think there's still interesting ways it could do things. But I get it. This is not an agent that has access to a shell and it'll go off to the races that way. But Kieran, maybe if we come back to you here and just sort of zoom out now for the so what of this, right? As a product marketing manager, it must be both interesting to see this being picked up by customers, but also quite difficult at the moment to just see very far around the corner in terms of like, what does this mean for the overall industry, right?

16:25James Wilson: I mean, software engineers were all supposed to be out of a job by now. Turns out we're not. cyber security some people thought it'd be a solved problem thanks to ai it's certainly not but where does this where's this go for from here for you yeah i think there's there's obviously a lot of change here um well i was obviously observing what was happening at blackout and just this week i was at the the garner risk summit as well so that there are a lot of unanswered questions around what the future will will obviously hold um a lot of what we're doing here operates within that kind of manual pen testing space and we're trying to really augment the tester.

16:58Historically, they've been outnumbered, outgunned in some cases, but the resources haven't been there to keep up with development throughput. And obviously, with AI-generated code, that's become an even bigger problem. So we're seeing pen testing and the act of web app testing moving to more of a continuous state, aided by automations, of which obviously Agentec AI helps with. But then equally, we're now looking to arm any manual testers with the best tools for the job and try and level the playing field to some degree here. So this isn't a case of AI comes into a bloated part of the business and we can now reduce headcount and just put all of our cash into automating through agentic AI.

17:41This is now around, I guess, those different modes coexisting where your testers can really benefit from the power of agents in their own workflows. But then you can also complement those with the automations that are frankly needed at this point for pen testing to become more continuous.

17:57James Wilson: Which makes sense. I mean, you're essentially putting very powerful tools in the hands of people that already understand this craft and it's a force multiplier for them. I imagine though that there's got to be a bit of a frequent if not constant sort of trade-off that you're having to make in terms of how do you prioritize and think about do you continue to invest money in making this the best possible tool for a pen tester to work more agentically versus work that you might do to make someone who's not a pen tester or is adjacent to the pen tester to actually become more of take on that pen testing role.

18:30James Wilson: Like for a software engineer, for example, for the longest time, pen testing was just not something I'd do. I'd write the code, I'd ship the code, someone else would come in and test it. If I had the right tools in my hands, then my adjacent sort of skill set of software engineering can quickly turn me into at least some degree of a pen tester. But that's got to require some deliberate, I guess, prioritization and trade-off on the part of Burp Suite and Portsquigger is, I guess what I'm asking is when that decision comes up, how do you navigate it? How do you navigate, do we continue to invest to make this the best tool for a pen tester or do we make this the best tool for someone to become a pen tester?

19:06So maybe answer the question slightly differently. We've had to make some hard calls over the years around how we've looked at Burp Seed Professional and arming pen testers with the best tools and and automations through our DAST capability at some scale. So we're fighting different battles in a way, but they're kind of serving the same ultimate outcome. And that problem kind of carries across this world of agentic testing. But now we don't have to really make the binary choice of like, are you going to invest in product A versus product B or user A versus user B? The capability is a constant.

19:37You can basically build the agentic core and the technology that can help many different use cases, many different people. and then you're thinking really around the level of authority that you're delegating to the agents and how you want to pivot them to help you best. So that could be at one end, you know, helping a junior tester get up to speed more quickly and become more effective through to an organization, being able to plug this into CI pipelines or scheduling platforms to allow for the benefits there on that level. So we're really trying to invest most in the actual capability and then think about how to deploy it separately from there.

20:13I think related to that as well I think one way I've been looking at it is you know the the skill of a pen tester isn't their ability to you know literally use a certain tool and click around and configure an attack or something like that the the skill of a pen tester what you're what you're paying them for is their their brain their ability to have that lateral thinking to think of the things that the developers didn't think of and all the quirky things and then I think what this does is it bridges the gap between that and being able to actually apply that idea in practice. So you can think, I wonder what happens if I do this crazy combination of things.

20:46Whereas in the past, you might have then had to spend half an hour, an hour kind of configuring things, trying individual requests back and forth. Whereas now you can, as long as you can sort of articulate the idea you have in a prompt, then you can go from idea to end of the test in a matter of no time.

21:04James Wilson: so what a time to be alive and in this industry well listen Kieran Hughes head of marketing and RJ product marketing manager from Portsooga it was an absolute pleasure to meet you both thank you for dropping by for a chat to tell us all about Burp AT. Cheers James. Thanks James.