SmartBear and Multi-Agent QA

23:45The QA lead, the system itself kind of is that orchestrator. In other words, anyone's able to create a task, whether it's an end user or a schedule firing or an API call from a CICD system to say, hey, create a task to run this test case. And then it gets spun up. So anything can create a task. But once it's created, then the system will create an agent to execute the task. And then again, certain tasks have the ability to fork or spawn another task. Can you create subtasks and do this whole planning? Exactly. And then we try to link all that up to it in the UI. So if you spin up a QA lead task to help with a test running task, then that QA lead task will be linked into the original task it was spun up within.

24:22Got it. And then I'm particularly interested in this multitask coordination. Yeah. What interface do you pass context around through for that? Are you just giving them a starter place? Do you have a tool available for them to fetch the original? Like, how does that end up working? Yeah, yeah. This is cool because this is one of the funnest parts to work on as an engineer in the system. So the test runner spins up. It's got a browser attached to it. It has access to an LLM. Let's say it hits a failing step. It spins up a QA lead. It tells the QA lead, say, here's a screenshot of the browser where I am.

24:54I have the functionality to do anything in the browser. So click, input, hover, scroll, keys, whatever you want. Tell me what I should do. And so then the QA lead has access to an LLM as well. And so the QA lead follows this predefined loop. You might call it a skill where it collects context. So it takes a screenshot. It looks at recent runs for the current test case that we're executing to see what they looked like at this particular step. So it collects extra context and it also has access to basically all of the data in the account. So that includes things like custom defined resources or context provided by our end user.

25:29So this is where that case of like, if you're ever executing a test and need a credit card, this is the test credit card that you should use, something like that. So the QA lead can assemble that context. It can actually manipulate the browser in a non-destructive way, in a non - side-effecting way. So for example, scrolls, other than infinite scroll, where like that actually fetches more data, scrolls are generally non-side-effecting, meaning that you can perform actions to learn more, to capture more screenshots about the state of the app without actually changing any of the state. Again, within reason, there's infinite scrolls and edge case there.

26:01So the QA lead can collect more context. It can query the LLM. It can take actions. And then it can send back to the test runner. And it says, look, I did this, this, and this. And it looks like you've gotten further in your test case. So it's not a true failure. You just needed to word your test case steps a little differently. Here's the wording for your test steps, for your updates, for your definition. It's been self-healed. You're good to go. QA lead shuts down and the test runner continues on its way. So they communicate to each other over like a predefined interface, just sort of using like a relay through the database, like a pub sub.

26:30Okay, got it. Interesting. So just I'm going to like echo back to make sure I understand. So you have tester agent is running. And it says, Oh, shoot, I'm stuck. Let me create a new task, which is going to spin up the QA lead, it has sufficient context that the QA lead then knows my ID, so it can pub sub to me, you have some shared pub sub. It has info about like what test case and all these different things that a QA lead has access to a whole bunch of other tools that it can use to. And based on what I'm hearing, you're using this agentic pull approach. where it's basically going and fetching.

27:02It says, okay, let me fetch this from this, not preloaded with a bunch of contacts, but let me fetch this account info. Let me fetch anything I need. So actually one thing that that leads me to, so if it's wanting to manipulate the browser, it's wanting to test a thing, does it do that by sending PubSubs to the tester agent, having the tester agent do it? Or does it have a direct route to that browser entity? Yes, that's a great question. I'm loving that you're getting to this level of detail. So it goes back to the test runner and the test runner says, I have these tools, these faculties, these capabilities that I can perform in the browser.

27:32Let me know if you want me to do something. So the QA lead sends a message to the tester agent. So the tester agent remains the source of truth for manipulating the browser. The QA leads the author or the visionary, but the manual tester agent remains the owner of the relationship to the browser. Now, conversely, the manual tester does not have access to all the account data. So in a happy state, we want that tester just doing what it was narrowly charged with doing. And then if it fails, it can speak up and pull in a more authorized agent to then go and access more of the account. And that agent is tasked with assembling the correct context.

28:09Now, one piece I left out is we do often pass information indirectly through blob storage. So rather than the manual tester agent sending 100 megabytes or 30 megabytes of images and data back to the QA lead, it puts that into S3, you know, blob storage. And then it gives an identifier to the QA lead to pull that down. So that's just sort of to minimize like copying and sending. Basically that data lives in S3 when the manual agent gets it. And so manual tester agent. So that's kind of that makes sense. That makes sense. So I'm going to still just keep digging in the details if you don't mind, because I geek out about this stuff entirely.

28:44So when you have the QA lead, I'm going to say remotely manipulating the tester agent, right? It's like asking for these things. Is it directly accessing the browser tools so that there's not an LLM in the loop on the tester agent side, right? So that it's accessing through that container, but not through the LLM. So the tester agent doesn't have the like history context, or is it sending a request and the LLM is like doing a tool call and kind of building up this like conversation of, oh, my lead asked me for this. I'm going to do this work and then send it back. Yeah, very much the former. Yep.

29:16Another great question. Like makes total sense. The QA lead has that LLM agentic loop. It basically has a manifest of tools that the manual tester contract that the manual tester agent is willing to fulfill. The QA lead manipulates that as if it were its own set of tools. And then at the conclusion though, the QA lead does provide sort of like a summary, a roll up, if you will, of like, this is what I did. This is what you need to know. These are the updates you need to make to your test case when you're done, but it does not share the entire conversation history. It summarizes that and gives that back to the manual tester.

29:49That makes sense, right? So you're keeping the programmatic ownership of the browser is still in the process that is running the tester agent, but it's running off in this LLM loop in the QA lead agent. And it's only giving a digest at the end of here's what's changed. Here's what I did. Here's all these things. Yeah. And the thinking there is like, that keeps the separation of concerns a little bit intact where if you have a browser session, so we have different services for spinning up these different parts of the thing. So we have one service that does the browsers. We want that to be owned by one sort of place in memory, one worker agent, one instantiation.

30:20So that's the test runner agent. If you find that test runner agent, you can find everything that we told it to do. We also want the expensive, you know, the full on account access loop to live in a separate QA lead process. That's always the thing responsible for that has the authority that has the expensive models. So that's always there. So, and they communicate over this pub sub. So yeah, like it's intentional to do it that way. Really the reason is if you get the manual tester, if the manual tester starts being able to access everything in the account, everything's a QA lead, right? You have one agent now and it's just, it's just, it's doing all these and maybe we get there eventually.

30:51But the benefit to now is it's a little safer in these early innings of the AI world, where like, we don't want the manual tester screwing up and doing something in the account that we don't, we didn't intend. Yeah, no, that makes sense. And I think it also, to your point, allows you to use much cheaper models or much in just like, the smaller the model, the more it will get confused if you give it lots and lots of tools, like all those different things, you can keep it scoped down. Exactly. This does lead to the question of what happens when something breaks. And I'm going to put a particular case on that.

31:21I'm going to say, okay, let's say your tester agent has delegated to a QA lead agent. The QA agent is manipulating your browser. And now for whatever reason, the browser crashes. What happens? Yeah. So in that case, again, and this is where we want the more complex, we'll say construction of prompts, if you will, like the chain of thought, you know, like all these different prompts that are coming through in that agentic loop. We want that in one place. So in the QA lead loop, it will do things at a couple different levels of abstraction. So at the smallest level, it's thinking, I want to try to get this form submitted.

32:00What's it going to take to submit this form? But it's also operating within the larger context of a test case and the goal of that test case or the description of that test case. So as it's doing things, it needs to keep periodically checking back into the context of the test case to make sure, have I strayed too far from my goal or my description? And then the outermost loop is just, is this app even up and working? Is the browser still up? Is it too slow to do anything? Is there an external issue? And periodically, we need to rise to that level of attraction. It's kind of like the human mind, how at any given moment, you can focus intensely on one thing, but you're doing that and you're also breathing and you're also feeding yourself throughout the day.

32:37And like, you know, is there a threat to your safety? Like that, you're kind of like different levels of abstraction. So the QA lead has to do that. And so if the app crashes, the QA lead will rise to that higher level of abstraction and basically say, I need to stop. The test case is not working. The application has crashed. We'll pause here and it won't do any further work. And so it sends that back to the tester agent and it says, terminate your session. There's nothing else to be done right now. The patient's crashed. Are there any particularly challenging like error states or error cases to manage, especially I keep going into anytime you have multiple things orchestrating, like where are the edge cases that you run into in that?

33:13The really challenging thing that we're dealing with a lot now is that test data problem. And so it's not so much within a single browser session that we have this issue. It's the interplay of multiple sessions running at once. So an example here, right? Let's say you say I've got this QA team and I want to spin them up. I want to hire a hundred people because I have a hundred test cases. I want them each to do one thing and I want to be done in three minutes. You know, every, every test case is three to five minutes long. You can do that, but you probably would give each of them their own QA testing account.

33:45And so they wouldn't be conflicting on each other. One person, the most recently created record in one of those accounts is the record we just created, that person just created it earlier in this test case, right? But now in the world of AI, at least today in BearQ, we don't accept, we don't take a hundred QA accounts. We do the same account, but we spin up a lot of agents at once. And so when there's concurrent rights or not even necessarily concurrent rights, they might be sequential, but if one test case adds something to cart, but leaves it there and the next test case goes to the cart thinking that it, there should be nothing in it and it sees something in there and that throws off a problem.

34:22So I actually think there's a startup worthy problem around test data management of applications. And it's probably also related to something that we do in BearCue a lot, which is trying to build up sort of a holistic vision of the application and the organization of it, the screens, the content, the functionality, like the user experience, how you interact with it. We try to build up an application knowledge graph, basically for your application. Test data has to fill a role in there. And that's one of our challenges that we're going to have to solve in our, what we call like the application model in that object we construct that describes your application.

34:56Test data and just the data relationship. Again, like you can't, you know, in lots of apps, you figure like you can't take one action without having created a whole bunch of records in the database that do X, Y, Z, you know? And so like managing that state is incredible. I'm imagining, right? Because your agents probably try to do some amount of self-correction in different places. I go to the cart. I see, oh, there's something in here. Well, let me clear that out so I can do my test. And then the other agent, which has been waiting on an LL, comes back and he's like, Like, where's my thing? Right, right, exactly.

35:24You're totally right. And so I think this is where people think like, oh, it's AI or it's computer-based, it's not human-based. So just spin up 100 agents simultaneously. It's like, we can do that from an infrastructure perspective, that's no problem. But the data coordination isn't there. And so I'm kind of solutioning this with you right now. I think the solution is to get, we'll run as many parallel agents as you want to give us test accounts in your applications. Then we won't be conflicting and we'll be sure that we can run at, at full parallelism. There still ends up, depending on what you end up testing, there still end up being challenges, right?

35:57If you're testing like an admin account, it's going to see different records popping in and out and other things. Right. I wonder if there's a world at which you sometimes, you start building up a sort of data model, right? You mentioned you build a component model, essentially, where you're inferring a component model, but do you start building up a data model or at least dependencies between the different - Data types. Data types and components. I add something to this form and something shows up over here. Okay, there's a linkage here. And then now you have an orchestration problem instead of a data problem.

36:30Yeah, you're spot on. You're spot on. I absolutely think there's a startup worthy problem there. Just basically looking at applications. It's effectively like the data model in the underlying database or if there's a NoSQL store or BlobStorge, whatever it is, just figuring out those relationships between fields in different data objects and how changing one has to have an equal reaction in the the other. What's interesting there, coming back to the AI coding side of things, is like, you may be able to build up a better understanding than the engineers have. I mean, not in a well-run engineering org, I don't think, right?

37:06Because in theory, we still are all understanding and reviewing things. But like, there's a whole lot of code that's getting vibe coded out there that nobody knows how it's working. Yeah. And I think the key point you're making too is the relationship between those two data types may not be apparent at the code level, but it may be apparent in the application experience level. And so if you submit a form in one place and that brings up a row and a table somewhere else, those two may be pulling from potentially different places in the code through vibe coding error, right? You know, lack of proper factoring.

37:39And could we uncover that? And could we actually have a, almost like a source of truth for the data relationships inferred or gleaned from using the app? It's an interesting problem. And I think it's a new frontier, right? That we're all approaching here. Can you understand the design of an application better from using it than from authoring it? It gets to this question of how do we build our understandings of what our things do, right? Like this has been one of the challenges that I've been talking with all sorts of people around with Vibe Coding. Because coding itself has multiple, back when we were hand coding everything, it was serving multiple purposes.

38:16One of those purposes was creating an executable artifact. Another purpose was helping us build up a mental model of a system and also a mental model of our user and our user's problems. And if you're delegating it all to a machine, where do you get those mental models? Well, maybe you get it from something like your tool. I totally agree with you there. I also think that we're not quite at the point yet, at least on our team, where people can author code without owning it. We make the argument, so the LLM may still type the individual lines of code, so to speak, but each line of code you commit, you own.

38:51And so we're still at that point. I don't know if you saw Brian Cantrell is at Oxide Computing. He wrote this post about proper use of LLMs for software development. I've shared that with my team. I really believe in a lot of the things he's calling out there. But basically, we're not quite at the point yet. Basically, everyone's using AI to author their code. But the expectation is that you know how it's working and the data that gets spit out from one function, how it's used in the next function, you know that relationship. Certainly, there's lots of pressure to go faster and maybe relinquish some of that control from the engineers.

39:21But we still own the code that we author on my team. And I think that's helping to push back a little bit on this loss of knowledge, deep knowledge about how applications are building, how the data is flowing. but you're totally right that there was like a big part of writing the code actually was getting you to keep it in your head and helping you to make sure that these relationships that you had or these invariants that you demanded were in fact being upheld and were still as you thought they were and it's not even necessarily like i think having humans have ownership in the end is really valuable and really important and there is a whole can of worms to dig into if you start unlocking that and or removing that yeah linkage but i think it is not a hundred percent clear to me that reading code is as effective for updating our mental models as writing it is certainly i see people with different amounts of challenges when they're reviewing code and trying to update things and things like that and so i was i was wondering yeah you have this like outside in approach of let me infer what the components are let me infer maybe what the data model is if not yet then maybe sometime soon.

40:25Is that like exploration approach, maybe with a tool in the loop, a better way for us to build those mental models? I think it's possible. But even if we infer a proper relationship graph, suppose, between the data, and we can explain that, that alone would still be text and probably pretty dense text. Or maybe it's a visual, you know, if you generate like a system diagram or something. So that to me is going to be consumed with the same efficacy as reading source code would be. I also strongly agree that the writing of the code, it's like the taking notes in school, writing the note down, you know, probably may help you remember the things better than just hearing it or, you know, seeing it on the blackboard.

41:05So I think there's definitely something to the act of writing code, writing lines of code yourself and how that contributes to your understanding. So it is possible that we'll need new tools in the future world. You may own it, you may read it, but are you really going to grok it? You may not. Is there some way to have kind of the external result read back to you by an agent that helps you to understand what it's doing better. At the end of the day, that's kind of like, that'll be the new normal, right? It's like, it doesn't really matter so much what the code says. It matters, does the application work as intended?

41:35And if it doesn't, let's say that the common retort to that is like, what if it fails under scale? Well, then we'll just define that into, does it work as intended? Well, to work as intended, it must now support 100 ,000 concurrent users. And then that goes and manipulates the code. In certain perspectives, like you could imagine the code really doesn't matter as long as you've adequately, in painstaking detail, described all of the requirements of your application. Of course, at that point, that requirement stock ends up looking like source code. And so it does, would it be more efficient to just write the code yourself?

42:06Maybe. So that remains to be seen. So we're kind of moving now into this concept of like, how do humans fit into this world again? And I'd like to bring that back to BearQ in terms of like, we talked a lot about the inner loop and things like this. What is the outer loop? What is the human in the loop do at this point in a QA process with BearQ? So it's validating that we have, in fact, identified components that are important and worth reusing in your application. It's validating that the test cases we've written are accurate and important. We kind of have a common sense grounding there, but there's lots of nuance for every individual application.

42:42And so we take human input for that? Are these valuable test cases, et cetera. We also take human input on the report side of things. So we make suggestions or recommendations for how we want to update your account. And you kind of can imagine like in this QA team analogy, we try to maintain this image of a person, of a human user, whether they're on the QA team or it's a smaller organization, maybe they're the VP of engineering at a five person startup, but they poke their head through the door and they bark out an order and they can get a response or an action taken in response to their command by this fleet of agents.

43:17And so the things that a director of engineering or the QA manager or the head of quality may care about out of various organizations are the things that we hope to surface in our reporting feature. So we'll do things like this is how many test cases we ran. This is a failure we saw across 13 of your test cases, for example. We think it's this same underlying issue. So we've suspended further tests that are going to manipulate that component because we think that it's like a, it's an external failure. We won't waste any more time. We did this, we did that. Performance is slower now today compared to the last three days or performance is right in line as things have been.

43:51So you're good. And then the human user can say things like, well, we're doing a big push on the, I always use the profile settings as my example. We're doing a big push on the settings page, go and create more tests there or run the test there again, but do it with a higher level of debugging logging. And so I can then mine those logs later. So we see the human basically as an orchestrator, a director, an approver, that type of thing of these QA agents. What do you have in terms of like guardrails? Like, let's say I want to use, I mean, I think we've talked a lot about test accounts in places, but let's say I want to use you on my production application.

44:28But there may be pieces of it. I'm like, oh, that's risky. I don't want to actually put an agent in the loop there or things like that. Like, what can I do to control it? Can I, are there ways I can disable destructive behavior? Like what are the knobs I have? Yeah. So the resources or we call it like resources or context is the open form input you can give us. It includes JIRA, GitHub, et cetera. It's also just open-ended instructions. So you can tell us things to do or not to do. And we adhere to those instructions throughout all of our actions. So those are always in context. You can kind of think those human level instructions.

45:02So things like don't ever visit this page, or if you're on this page, immediately navigate away. That's an instruction that's always in context for all of our LLM interactions. Okay, but I'm a paranoid VP of engineering. I don't want to trust the LLM to this. Are there any like hard lines I can put? Yeah, so exactly. So we don't do this yet today. We're still in the early stages having just released. But what we envision is something along the lines of like what the session replay tools to where you can annotate individual elements or pages on your, in your application. So I'm thinking like LogRocket, you can tag certain fields in LogRocket, certain elements as just like no record or, you know, whatever.

45:40And so then LogRocket just doesn't even pull them into its DOM map. And so we would do kind of the same thing there. That would kind of get you back into reality, right? Getting back to this notion of getting out of the living in the, building the house on sand and instead being in touch with reality. That would be something where it's like, no, you're, you're in the code, right? As long as your engineering team has those annotations in your application, we'll honor them and we won't even see them. So, you know, that would be a way to get you back into touch. Got it. That makes sense. And so that's kind of at the UI level.

46:09Can I put in any sort of like browser level blocks or things where I'm like, OK, any path starting with this or any of these things, like just don't even hit it? We don't support that today, but there's absolutely no reason we couldn't. we're spinning up these browsers, we're setting all sorts of flags and capturing the HAR file and the video and all that type of stuff. So that's a great feature. Actually, I'll take a note after the call here. I think it's a great idea, basically just to intercept that call and block it. I think, yeah, absolutely. There's no reason why we couldn't support something like that.

46:37For example, we already support setting custom cookies or setting custom local storage values in the browser. So all of that stuff is configurable. That's awesome. So this kind of brings us into this sort of future looking conversation. Where do you see autonomous testing and just AI related quality work going over the next couple of years? You know, I think the trust question is still the biggest unknown. And I think that's where the bulk of the work has to come. Really, actually, I think it's the same in the software on the coding side of things, as well as the quality side of things. It's what's possible now is so much more than what was before.

47:15And I think we're still kind of towards the tail end of discovering the bound of what is now possible. I'm selfishly kind of hoping the velocity, the rate of velocity of change, like slows down a little bit here in the bottle. It would be nice, wouldn't it? It's so much to digest, you know, like every week it's like something new. So I'm hoping, I think we're kind of maybe near the tail end of kind of this wave here of the last call the last like year and a half or two years. So now I think take a breath, see what's possible. And now we set to the work, the engineering work of building on top of this new capability.

47:49Hopefully that will be satisfying in a different way. I think there's so much curiosity and like, whoa, like mind-blowing, mind-blowing stuff in the last couple of years. But I think it'll be really satisfying from an engineering perspective, again, to get out of that building on sand and building on rock and starting to establish patterns and bounds and scope and best practices with these tools and around how to build software with them. That's kind of what I predict. And so that engineering work is where you'll start to build up trust that, okay, we have a new way to build a bridge now. There's new materials, they're hyper, whatever, resilient, and they don't break or whatever.

48:24So now we have the engineering task of what is the right way to lay the bricks to build this bridge or to connect these pieces here. I think that's really what the focus will be for the next couple of years. And so QA will be impacted by that. I think software engineering will be impacted by that. I think it's all about building up the trust that the software is going to work with 99.99 % reliability the way that the pre-AI software did. Not speaking about bugs, I'm talking about like gamma rays coming in and flipping a bit to break your non-AI software. We want to try to get to that level of error.

48:55That makes a ton of sense. Related to that, everybody in the software engineering world right now is kind of grappling with like, what does my role look like now? What am I still doing? What is still important? What is more important now what things am I getting rid of? What does that look like for QA folks? It's a great question. I think it's really similar. I probably shouldn't be testing the basic create, update, delete record flow in my application anymore. At a minimum, even if I don't use a bare queue, I should probably be using an agent to perform these three to five steps that I do every day as a smoke test in my application.

49:33The QA person should probably think in terms of how can I use an AI agent to do these mundane tasks at a minimum that I'm doing, these repetitive, these well-known, well-understood, we'll say high value, like the fact that they're working, but low complexity tasks. That's what they should be focusing on, getting out of that loop. And then I think after that, they can start to think about maybe, and this is where we kind of, we haven't talked about this, but a lot of people talk about maybe the the merging or the blending of roles at organizations, you kind of start to see the QA person kind of becomes almost like a junior PM or maybe a regular PM.

50:11And the PM, you know, who was doing some QA work, maybe now they have agents doing the QA work. And so there's less of a need for them to perform in that role. You know, at organizations of all different sizes, you have engineers doing QA, you've got engineers doing customer support like I did for years at my startup. And you've got, you know, marketing folks who are doubling as customer success and you've got other types of role boundaries getting blurred. So I think AI maybe accelerates that. And so in some ways it can be it's a risk, right? Is someone going to someone else going to perform the QA role that you use to perform?

50:43In other ways, it's an opportunity. Can you now contribute in other ways and actually more valuable ways to your organization by using AI properly? So I think it's a rising of the level. It's the raising of the level of abstraction. You know, people are not going to operate at that low level anymore. Yeah. I've seen it described as we're moving from T-shaped people, right? Where you're deep in one and shallow in a lot to like plus shaped people where like you're still deep on one, but like maybe with the AI, you can be like reasonably good at a lot of different things. Yeah, absolutely. I think it's an amplifier.

51:12If you have talent and skill, you can contribute in more ways to an organization through the use of AI. One thing I actually wanted to mention that if you kind of think of like the golden era of SaaS is under attack, say, of like the CRUD app, where I built this custom app for barbers. It's really great software that knows it was built for their custom built for their domain. And it's really great. Maybe that's under attack a little bit in the next five to 10 years, because AI can build a lot of it. And so the basic CRUD app is under threat. So what that means then is if you're building software, you probably want to be building something more complex and harder.

51:48And I think that's where like, you'll see a lot of innovation now is in bringing software into the physical world in the form of drones or robots or whatever. And so it's like, it's a raising of that complexity factor. Or the other thing I was going to say is dropping down a little lower. If you're working on something like a compiler, that's something that you probably need a human in the loop for a lot of those decisions. Maybe still not for like the individual function writing. You can still have AI for that, but there's a lot of complexity that the AI is not necessarily going to nail out of the gate.

52:21Some of those lower level software engineering tasks might increase in importance as well. To make that kind of concrete, let's imagine that your LLM, all this LLM has taken over all the building of the CRUD software, but they're all making sort of one common, quote unquote, best practice mistake where they do something very inefficient. If you're looking, say, at the lower level and you're observing how memory is allocated in these apps or you're working, say, in the JVM or something like that, there could be a real outsized impact to all the 10x amount of software that is now being shipped and released on top of the JVM, that one, you know, compute cycle that you're saving for every loop iteration might actually be, you know, millions of dollars in savings.

53:00So that type of software is worth a second look as well, I think, you know, basically go higher or go lower, but don't stay in the CRUD app. Yeah, for sure. Where's SmartBear looking for the next? You're talking about embedded. I have this little embedded device I'm playing. Are any QA harnesses for embedded coming down or other different things? Nothing in embedded. The BearQ approach is visual by nature. So we could support something like a kiosk based app, you know, something like an ATM or something like that. I know historically there was a whole market for testing specifically kiosk tablet focused apps that was sort of underserved, or maybe there were one or two big players, but very legacy players, opportunity for some disruption.

53:39We don't have anything in the embedded space, but we would hope to extend BearCue to desktop, to maybe to kiosk, but certainly to mobile and obviously hope to dominate the web. So that's kind of where our head is right now is BearCue and visual processing. Embedded would be very cool, but that kind of aligns with my point. I think that there will be a whole market. As more of this tech, more software gets pushed into the physical world in the form of robots and toys and drones and whatever machines, I think there will then be opportunities for testing harnesses of those machines. Yeah. Well, and given a visual outside in black box approach, y 'all will be pretty well positioned to start exploring that too.

54:15Yeah. If you could tell me from the exterior that things are working, I can do these 50 ,000 things. It's probably the case then that the software is working as intended, as long as 50 ,000 things is an exhaustive list of what you wanted it to do. Right. You can kind of always prove that it does everything I want. And if it doesn't, oh, it doesn't do X. Well, X wasn't in the list. So let's add it to the list and let's make sure it does it. Yeah, I love it. Well, we're just about at the end of our time. Is there anything we haven't talked about that you would like to discuss before we wrap? This was really interesting.

54:43I'm really glad we got into the nitty gritty details of kind of the multi-agent passing. I think it's a fun problem to work on and I hope it resonates with your audience. So yeah, I think we've kind of covered all the stuff on BearQ and all my views on AI and the velocity rate of change, that type of stuff. So yeah, all good on my side. Cool.

55:05Thank you.