Automate all the UIs!

0:28Welcome to Practical AI. No ops required. Learn more at fly.io.

0:42What's up, friends? I'm here with Vijay Raji, CEO and founder of Statsig, where they help thousands of companies from startups to Fortune 500s to ship faster and smarter with a unified platform for feature flags, experimentation, and analytics. So Vijay, what's the inception story of StatSick? Why did you build this? Yeah, so StatSick started about two and a half years ago. And before that, I was at Facebook for 10 years where I saw firsthand the set of tools that people or engineers inside Facebook had access to. And this breadth and depth of the tools that actually led to the formation of the canonical engineering culture that Facebook is famous for.

1:22And that also got me thinking about like, you know, how do you distill all of that and bring it out to everyone. If every company wants to like build that kind of an engineering culture of building and shipping things really fast, using data to make data informed decisions, and then also informed to like, what do you need to go invest in next? And all of that was like fascinating, was really, really powerful. So, so much so that I decided to quit Facebook and start this company. Yeah. So in the last two and a half years, we've been building those tools that are helping engineers today to build and ship new features and then roll them out.

1:57And as they're rolling it out, also understand the impact of those features. Does it have bugs? Does it impact your customers in the way that you expected it? Or are there some side effects, unintended side effects? And knowing those things help you make your product better. It's somewhat common now to hear this train of thought where an engineer developer was at one of the big companies, Facebook, Google, Airbnb, you name it. And they get used to certain tooling on the inside. They get used to certain workflows, certain developer culture, certain ways of doing things, tooling, of course. And then they leave and they miss everything they had while at that company.

2:36And they go and they start their own company like you did. What are your thoughts on that? What are your thoughts on that kind of tech being on the inside of the big companies? and those of us out here, not in those companies, without that tooling. In order to get the same level of sophistication of tools that companies like Facebook, Google, Airbnb, and Uber have, you need to invest quite a bit. You need to take some of your best engineers and then go have them go build tools like this. And not every company has the luxury to go do that, right? Because it's a pretty large investment. And so the fact that the sophistication of those tools inside these companies have advanced so much and that's like left behind most of the other companies and the tooling that they're they get access to is that's that's exactly the opportunity that i was like okay well we need to bring those sophistication outside so everybody can be you benefiting from these okay the next step is to go to statsig.com slash changelaw they're offering our fans free white glove onboarding, including migration support, in addition to 5 million free events per month.

3:44That's massive. Test drive Statsig today at Statsig.com slash changelaw. That's S-T-A-T-S-I-G.com slash changelaw. The link is in the show notes.

4:11Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm the founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing well today, Daniel. How are you? I can't complain. It's like these two weeks in the Midwest in the United States that I enjoy each year where it's like between hot and really, really cold. And so, yeah, I'm really, I'm excited about that. And yeah, just cranking away, I'll be at the Intel Innovation Conference next week, which is going to be a fun experience to see some of what they're doing in the AI space and talk to them about some of the stuff we've been trying on Intel hardware.

4:57So yeah, it's a lot of good stuff coming up. I'm really excited. Actually, it was through that Intel community that I met Dominic from AskUI, which is our guest today. So welcome, Dominic. It's great to have you here. Hey, thanks to having me. Yeah, well, AskUI, could you tell us a little bit about first, maybe just what is it about UI that AskUI is concerned with? And how did you get to start thinking about UI and automation? Maybe a little bit back on what are we doing? We try to free humans from being robots. What does it mean in particular? Not only do you have repetitive tasks on user interfaces, but what we are trying to do to bridge the gap between you have to be the programmer versus you can describe in a natural language what you want to do.

6:00So your intention, what you want to automate. For example, if you want to log in in your Facebook, then you can say with us, please click the log in button. Please fill out this and this credential with natural language, which allows also non-technical guys to automate user interface and lower the border, which you can automate or start to automate things. And the second question was a little bit about... Why did you start caring about this problem? Why, why, why, why I started caring? A little bit of background about this. Previously, my background is a software developer. I was working previously at Siemens before I found it with Jonas, my co-founder of ASCII.

6:45So there, what I was doing there, I was on the plant automation system where we had to test everything because our systems have to run 365 days a year. So everything should be tested well enough. So we were standing a lot of inside the testing level and test everything through. Then I switched also a little bit the part month, the same thing. You have to test, you have to stand directly on to test the application. And then I was moving to a new organization, which tries to modernize or bring the agility within Siemens. I learned a lot about Scrum agility, bring all the new stuff, tools. But I had also the problem because Selenium and other tools couldn't solve the pain to write unit tests, the user interface tests in the way.

7:35And then I was thinking, hey, can we not solve this with AI? because AI can understand visual information and can understand initial information. So I thought, hey, can we not combine this? And all of this, the journey started. This might be a completely ignorant question, but I just want it for my own context as well. Now, in my background as a data scientist, I've participated many times in the fun activity of web scraping. and through that I know like oh there's a lot of ways in which web data or UI data might be exposed so could you talk a little bit about like how what does the data look like like what does the problem look like as you're trying to get an AI model to interact or use an AI model or natural language input to an AI model to then interact with either a UI or a web page, especially when a lot of those things can be quite varied in terms of how they're built.

8:40Is it more of a visual thing or is it something else? Yeah, so we are on the visual part. For example, when you're talking about Selenium, Selenium is only working with web interfaces and you cannot use this, for example, on Android and so on. What we are doing, we are doing really a screenshot of the system. So it means we can take a screenshot of the application. We can take a screenshot directly of the operating system. And then our AI model, what we trained, can detect the user interfaces. That means we're detecting buttons, we're detecting text, we're detecting text fields, we're detecting checkboxes, we're detecting icons.

9:18And so we have already trained a model which can understand the visual representations of user interface. And then we connected our natural language part to it to match the intention. What do you have? For example, click on the lock-in button to the real lock-in button on the user interface. And then we are moving them out there. So this is a little bit different from the concept versus I connect directly to the application. and then try to scrape the source code of the application to get my information. Maybe this is the best way to differentiate. So you're kind of starting with a screenshot instead of doing web scraping.

9:57Correct. And then just doing classification on the screenshot itself to do that? Correct. So we have object detection model in place, which really takes the screenshot and detect all the elements on it. As a quick follow-up, when you do the classification and identify what you have in the screenshot, how do you tie that in with the tests desired? So you have a collection of tests that you're trying to automate. How are you tying, like, I got a screenshot, I have a button, I have a text field. How does that tie into a particular unit test or something? Currently, we have our TextScript application, which you can download, which is also available as an RSI NPM package, which you can use.

10:36And with this, you can directly start, install it, and then generate a standard test. And then you have, for example, the Jest test block or Java JUnit test block, where you then can write ask your eye dot click dot with text dot so on. And the background, what happens, we have a controller then installed on your local system which connects to the operating system, takes a screenshot and also have the ability to control the mouse, move the mouse. Then we're taking the screenshot, we are connecting the instruction, the click on button for example, then this to our inference backend, then we get the result back and then we're moving the mouse there.

11:21And with this single steps as you would do it in Selenium, you can then write your workflows or tests to automate every operating system. We are currently running on Windows, on Mac, and on iOS, especially for legacy application and the Windows environment. We can test also these applications that we are not limited to web. Did this answer the question? It does. It does. Thank you. I appreciate that. It was a good explanation. So you obviously had a certain perspective of when you came to this problem because you had worked at Siemens, You had thought about this sort of automation. I'm wondering, as you've developed this technology and seen others that might have had this need or experienced this pain, what is the sort of range of things that you're seeing people either do or want to do with this kind of automation technology that might fit the use case, sometimes fit the use case that you had in mind initially, but sometimes might be a sort of new things that you didn't think about previously.

12:30I think currently, because we are using AI technology and with AI technology, you are a little bit more flexible what you can describe because you can learn it based on data and have not too explicit describe it, which things you want to do. New use cases are what we started where we want to do regression tests for classical user interface tests. Then we moved a little bit in the direction of robot process automation. I think where we would landing in the future is to automatically transfer unstructured data to user interfaces. So if you have, imagine you have a PDF, which in a formula, and then you want to say, hey, please, as normally normal workers will do with this, which are copying, for example, the PDF or information from the PDF over to another formula.

13:28And what you can do then with our technology is to say, hey, please automatically copy all the information over, which you see on the left side to the right side. This is where we think where the technology will go so that you can have not defined the matching of the files. Everything, when you then connect also large language models, which have the ability to understand the language a little bit better or a little bit closer as we could, as humans can define it, then you can build such systems. Yeah, that's really interesting. I'm thinking of even like in I had a friend that works in the offices, same office building that we're in here.

14:08And he stopped by and we were talking today. He he takes screenshots of his work as he goes along in the day, like every so often. And he does this because he wants like he can't always remember like all the things that he was doing throughout the day. And so he takes these screenshots as a sort of historical record of like the various interfaces and what he was doing and such. So I had that in my mind as you were talking through all of these things. I'm like, hey, this would be really cool for him. He could, you know, potentially query certain things about his day and these interfaces and what he did and like create potential automations off of those.

14:49This is also a use case where we're thinking to be a recorder in the background, but then collecting repetitive tasks and then say, for example, hey, we have now detected you have done this over the last week five times. Should we automate this for you, for example? Yeah. There are also cases where the technology can go because we have the understanding of YouTube interfaces and with a lot of engineering stuff around this, then you can build such nice systems. That seems like an automation, like automation and AI seems to like scare people for some maybe justified reasons, some unjustified, or I don't know what's justified.

15:27But that's like an automation that like, why wouldn't I want that to save myself some time? I would love to automate some of my repetitive tasks. So one of the things I'm wondering is you're having looked at your website is you talk about like all the different platforms, you know, you get web apps and enterprise apps and everything. Are you able to take the same approach across the different platforms that you're targeting for that? And is that what makes it so flexible by doing screenshots and stuff? Is that what does it or do you have? How do you break down the challenge of getting from, you know, addressing one platform in your initial and then starting to spread out across the other platforms?

16:05from the other the main technology is to accessing the screenshot from the operating system and including controlling it i think you if you have tried to do it there are a lot of open source software already available which can do this if you have such technology accessible then you can take the screenshot for example we have android support where we can take the screenshot and it's the same model and the same technology behind them as we are using this under windows operating system or under Linux operating system. And we have a general model which can solve all the tasks. So Dominic, I'm already thinking of a lot of use cases that maybe I might want to automate and interact with the system.

16:52One of the things we were chatting about prior to actually hitting the record button on this episode was maybe the unique approach that AskUI has taken in terms of more of a software engineering approach to understanding how these machine learning and AI systems work and utilizing them in a in this sort of like more practical software engineering approach to this. I'm wondering if you could talk about that in a little bit more detail and how you've approached these problems that you think is maybe unique, or at least represents your perspective on how to build systems like this? Yeah. First of all, what I see or what we saw already in the past, the research area has built a lot of models, has released them on the public, but then it stopped it.

17:46After you have published your paper, you have no interest anymore to bring this to production. What we also see in that currently beginning from the 2020s, that a lot of new applications are coming which try to solve this, to formalize this as software patterns as I know it at the beginning. I started with the machine learning and I was wondering, are there now software patterns like metric pattern, like trainer pattern, like other kinds of pattern which you can reuse and which you can communicate in a better way? why are we the good at software development nowadays is because we have standardized the patterns which we use and used them everywhere.

18:27So this is what I was where we're always searching a little bit and also the tooling. And how did we approach this from the machine learning? We have different teams a little bit. The first thing what we have done, we have built an application which used directly our model. And we used Plugin, the best of the first model, which we can produce. And we started with our model, which has only five images for object detection for detecting this five images training data set, which is not a lot. So, but we've done this to prove that this end-to-end possible. And we mocked a lot of things away. And then we go on out to the customers and say, hey, can you work with this?

19:11And they complained, yeah, it's nice, but your modular object detection model don't work so good. So what we've done in the next one, we collected more data, trained the better model. Then we go out to the customer access now enough. And then they were, okay, it's going the direction, but we could only support one application at the beginning. And then we had to relate it based on this and tries to connect all our things together. So what this means on the other side, we started directly to going out to the customer and then increasing everything. And now we are making, because this was hard at the beginning, because I mentioned already that in 2020, all the tools, Pipeline and Metaflow, and so everything came up, which you could reuse.

20:01And now we are migrating more and more to the data pipeline and trying to bring everything to the customer so that they can also train by themselves. But this is our software engineering approach where I say, hey, bring everything to the customer and let the customer complain. Then you're doing the next iteration steps. This is the last note. So as a follow-up, I want to ask you to kind of flip-flop what you just covered a little bit. And what's it like to engage from the customer's perspective? because you're kind of taking it from your perspective and we take it over. So if you're a customer and you start to use Ask UI and deploy it, what does that picture look like?

20:37What does the customer go through as they start deploying or start utilizing the service? What he's doing too, there's also a little bit of history, but what is this now? You're going now to our website, then log in with your credentials, and then you can directly upload your first screenshot and then simulate on this. So do a simulation directly on the screen show so that you see the aha effect. And then the next step, what you want to do, if you have created your workflow a little bit, then you want to automate this. And you send, then schedule this, for example, in a Docker container in the background we have already.

21:19Also, if you're in the web environment, and then after time, you get the result and you're happy that you have automated with really easy things, the workflow. And what we are now doing is to reduce also the hurdle that you have to learn less as now, because this is the thing that we're trying to bring all problems from the user perspective away. The user has a really easy life to create automations, to maintain automations, and to schedule all the stuff, set up all the testing environments because it's not only the automation part what you're interested you're also interested in where can you schedule it and all the broadizing how can you connect data and so on this is what we are doing what we are currently doing that's awesome i want to propose it what i think is probably a bad idea but i want i want to get your reaction to it so i'm wondering there's this way now that you've enabled people to automate their interactions with various UIs.

22:26And there's plenty of cases where I don't really care to interact with a UI, but I do need to accomplish a task. But I also like, for example, one of these that I struggle with all the time, right, is like AWS and its interface, which is just like you can do everything right but it's super hard to understand how to do anything right is there an opportunity let's say and again i think this is probably from the start a bad idea but let's say i just gave some sort of agent tied in to ask you i my credit card information and whatever and just said hey i want you to create a aws account and spin up this infrastructure and do this and then when you're done, like give me the URL so I can access my, like, here's the GitHub repo, tell me when it's ready.

23:23And here's my URL. Now I imagine that would also need to tie into like other external knowledge, like the documentation from AWS or something. But is this type of scenario, anything that like you've been talking about internally or see as things that might come about in the future, as soon as you start automating things around UIs, there's the one side of it, which I think we've talked about a lot, which is automating the things you've already done with UIs. But what if you want to do things with UIs that you haven't done yet, but don't really care to learn how to do? This is also our plan to do this to leverage that we already played a little bit around with large language models, giving all the documentation to do this, and also tries to translate, for example, Google documentation and create out of this Ask Your Eyes step.

24:16This was working quite good. And the direction will go there. So in the future, we can do this. So there was another part. This was the attention base. Please create me an ET2 instance. And here are my credit card information. This was an intention. Please do this. You can do this. But the main problem with this is humans, if you talk also to your colleague or to someone from another nation and so on, you will always have communication hurdles. so you have always to have the tiny steps between where you then can correct a little bit a little bit up this is one thing the other thing is uh are we trustful for credit card information for this i have to say um you're submitting also in github or gitlab your security tokens to your to your awa success so there are standards already in place which are also validated which can do this and this is what we have to follow we have also business also enterprise customers which wants to have the standards so we have to be compliant with the standards so from our side it's no no problem you can trust us but on the other way you have also the possibility because it's nice to create the tests online download everything and execute this locally on your machine without any access on our side you can also hide all the information all the secrets i would say it on your device and i have the guarantee that they are not leaked on anything yeah that was another thing i was gonna ask is like hey if i'm going to a site like let's just say i'll give an example because i i love the product and what they're doing um so chris you remember we had josh from koki on talking about their voice studio and all that i've been using that recently you go in you can input your text like a sentence and synthesize a voice and then you can change like the the language or something and then like go through export the file so there's like there's not just a ui interaction here there's like input data and output data that might include things like passwords it might also include things like hey i generated a file with this ui like where does that go like a quick download or something i don't know what's the best practices around as you're automating these things how should a customer think about these like inputs and outputs and how do you handle those in terms of what you're building i would forward this question to how are you normally doing integration tests or end-to-end test you have always the same data there so first of all to recommend customers try to use always synthetic or generated data.

27:00Don't leak production data to it because it's security. This is the first thing what you learn when you start automating and testing. Don't use production data. And then the other thing, you have to apply security standards. So there are environment variables or secret files, which you can inject to it. Go with it. Do it. Use our security function that you're not sending to us anything what's related to it. And this is what I could recommend to the customers. I'm curious, we've been talking about testing for a while and most of the use cases have been unit tests. Are there any other types of testing like integration testing that you're able to do?

27:41Is there a workflow for those kinds of things? Or is this really focused on kind of the screenshot itself and everyone stands alone? Is there any way to tie them together? Yeah, you can tie them together. what we are, we are only a library which you can use in TypeScript. In the future, we will also support Python and other languages to make this technology also available. But you can also combine this with Selenium or some other techniques. You can connect to a Mongo database, getting the data out, processing this in terms to other systems and so on. Yeah, we are really flexible in this race because our main concept, this is maybe also another thing, What's unique on our side, we are thinking that automation or user interface automation as a low-code user interface automation is to a certain level nice, because it can give the ability to other people.

28:37But on a certain point, we are reaching the limit. And for this, you need developers to build some nice stuff to automate or to connect, for example, to MongoDB or some other sort of stuff. In this case, you can always have the possibility to go from our low-code view to our code view and insert directly code. So there's no problem. You can do this. And you can also install other libraries.

29:09This is a changelog news break. PageFind looks pretty cool. It's a fully static search library that aims to perform well on large sites while using as little of your user's bandwidth as possible and without hosting any infrastructure. It runs after your static site generator, like Hugo, Eleventy, Astro, etc., and generates a static search bundle to add to your built files. It then exposes a JavaScript search API that can be used anywhere on your site.

30:06Quote, I'd love to see a comparison, link me up if you know of one, but my guess is that this could easily replace Algolia on lots of open source docs and websites. One less service to depend on, why not, right? You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com slash news.

30:44I always like to ask guests who have really manifested some new idea that's really driven fundamentally by AI and machine learning this question. And that's, you know, as you are building out this product, what challenges did you find in using? You already alluded this a little bit with like, oh, researchers release like all of these models. And then like what happens after that? They're not really supported or maybe they die off or other things. So what specifically were the kind of machine learning or AI challenges that you faced as you were trying to make this work? You alluded a little bit to the data side of things and kind of adding data over time, but I imagine there's much, much more than that.

31:33So what are some of those things that stand out, just practical things that you faced in trying to apply this technology to a real world automation problem? Yeah, maybe to start at the beginning, previously I was a software engineer. I had no clue about machine learning. I get a little bit theoretical knowledge, but theoretical is nice. but in practice, it's completely different. So I had no clue that, for example, if you adapt the learning rates, you can bring your model to convergence. So such things I struggled at the beginning with also how you connect the layers at the beginning to make everything work, but then you've solved this problem.

32:15Then the next step where it starts to struggle is to making the experiments visible so that you generate the right metrics, that you see, that you understand what you're learning or what you're not learning. Then when you have solved this challenge for yourself, so you have tried out a TensorBoard, you have the next challenge to go to how can I manage the data? How can I increase the data? How can I version the data? So, and then you came in the challenge, how can you do repeatable experiments that you can say, hey, you have done now progress step by step. And for this, you have to then to search a lot about what tools are out there, which tools are good, you need to know a feeling.