AI for Network Management with Shirley Wu - #710

0:00Recently, there is a pre-trained time series anomaly model, right? That's also exactly the need. But there's a key difference that existing pre-trained anomaly model time series is for single feature. What we are talking about is multi-feature. So, for example, you want to look at the throughput for your given site. You definitely should look at the number of clients and the total traffic volume we are going through and how many destinations they are going through and how many different applications they are routing the traffic through. So this is going to be multiple features.

0:47All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Shirley Wu. Shirley is Senior Director for Marvis at Juniper Networks, where she leads the data science team. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Shirley, welcome to the podcast. Hi, Sam. It's a great pleasure to be on your podcast. I'm looking forward to jumping into our conversation. We're going to be talking about the many ways that you're applying data science and machine learning to the challenge of ensuring high-quality network connections for your customers.

1:27before we jump into that topic i'd love to have you share a little bit about your background yeah so yeah and so i started my career as a software engineer focusing on the data analysis you know trying to find the insights from customer data that is beyond before the big data hadoop and the cloud um at that time there are a lot of consistent challenges regarding the data volume and the latency issue. Then big data, Hadoop, technologies and the tools and the cloud computing came to the world and quickly took over the whole industry for the data analysis. So the capability to process large amount of data really accelerated and the development and deployment of AI-related applications.

2:22So the primary application we're doing the business insights are definitely not enough anymore. We quickly moved to the capability to predict and forecast for customers' needs. So at that moment, I took the leap of my career to get into the startup. I joined the early stage startup in cybersecurity, trying to use AI data to solve the cybersecurity-related problem. So we are, you know, one of the first industry, trying to profile user behavior to identify security threat. So that company eventually was acquired by Aruba Networks. Then that's our first early stage startup. Then I came to Myst. Myst was a Wi-Fi company, but it's different from the Wi-Fi networking company.

3:14It is also a cloud company. So I joined the minister to start to develop AI-related technologies. So the company spent quite a few times, quite a lot of rounds of development to cloudify all the networking Wi-Fi access points and build the cloud infrastructures to be able to capture the data. Then we have data in the cloud. That's pretty much the prime time to develop AI ML solutions. So I was kind of architected with a group of very talented data scientists and engineers to implement Marvice self-driving network solutions. And currently, we continue expanding Marvice to the end-to-end networking, starting with MIST with Wi-Fi-only deployments.

4:05Now we are developing into the Wi-Fi, wired, and the van, trying to utilize the data and AI to help the customer to have a better networking experience, have a better internet experiences. When I think about networking systems, and I spent my early career at AT &T working on data networks and configuring routers and all that stuff. So I have some degree of familiarity with all that. You know, I think a lot about the type of data that, you know, those systems tend to generate. And it's mostly log files and like time series data. And to a larger degree, you know, while that space is evolving, you know, rapidly, some of the most exciting changes in AI haven't necessarily.

5:01you know included time series it's been about language and images so i'd love to have you share a little bit about how you think about the evolution of mlnai from a time series perspective and also more broadly you know networking as a application area for mlnai yeah so your question is right spot on why networking is really the domain suitable for AI and ML. First is networking is a mature domain, right? Not much of the significant, you know, revolutionized changes for the networking domain perspective. So because of a heuristic and the domain level, it's maturity, so that's capable for AI and ML.

5:55So this is the first layer. Second is the MIST. We started to develop the network gears, purely like an IoT device. So purely, we have a visibility about the stats and events from the cloud. So as the device started to power it down, and they auto connect to the cloud, starting to report its stats and events. So in the cloud, we have a large amount of data purely can clearly identify current state of that device. So that gives us a capability to do the AI and ML. Now, a third piece is about revolutionize from AI and ML perspective. Like you said, currently a lot of revolution development in the AI ML is focused on the image and the large language model.

6:53Time series, from people's perspective, it's mature already. It's a down problem, right? But for networking, it's another domain. If you talk about time series to forecast a particular stock market or for the advertising, for financial marketing, that pretty much is mature. But for networking, it's a special domain. A lot of the, you know, you go to market, you can find a lot of the people that have a good experience with data science and ML in advertising financial services, not necessarily networking domain. Networking domain is a very special domain. So it's really hard to find that type of talents, which they know the expert about networking and also about AI and ML.

7:44You know, when you think about time series as a problem class, a lot of the things that you're trying to do are very similar in terms of prediction. Are the problems fundamentally different in networking? Problem is fundamentally different. For example, for Wi-Fi, how the Wi-Fi mobile client to join a network, there are multiple phases it goes through. So this we have to sit together with our domain experts to understand multiple phases and each one of handshake between your Wi-Fi client with your access point, the acceptable latency, which layer they're going to retry. So this is a type of use case perspective is purely well-defined while we work in a specific use case with our domain experts.

8:35So this is one point. Second is, like you said, if we can really interpret the domain-specific problem with the data, then the problem is pretty much done. A lot of times the data scientist team, we're working together with the domain expert. first to identify the true problems they discover at the customer's site. Then we study the data to make sure our data be able to interpret, have all the descriptions about that particular problem. So in the data science perspective, they call it the feature engineering, right? So when we do the feature engineering, a lot of times we kind of really identify that, hey, maybe we even don't have the right data.

9:15So our device did not send that particular stats or events to the cloud. Then we had to go back to work on the AP firmware side to able to capture those stats, send it to the cloud. Then we started to redo the feature engineering. Then the third piece was that when we identify this type of issues, for example, for retail customer, because their Wi-Fi configuration is different compared to college campuses, which also could be different compared to the business offices. So we take that particular domain, the deployments as a problem. We also try to generalize for college campus and for the campus deployments, for enterprise campus deployments.

10:04So in this case, we have a generic solution, not just for a specific one customer, which is not going to work for customer B, right? So it's the three layers of the problems that we are talking about. Yeah, it's kind of lay out the problem. You've got these Wi-Fi access points that are spread throughout whatever facility and you've got these clients that are trying to connect and you ultimately want to have the clients have the best experience possible. One aspect of the problem that's kind of offline is there also an online component where you're deploying models to either clients or access points that somehow aid them in making the right connections to one another that facilitates a better experience?

10:52Yeah, there's multiple fronts, multiple layers, actually. AI ML, it's not just one solution. It's every front. So, for example, let's say I can give you multiple examples. So first to start is what is the feature you are talking about, the auto radio resource management. So when a mobile client is trying to join an AP, you need to make sure the client can hear the AP on the particular channel. You can hear the Wi-Fi signal strength should be optimal, right? So you deploy the access point everywhere in the parking lot, inside the roof, on the ceiling of your room, and in the dorms. So every environment that also, you know, maybe the other radio signals, you know, if you're close into the airport, if another, you know, if the shopping mall in the retail store, the next door is another access point maybe from the different vendors, right?

11:46So all this very dynamic environment, we need to make sure our access point have a right signal and on the right channel so that as a client for this particular store, they are able to hear you clearly. So in this case, we have the models deployed inside the cloud and they are adopted for each environment. And based on, you know, we will call the reinforcement learning. So we select a particular channel for every site. And based on the feedback information from the stats and the events, we continue to fine-tune the selected channels and the power strengths. So this kind of more like a global model, but being fine-tuned for each site.

12:37And also based on the feedback loop, based on the stats and the events data, we continue to fine-tune it for each environment. So it's kind of a wonderful use case, right? Of course, when we start this journey, you first need to do the offline analysis to define that type of model. Then we deploy it to the cloud. Then utilize each side data to fine-tune the model. So this is kind of a wonderful use case. So this use case actually compared to traditional, you know, networking company for Wi-Fi, they typically use a controller. Controller is a piece of equipment. they deploy it at the customer's site.

13:13Then you need the network administrator to manually configure that controller, right? So in this case, our solution is purely utilized cloud, utilized the data for each access point. We totally eliminate that extra components. So this is kind of enable our customer, for example, one of the largest retail customers, they have a very small team. they can support the Wi-Fi deployments globally all their retail stores. So our solution enables them to do that. Another, of course, we call it the AI for operation. So we use AI now to support the customer's operation. What does it mean? Is that you first get your networking configured, deployed, everything's great.

14:02Then our SC work away, so there's our networking running. But, you know, typically network hardware companies, they do is if there's a problem, your IT administrator just pick up the phone working with the support to get the issue resolved, right? For us, actually, after the first initial site going alive, the data and the events continue streaming to the cloud. We monitor your network operations. If there's something that's not right, we are supposed to be able to notify you ahead of time. So, for example, we are monitoring the site, number of clients, you know, the counts. This is just, you know, if you're the college campus, you can see daily trending, right, up and down.

14:50You go into the college room and go to the classroom, then they come home and back to the dorm. You know, the line is up and down, up and down. We can build that line to build the anomaly detection model for every site. We can notify you of a normal sudden drop of a client count that definitely indicates something is not right for your given site, right? So this is kind of another, back to your question, this is a time series detection anomaly model. But here's a challenge for us is that how you want to make sure your time series anomaly detection model is accurate Think about our scale we're running.

15:32We have close to 100 ,000 different sites deployed across the world. And if we have every site, we have a anomaly model, which is a neural network-based model. So which means we're going to have 1 ,000 neural network-based models. This is too expensive, right? One model is fine, but this scale is too expensive, too costly to maintain. So cost also is another factor. Efficacy is definitely the number one issue, but cost is also second to the most important factor to make a decision. Can you elaborate on that? What's the relationship between a single model and the cost? So, for example, let's give an example.

16:24Say, hey, for the retail store, you have a daily pattern. You know, the customer going to your store to visit. Daily pattern probably is a weekend should be a higher volume versus a weekday, right? And the weekday during the normal business hour, probably not many people go to retails, go to your store. Probably going to be even time. But versus for the - The university is going to be almost opposite, right? Yes, yes. And the training is different. So in that case, you can't have a neural network-based model to learn your data pattern for the retail store versus for the university. if the pattern is totally opposite or totally not in line with each other.

17:07So in this case, the best efficacy training was that for each, you know, the nine, each trending nine, we created their own neural network model, right, to the forecasting. Same thing, you know, like say, hey, for a stock market, you want to predict a particular stock market, stocks trending, you train a neural network. But if you want to follow multiple stocks, they're trending. You probably need to train multiple neural networks, right? So we talked a little bit about time series as being somewhat of a, you know, solve problem, quote unquote. But that isn't to say that people aren't trying to create transformer-based models for time series.

17:50I'm curious if you, you know, to what degree you've explored or found any interesting solutions there. Yeah, it sounds like that could be interesting for your problem where you've got some base level model that you can fine tune as opposed to creating many distinct models. Yes. So that's exactly what you know here. Recently, there is a pre-trained time series anomaly model, right? That's also exactly the need. But there's a key difference that the existing pre-trained anomaly model time series is for single feature. what we are talking about, multi-feature. So this is actually, that's another piece.

18:34A lot of times, recent development advancement is only for single feature. Got it. So you've got a log of stock prices, and you're trying to predict future stock prices, or you've got, in your case, a log of RSSI. Would that be one of your... Exactly, yes. I forget what that stands for, but it's a measure of noise. Signal strength, yeah. And you're trying to predict the future RSSI so you can figure out which channel to join or something like that. But I'm guessing that you've got this log with a lot of different features and you're trying to make multiple predictions based on as opposed to a single one.

19:17Yeah, yeah. So, for example, you want to look at the throughput for your given site. You definitely should look at the number of clients and the total traffic volume we're going through and how many destinations they are going through and how many different applications they are routing the traffic through. So this is going to be multiple features in the real world problem. So this is also the reason why the pre-trained time serial anomaly model is very promising, but unfortunately it cannot work for us as a single model. I'm relating it a little bit to my personal experience. I've got a small network here with multiple access points.

19:59And I remember when I first installed this latest version, I would go into the management software and look at what client was connecting where and to try to address performance issues. And I noticed that the decisions didn't really make sense to me. I would be in one room and the laptop would connect to a distant access point. And I'm just thinking about if it was that complex for me. And I was never able to resolve it. I just had to wait for the software to get updated and it kind of resolved itself. But if it was that complex for me, like at the scale of a campus, the complexity multiplies, obviously.

20:48And so it seems like an obvious place to add some intelligence, I guess is the point that I was getting at. So it's another thing that is, for example, we wanted to transform a networking. Just networking previously, just like I said, you need to really domain expert network administrators. They do a lot of troubleshooting, doing a lot of cool stuff to find the data, to correlate, to find the root cause. Because that domain is mature enough so that we can write the software to embed those domain experts into the software to help some of the no-handling fruits. So the network administrators, they can focus on more complex problems like security-related, like a large deployments configuration.

21:41So we are also transforming a lot of network administrators from just a manually cool shell guy and understand the network gears to be API engineers. Because we export all our data to make it available through APIs to the customer. So they can utilize the data to develop use cases for their support, their own specific environments. So it's a key difference here as well to empower our customers. You mentioned in your earlier explanation the use of reinforcement learning. That is sometimes associated with using simulation. Is simulation a technique that you use often there? Actually, we did not really utilize simulation.

22:33We just test our own. So our office is a test lab. So we are all part of our test, you know, our latest algorithm to reinforcement learning. Reinforcement learning is a permit you develop that, you know, the awarding function, right? What is a positive or negative award? So this is the one that we continue in the cloud. We always do this type of AP testing. So first, put a smaller, friendly, different type of deployments into this latest algorithm. So after we have a confidence, then we deploy it to the MIST. We call it the MIST universe, which means globally to all the cloud environment. And so MIST is kind of an umbrella term for all of the Juniper AI-related capabilities?

23:25So actually, MIST was a startup. you know, actually this week will be hit our 10 years anniversary. We started as a Wi-Fi, only focused on the Wi-Fi. But the next is Wi-Fi is different. It's a Wi-Fi access point. It's totally cloud, cloud-driven access point. So about five years ago, Juniper acquired the MIST. So this acquisition is a benefit for both sides. You know, for MIST, we used to use Wi-Fi after we parted the Juniper family. We got the switching, wired, and also get the VAN gateways. For Juniper perspective, Juniper does not have a Wi-Fi, never been a Wi-Fi company. So now Juniper got a Wi-Fi.

24:10And also Juniper is not a cloud company. So now it's a cloud. And of course, Juniper continues to utilize the most cloud technology and the AI ML to try to expand into not just for enterprise, which is wired in the VAN, and also to data center. So this is kind of currently the five years after acquisition. Then we are going to be soon embarking another journey to be part of the HPE family. But the acquisition is the same goal. We wanted to continue to utilize the MIST's AI ML solution for networking to be exposed to the much larger networking deployments. I came across one of the initiatives under that umbrella called Marvis.

24:56Can you talk a little bit about Marvis and what that is aiming to accomplish? So now, you know, one hand, we have a lot of bank-hand jobs running, right? Conscious numbers to process user data, to generate alerts, events. Now, second is how we want to make sure, how we want to change the user interact with our application with their data. So we utilize Marvis. Marvis is, think about it, actually Marvis where it came from is from Jarvis. Ms. Jarvis? Yes. And it's a bot. They can understand a lot of languages. And we replace the first letter is a mist, right? That's why it's called Marvis. So first, you know, Marvis is how users can interact with their networking, their network issues.

25:47So we first started with troubleshooting. so you can always go to the UI, just troubleshooting for a particular client, particular device, or particular issues. That's early days, you just commanded to do the troubleshooting. Now we evolved into a chatbot. So you can just go to the UI, you can just go to the chatbot asking questions about why I cannot connect the Wi-Fi, why my AP is continuous rebooting, why my AP was disconnected from cloud, right? Why my AP, can you, how can I do the AP firmware upgrade? So we added the chatbot support. But the chatbot support actually was developed even before this whole large language model took the world.

26:39So now we are sort of another, you know, enhancement to our current chatbot to integrate with large language model. and you know recently you had a great conversation with Nantium CEO, Nantium Nantium, so actually we'll integrate that Nantium graph into our chatbot so and also you know you could go to Marvis to ask questions about a specific networking domain or Marvis or Juniper related product questions instead of we give you the documentation to grade three, we can summarize the specific answers and provide a real precise response to you. So this is kind of, you know, just the way, first we develop our own RAC solution to be able to do this type of summarization, focus on our public, you know, documented in the release notes and documentation pages so that we can provide a precise Juniper-related product features to our customers.

27:51So you don't have to read through all the technical documentations and the wiki page for yourself. This is one part. Yeah. Second part is that we want to utilize large language model to better understand what customers asking and also develop our own text to sql text to yes type of solutions because all our data inside the sql database or elastor or the non-sql database we so before in order to expose those type of data we have to create a lot of different apis while we just use a large language model help us to expose all the data together with the customer, right? Got it. So based on whatever the customer's intent is, you turn that into a query against the database?

28:41Yes. That's another use case. Because as of now, we are not, we're committed to not sending the customer data to large language model. But we actually utilize large language models to create a lot of embeddings for our internal metadata. So for example, how we store the data in our database. We create this data catalog, or we actually create an embedding and also provide the RAC solution so that we can match the customer, what they're asking with our data much better. So a couple of follow-up questions there. First, you mentioned working with LangChain and LangGraph. I'd love to hear a little bit of your experience and motivation there.

29:32I talked to, there seems to be bifurcation. Some folks appreciate the kind of standardization, if you want to call it that, that the LangChain offers. Other folks like to go straight to the metal and kind of do it themselves. and skip an intermediate framework that might hide some of the either complexity or features that they need to know? What drove your decision to use that? So currently, our integration with NAND graph, we did not go fully. So for example, a lot of large language more related features, we build our own smaller model. So it's kind of multi-model type of a scenario in order to, because like I said, first, networking is a very special domain.

30:26And a lot of, you know, networking jargons that our fine-tuned model probably going to do better. But on the other hand, some of other questions that the large language model can help us to close the gaps. So we actually running this as a large language model, our own fine-tuned model is running in parallel. So because of this, we does not go full in into the name graph to handle all the dialogue and also the assistant. So we kind of build those tools a little by little. Like I said, we feel don't have a confidence that we want to all in with the name graph. Like exactly what you said, you have some customer some of our audiences, they want to go all in with Niagara.

31:16Some of them just, we are probably the second layer. We wanted to have our own control and do our evaluation. Then we kind of integrate with Niagara piece by piece. Got it, got it, got it. Yeah, I hear that quite a bit. And then the other question is with regard to RAG. You've mentioned that a couple of times. I have talked to many organizations that, you know, start down this journey with RAG and primarily expose it via a chat bot, but find that, you know, kind of going beyond that chat bot and integrating more tightly into existing, you know, workflows and user interfaces without requiring that chat dialogue experience is a source of even more value from that.

32:12that from the RAG as a technology? I'm wondering if that's something that you've seen, and if so, what are examples of those workflows that you're looking to enable? So, for example, for us, when customers try to open up a support ticket, before they open the support ticket, when we look at their questions they're asking, the problems they're dealing with. So if it's questions about our product and the features, then we just use Rack to generate response for them. Just another layer to say, hey, maybe this is what you want. You don't have to create the support tickets. So this is kind of a lot of integration with customer support.