Closing the experience gap - Interview with Qualtrics executives from X4

30:34Now, once you understand that new lens, then everybody has to go through the transformation process of, well, what does that mean for my business? So that would be the biggest thing is just how can we get the new... I hate to... It's overused, So I hate to use the word paradigm, but it's the right word to use. What is this new paradigm that we find ourselves in? What, how does this work? What is different now from my business in this, in this age? It's like, I tell you what, it's an alternative way of describing it. A new operating web. There you go. I like that. That works. Because there's a kind of a practical thing.

31:07Yes. Yes. So being able to do that re-imagining, then the next biggest challenge everyone's going to face is, okay, I can stand up these new novel ways of doing everything. also i have all of this existing stuff that everyone is depending on that doesn't go away yes yeah so how do we navigate from here to there how do we bring everyone along um part of what you will find in a lot of the industry broadly right now is upstart startups don't have that any of that legacy so they seem like they can go faster yeah And people get worried about that because it's like, oh, well, they can be so nimble in doing all of these things.

31:49They also don't have the giant established base of... They also don't have the brand equity. Yes. And they don't have that sort of like surety that comes with that, which is where I always think that people get really antsy about, oh, there's this disruptor. And then that means that they think they're kind of all their customers are going to like switch kind of like tomorrow. Yes. It's a bit like, you know, back in the day, I mean, this is quite a relevant thing for call tricks. You remember the time when feedback or asking your customers for feedback was like a new thing. And many companies resisted it because they thought, well, actually, why would we ask our customers kind of like what they think?

32:30Because they're just going to shout at us. Yeah. And you think about it and just go, well, that's ridiculous. Yes. Yes. it's like why is it not a line of people down the street knocking on your door getting ready to shout at you it's like one of these false expectations appearing real right yeah no because managing that that sort of that that transition because the um it's no different from i think in any other sort of transformation because actually you've almost got to it's sort of like twin track things there's the new thing and then there's the old thing and then you've got to find trying to migrate more of the old thing onto the new thing, re-platforming and whatever it might kind of be.

33:12And so it's an old and familiar challenge, whether just a new... That's right. This is no different from previous tech waves where the same kind of thing happened in a sense. What I would encourage people to think about when they're starting to go on this journey is there's going to be this, and this is also true with previous tech waves, so this is not novel too. There's going to be this pressure where it's like, oh, all of this, the world will be different in six months. If we're not there in six months, we're toast. The reality is we can put the technology out in six months. That doesn't mean the world moved to the new technology in six months, right?

33:55But we're a year or 18 months or whatever it happens to be. We all collectively have to make that transformation. And that happens at a industry level. That happens at a society level. That's not a technology level thing. The technology is just there to enable it. And so I have to encourage even people on my own team, when they get very excited about the newest, latest technology and how it's going to change everything, and I'm like, it will. Also, we have to make sure that we are keeping pace with how quickly that can be adopted, how quickly that transformation can happen. though yeah it is happening faster in this wave than perhaps in some of the previous ones but it's also not happening at the pace where all of this is going to be completely upended six months from now so no i think i think there is a reality where it's like some people are not as far advanced or as ready to change as the industry commentary you kind of might suggest of course and and if we know about if we think about two other sort of technological ways i think about the dot-com kind of like boost uh boom and bust sort of thing back in the turn of the century the thing that struck me out of that was that there's when you see things rise and then fall is is there always there's always been a difference between what probably will happen what possibly could happen and i think the same probably holds kind of here you've got to kind of almost i'm not saying you should dampen dampening excitement and ambition but there's a reality of like some people can't just go from here and make that quantum leap.

35:29And you almost have to take like steps at a time and be realistic and understand it's like, it's a journey that you're 100%. The other thing that I would encourage people to think about, because this has been very consistent throughout technology waves over many decades now, is people consistently overestimate the near term and underestimate the long term. Sure. It's very consistent. so as much as you're concerned about oh if we don't do this in the next three six months we're toast also the things that you're envisioning for two years are probably it's probably going to be even crazier than that than you're thinking so we all want to be on that journey together we all want to help do this transformation and realize all the new things that we can do but it's a collective activity yeah perfect thank you yeah my pleasure love me yeah well that was another great conversation.

36:23You know, big thanks to Mark for sharing some really fantastic insights. I mean, a couple of things that really stood out for me for my conversation with him include how AI doesn't understand your business yet. And how context is the new gold. I mean, lots to think about in that conversation. So without dwelling on it, let's get into my next chat. And this time I'm going to be speaking to Asaf Karan. This is the next segment of the podcast I've been recording at Qualtrics X4. I am with Asaf Karan, I believe. Is that right? The chief security officer at Qualtrics. Asaf, tell me a little bit about what you do.

36:58Chief security officer, that feels like you might be in charge of bodyguards or something. Actually, I also... Okay, yeah. It's like a two role. A two role. My most lamented role is telling execs that they need to actually have protection. Ah. But that is a side, not a side, let me take a step back. My role is to protect the company that is inclusive of the physical security of the operators and facility. It's a broader thing, not just at a technology level. Yes, and enable the company to serve our customers in the sense of building a layer of trust. And we try to look at it in a holistic manner.

37:44So cybersecurity is one element of that, and physical security, more employee forward, but there's a second layer of that in the sense of how do we think about creating an environment where our employees, our customers, our company is protected and we're enabling the business to do what it needs to. I love that you've mentioned the word trust. So I'm really interested in just service experience and the things that we do to try and help people have better service and better experience. And time and time again, in the research, it's shown up that trust is one of the biggest barriers to adoption of new technology, both at an organizational level and also at a customer level.

38:34And I wonder if, one, you agree, and if you do, then how are you approaching building that trust at both the organizational level and also helping organizations do that at the customer kind of level? Yeah. Well, I have a really good, I've had a really good playground on, or not a playground, a learning experience for my career on trust because I have, in my history, I've led the building security for the Israeli government. Okay. which is the governmental service organization, which is highly dependent on trust. And then I was a chief security officer in PayPal, which is entirely a trust. It's about money, but it's more about trust.

39:21Well, exactly. It's a bit like you're applying the same values and demands to something like PayPal that you would do to your bank. So you can go and you give them the money and you're like going, I expect you to look after my money. Yeah, no, PayPal succeeded because, let's take the last 20 years out of the equation and let's look at the first three years of its success. It was a brand new internet thing. You didn't know what to make out of it. You didn't trust any of it. And then there came a company that says, we will take care of you and build trust with you. And we'll be the bridge. And we'll be the bridge.

39:59So you don't have to put your credit card everywhere. We'll be your credit card provider. And if something happens, we'll take care. So the impetus of trust in that conversation, I want to avoid monopolizing trust as a security concept because it's not. It's security, it's availability, it's reliability, and it's resiliency. And it's also about taking care of your customer when you need to take care of your customer. 100%. And I think that at that point, people did that very well to build that rapport of trust. Now, when you think about a company like Qualtrics, one of the reasons why I'm here is because a good customer experience program actually allows, if you do it right, allows you to build trust with your customer because you know where they're hurting and you're able to do the right thing to support them.

40:54Yes. And I think security has a lens to play within that role of building trust. Now, when you think about B2B, which is where most of my operation is, when I think about how do I represent Qualtrics Trust, the company, in front of our customers, most of our customers are businesses, or all of our customers are businesses, and because of that, it's a lot of B2B work. There, it relies a lot more about, in my mind, about transparency. It relies on certification, and it relies on showcasing that you're doing the right things for them. And when there are disagreements, what I found counterproductive is to agree to something that you're not going to...

41:45How do we understand that? If somebody comes and says, I want you to do X, and you think it's the wrong thing to write, and coming and saying, okay, we'll do that for you, and you do a one-off for somebody, I think actually destroys trust in a long-term perspective. Actually being able to come in and say... It's an integrity type of thing. It's an integrity, but even more than that, what you are asking us to do is counterproductive to your security, right? We do not want to do that. And we want to sell, but we will not do something that is counterproductive to your security. And sadly, I've had more than five these examples this year.

42:21Right, okay. And saying, we will require you to do, or not require, we will not serve you in the way you're asking us to serve you because we think that that is not okay. You should be doing the work, not us. And sometimes that actually gets in the way of a deal, which is fine. But I think long term, we build a better rapport and conversation with those security. Well, you know what you stand for. Yeah. And one of the things I've done when I got in, we had a team that just the nature of our security services in B2B. You have a team of people that were doing questionnaires. Okay. So they would wake up in the morning.

43:01They had 14 questionnaires from customers. They had to fill them. They would go to their body of text and then copy and paste and copy and paste and copy and paste. And we actually enabled AI to do most of the copy and paste. Right. and now that team is doing what we call customer trust. So they are a customer engagement team that talks to the security team, that talks to the security team and creates knowledge and publishes white papers and best practices and helps drive the conversation between our security practitioners or our security customers and the product team and where there are misses and really build that rapport because that is a massive area of building trust with our customers.

43:45Even to the CX practitioners, helping them navigate their security teams because in the larger organizations, my role is to help the CX practitioner to get to yes from their security team fast. I mean, I think if you're operating at that level with your customers, so it's largely B2B, if you're trying to help the CX practitioners within their environments so they can deal with their own security team. But at the same time, you're trying to help the CX practitioners within your clients to actually help their end customers. It all kind of flows down. Yeah, it all flows down. And you said something that's really, really right.

44:25And I think that in this day and age is very interesting. It's kind of a dichotomy that we live in where people are binary on who they trust. Yes. So if you trust a vendor or a brand, you will trust them and you give them all of your data. But if you don't trust them, you will not give them any of your data and not use them. And now because AI shifted so much of the industry, it's shifting all the time. So people are reestablishing the barriers of what they trust. Maybe in two to three years, it will be a lot more contained. Maybe in two to three years, every customer will have their own agent.

45:06And then they'll be like, the agent will trust. they'll be having a graphic equalizer around kind of who I trust and in what domain and what sort of thing. And then that will just change the environment completely. Yeah, but that's the way you interact. In the end, I think that there will be a new barrier of like who you trust because out of the bat now, if you look at the new AI hyperscalers, OpenAI and then Entropic, people trust them now. People share a lot of data with them. right or wrong, we'll see in three years is there are more incidents, breaches, privacy, legal, all those. I'm not saying any of them, not only names, but I think this new frontier is shaking a lot of that trust boundary.

45:49But we're also, we do the CX trend report, and what we're seeing is that people want personalization. They are looking for that hyper-personalized experience that AI can give them. By the same time, they're really worried about what's going to happen to the data. But here's the interesting thing. So that phenomenon is not new. About 10 years ago, there was some research that came out, and it was the same sort of stuff that came out when we had the explosion of big data and analytics about 10 years ago. I remember he was like, wow, that's going to be amazing. But I'm not so sure. I'm going to give you all this thing.

46:22And Forrest came out with a piece of research that they talked about the personalization paradox. They said that there's, what, 75 % of customers at that time were saying, yeah, we want all of this. But then there was nearly 60 % of them said, but we're really kind of nervous about giving away our data and the security there of it, and maybe sort of like cyber breaches or identity theft or just being generally intrusive. But then there was also another layer in that with another piece of research that says that whilst marketeers wanted to try and deliver more personalized and kind of experience customers really re-agreed many of the many of the marketeers about was it 60 percent of the marketeers admitted that some of the tactics that they'd employed they would they would have felt they would have characterized them as being creepy so they'd almost like overstep their own bounds yeah and so there's an ethics their professional conduct thing within this as as well because I can't remember who was on the stage I think it was Sandra from the LA Raiders said that trust is increasingly today trust is hard to win but so easy to lose and so it'd be fun it's like what would be your advice to people listening in and going like oh yeah trust is the thing and security and transparency and conduct and ethics those are all just inputs into this cloud that is trust, what would be your advice to them to say, because it's a journey, you've got to build up the muscles as it were.

48:04Look, I think I used to have a chief risk officer that used to say, look at it as if it's going to be published in the Wall Street Journal. Okay. And now, incidents will happen. Okay. and there will be some sort of mistake that somebody will make and the model that we built or somebody else built will give the wrong answer or there will be something that will be left open, etc. But if you do the right things, fundamentally do the right things, and you communicate well and you're transparent, and if in case of a trust loss incident you say this is what happened this is what we're doing about it this is how we're fixing it and this is how we're making it right for our customers and you did genuinely not trying to crisis manage the situation or even or even shove it away yeah so you don't ignore it then then you you you turn the page on an opportunity to build more trust even though you had something that happened.

49:14Now try to do your best because what you don't want to happen is when the Wall Street Journal reports on the facts of the incident will look like you didn't do the basis. Sure. So striving to do the fundamentals well, striving to do all the checks that you want to do. And if it's something that's genuinely surprising or something that is genuinely you've done your best, then then the world will forgive you and will will work with you yeah if it's not and people perceive you to be disingenuous in that kind of situation that's the biggest trust loss yeah driver i think that's that you know that's fair it comes back to the the the best gauge of it is i think um if you ask yourself how do you who do you trust in your personal life what's the characters what sort of behavior do you trust and it's the people that are credible that reliable that are steadfast that are honest that are open they're transparent those sort of things however much you might not like what they say but at least you can go yeah i get what i what what i see and when they make a mistake you know that a mistake and not a pattern and they admit it and they're willing to have a conversation about how to make it better, right?

50:35And I don't think that we should treat our customers any way different. I think a lot of companies get into over-promise, under-deliver in that space. I love when websites says we have bank-grade security or military-grade security. I used to work for government. No, don't say that. I know. Yeah. No. That was worth it. So I think that, I mean, that's, I think you're absolutely kind of right. I mean, there's some fundamentals kind of things that do. I mean, trust is, trust is an issue and it's real, both at an organizational level in terms of technological adoption and security and privacy and all those different things, that it's real at a consumer kind of level.

51:21And to ignore it is to ignore an opportunity, I think. And therefore, you have to lean into it. But then I guess one of the things that you, it strikes me that companies can do is rather than just talking about trust is be more trustful yourself. So if you embody the behavior that you want to build, then you're more likely to be successful. Yeah. I think, especially in this day and age, I think I said it in the morning in the session, unless you are of a cohort of up to 10 companies around the world, you will have to use third-party vendors. and building a relationship and a partnership with your vendors and really coming to it as a partnership and building this ecosystem of environments that is going to help you build a product that will drive trust is really important.

52:27And that means that you have to treat them in the same way. You have to identify which companies you want to work with. You want to make sure that they share your ethics, you share your transparency, they share your care. But then it becomes a shared fate. Because much like, I think we went through this in the cloud transformation, when we shifted from shared, everybody had their own servers to shared fate, shared accountability, even more so now. Like the models are the models, and an impact based on the model for someone. One company is going to destroy the entire chain. not just by the way not just the entire chain of company a using model a using hyper build by hyperscaler yeah is that every other company using this model now will have to deal with it with the with the after effects of what happened there yeah i think we got a really interesting glimpse into it in the recent wars between anthropic and the pentagon yeah on and i'm not taking any sides in that, but what happens when you have somebody like us, which we provide services to the US government, what happens when they say you can't use anthropic anymore?

53:47How do you manage that supply chain? What is the trust factor? What do other people think about that? Well, then if you think, if you kind of zoom out and look at it kind of like in a much broader sort of geographical sets and you start thinking about sovereign data and sovereign ai and sovereign software yeah and all of that sort of stuff and then it becomes even more complicated yeah i think it's already complicated well my team yeah but it will be it will be i think the there was this attempt at national clouds um which still is a an attempt in some places. It's not the best pros and cons. Data sovereignty is a pro in places.

54:38The resiliency problems it means with it are a con in other places. It will be very interesting in this day and age where the data has become the new moat, not the model. I don't think the models matter anymore. I think the models are very interesting and nice and great, etc. And we'll all use them, but they'll be commoditized to a point. And every other month we'll have a new model that will be more intelligent and better and faster, etc. And this is great. Well, almost like Moore's Law, we'll become the new chipsets. We'll become the new CPUs that we operate on. The data is going to be the interesting piece, but now you're getting into a real problem that the industry needs to solve, which is how much data can you actually use to train on when you're a company it's coiffing data, how much belongs for the customers, how do you drive those things how do you create the shared value for both the customer and the company to build that capability and overlay on top of the sovereignty issues and then you get into a really complicated because particularly you've been talking over the last day or so about so synthetic data and how you create that um or synthetic personas or synthetic data but you're training it on there's this real data that's been created by doing research with actually human beings and then you can look you can model that and then then extrapolate from that but then you've got to keep refreshing kind of that and you're looking and go that's a lot of data yeah and a lot of permissions around kind of using the data and then as you say creating than the value can eventually that.

56:18And it's like, you can only do that by either being explicit about permissions and about ownership and about value sharing and all these things. And that has to be about partnership. It has to be about partnership. Because it can't be done in secrets. It can't be done in secrets. No, no, it's not done in secrets. Specifically, synthetic is an interesting one because we've done a lot of first-party research. We actually went out and did the research to build the model for synthetic, which is our data. Is that quant or qual or both? I think it's both. But now you need to ask the MR team. I'm going to speak to Ali tomorrow because I've got to ask Ali about that.

56:54But the general statement stands in which, and we never use direct customer data. We always anonymize and aggregate on top. So there is nothing that is specific to a customer in the data. But still, in order for us to provide better value, we need to use that data. And he is always a conversator. on what value you're driving for us and how are you using this data and how we're learning. And we have customers that opt out from that. And that's fine because we want to have the conversation. We would love to get to. Is black cat approval to use all of that? Yeah, but coming from a place that the value is good and yet they trust us.

57:35And that would be my goal. My personal goal as chief security officer, where I succeeded, where all of our customers approve us as a vendor within a week, and that they trust us enough to say, we trust you to anonymize and to aggregate and to use this data to give us better outcomes. It happens most of the time. Most of the time, a high percent. But we still have a subjective gap to do. That's great to hear, Ren. but thank you for sharing some of your thoughts and your insights with me. I mean, it's interesting because I think trust is this massive thing and oftentimes you don't really dig into it sort of thing and it operates at all sorts of different levels.

58:24So thank you for sharing your feedback. I'll tell you one thing I'm fortunate is that the company gets this and most of the places I've been in my journey get this. So it's very easy to have a conversation about how are we building trust in the platform. It's all an afterthought. Yeah. where I talk to some of my peers in the industry and it's not that clear. Yeah. I've been fortunate in my career so far. Well, congratulations. Yeah. Thank you. Thank you very much. Thank you. So, I also, I mean, again, another conversation which I really enjoyed. And I particularly really enjoyed the insight that Asav gave us into his role and his views on trusts and particularly the role it plays at both an organizational and a customer level.

59:05And again, before I start going down a rabbit hole and musing further on that, let's keep going and get into my next and final chat. This time I'm speaking with Ali Enriquez, who again is the Global Senior Director of Market Research at Qualtrics. And this is another installment. I'm super pleased to meet Ali Enriquez, who is the Global Senior Director of Market Research at Qualtrics, who I've seen on stage from afar, previous X4s, but now I get to sit across the table from you and ask you and say, well, welcome to the Punk CX podcast. Wonderful. Glad to be here. You're welcome. And now as the event is coming to a close, I guess, we're on the downhill slope now.

59:45Tell me, what's been some of your highlights from the event so far? Yeah, it's such a great question. This is my easily 10th X4. So I was client before. I've attended when it was called Summit. Right. And now X4, not in my home city, which is mildly annoying. but my Salt Lake City baby. Right, okay. But I've gotten over that. It's funny that you mentioned as we first met the evolution of what X4 has meant. There are a couple of things that I think are unique to us and will never go away. It was like Dream Team and things like bringing some really high impact, just cool stuff to our attendees that I've come to love and enjoy every single time.

1:00:33um what i i'd say this one's different for a number of reasons first time in seattle it's a beautiful venue some it's new to all of us i think it's attracted a new attendee base to slightly different so i you know salt lake isn't isn't maybe as accessible to everybody and so there's a lot of great tech here too um so i think it's been fun for me i've met a lot of new clients that i've just never come across before from a content perspective i think we are striking a very different balance we're not talking um inwardly about what we're building and how we're building it and about and there's pizza brad about that and he said when he did his keynote he was a bit like i feel slightly nervous i'm not going to go on and do a demo isn't it strange yeah every single time you know he's shouted out the poor se who's typing live on stage right um we've all been there right seen brad do that it was a very different tone and tenor for him but um and i haven't gotten feedback yet, but personally found it more relatable, right?

1:01:32The types of things we're talking about. Over the years, too, we've maybe broken content a bit too much into silos of even on main stage, you know, CX, EX, and MR. And instead, you know, we're focusing a bit more on what unites us. And our framework of listen, understand, act is something that we can all apply to our day-to-day. So I think we've come a long way with that and finding more unifying content. There are fewer celebs, which is a little bit sad, but they're incredibly inspiring leaders in different organizations. I know the Raiders, Sandra was fantastic. I love to leave inspired by new clients as well as just new faces and voices on stage.

1:02:16Absolutely. I mean, I think it struck me, I'm going to speak to Brad about this because I was really interested about the change and sort of direction, I say in inverted commas. But he said, well, actually, we're almost like drinking our own champagne or eating our own dog food because you got feedback from last year and people were saying, hmm, actually, we need to see something that's a bit more relatable. They're almost like cases that are like us or maybe just in front of us and that we can aspire to that. And I think that's fascinating because actually, I think there's a mood in the market.

1:02:47That's what I'm seeing, is that a lot of people are excited by the technology kind of development and the possibilities of it but sometimes it feels like it's a bus that's accelerating off into the distance and then like going did i just miss the bus or we're not ready to get on the bus and it's a bit like aligning yourself with their kind of journey as it were which is really i think it's a really interesting ship and actually quite different to other vendor events i think so that's good but the thing that i wanted to get slightly nerdy about because there was an announcement that I also ran particularly in your area which is MR research um and about synthetic panels which so I hate the name because it feels a bit west world or or kind of like Blade Runner or something no it's gross I mean it's gross right I like simulations right okay so tell me about that and how they kind of like work because I know there's lots of talk about people creating kind of data, synthetic data, particularly to populate kind of these big kind of models, but there's more to it than that.

1:03:56I want you to understand how we understand about that because just to see how valuable they could be. Yeah, I love it. Synthetic's not our word, right? I mean, we've had synthetic oils and synthetic materials. And so sadly, it just means fake. And that's not accurate, right? So if I think about, if I rewind, right? 20 years ago, I was a customer of Qualtrics. I was at Royal Caribbean. I'm telling you this story because our history is really important to not only explaining what it is and how it works, but how we've interacted with our clients over these past couple of decades. So when I was there, I have my own database of customers' guests, right?

1:04:36That I could reach out to and ask about just about anything. But I said, you know, I'm really curious why travelers don't take cruises. And so I need access to 500 prospective cruisers. And I called my panels rep at Qualtrics, and that's the birthplace of the team that I'm with now. And said, yeah, no problem. We'll get you this from third-party panel sources, right? So that's how I get access to audiences I don't have in my database. That's really, I think, the cleanest way to even define and describe what market research is in our terms. And so the nature of the work that we've done with clients and commissioned ourselves over, you know, these now two decades has something to do with advancing a product or a service, right?

1:05:16Just like I was, you know, what destinations do you visit? Do you stay in all inclusives? And, you know, what do you think about this type of package or, you know, private island experience? And how should we price it? Something to do with competition, right? So sticking with the royal example, how often are you going to casinos? How often are you going to the, you know, and why? And why do you spend money there and not with us? And then finally, audience exploration. I want to understand the multi-generational travelers and, you know, why they choose what they choose and, you know, what motivations and behaviors are unique to them.

1:05:50So that's the nature of the research that we do in partnership with our clients and, again, commission ourselves for our own thought partnership. That's what's then used to train and tune the model. So the way that the model works is at its core, very thin layer of publicly available LLM. So it could be Lama, it could be Claude, it could be Chowchee PT, one of those. It's thin, it's important context, it gives us, you know, worldview, bigger, broader than this next layer, which is 90, 95 % survey change data. Right. Across those three camps that I was just describing. Now, extrapolate that across 20 plus industries, right?

1:06:31hey, we've got automotive and we've got consumer packaged goods and we've got child and hospitality. That's the nature of the data that we've used to train and tune the model. So you're almost, as you say, let me just friend this and make sure I understand it. So you're using almost some of the large language models, pick your, whichever one you can, and as you say, that's for world context. Then you're building a small language kind of model that is built on actual human insight that's been kind of commissioned across either your customer base or something you've done yourself. And obviously you've anonymized it and aggregated it and all these different things and got the right sort of permissions for it.

1:07:15And you build a model which you can then interrogate and go, if we did this, how would that work? Or how would people respond? Yep, back to the what if counterfactual. That's exactly it. And so the way that the model works is we like to think of it as a representation of human behavior, right? If it's U.S. consumer right now, what we learned early on, this is a year and a half long journey, right? A lot of scientific experimentation and then model build. Throwing more data at it did not yield better results. Okay. Right. We reached a point of diminishing return pretty quickly. And it was instead, I tell you about the nature of the work that we do with clients, because that randomness and that breadth and that depth of industry kind of coverage is what turned out to be incredibly valuable.

1:08:08screen model performance. And so, you know, if you think about accuracy and validity, right, the way that Ali, the self-described research nerd, would want to see this is alongside to it, right? So we're still in this right now, comparing collected human panel data. I'm going to run it through synthetic. How does it look? And it's not just visually on a chart seeing bars side by side. The way that we're measuring it, we're looking at a couple of different statistical measures of variability of the data because humans are irrational and unpredictable. And we all have very distinct human lived experiences.

1:08:52We want to represent that in the data as well. Models tend to centralize and around a point. So they also hold things fixed. That's exactly right. Right. So you can have two different people, I guess, that might go through the same journey or the same experience, but they could have had two very different days. How they respond to that is beyond the control of a constructed model because that's just something that's a very individual thing. thing. That's exactly right. Yeah. And so all that context, right? And we don't have that necessarily in the survey data, but you can start to discern, right, these patterns and behaviors.

1:09:32And so, but what's also incredibly rich in our data set is I ask my product diagnostic questions very differently from you, right? And so, and it's learned, you know, what outcome we're trying to drive ultimately. So we measure performance across the variability of the data. And then in in conversation with clients, very colloquially, would I have come to the same conclusion? Would I've made the same decision with synthetic as I did with human? So we're measuring that in a couple of different ways, but then most simply are the same top three responses, the same across human and synthetic, right?

1:10:09And if so, great, I would have come to the same conclusion. So those are the types of measures that we're looking at around accuracy and validity. When we think about application and where we guide clients in good fit, bad fit. There are still plenty of bad fit examples and use cases. Dinner last night, right? There's no way that the model could know how quick the service was, temperature of the food, and what the general ambiance was, and all of those types of temporally relevant past lived experiences are not a good fit for synthetic. So instead, where we see great performance and success is future focus, likelihood to purchase, you know, invest, you know, fill in the blank, and appeal of a product or a service.

1:10:56So Likert scale, future tense, all of those are quite good. What if scenarios type of thing. Exactly right. Okay. Yeah. And how is it being received by the sort of the market research sort of community because some people might kind of turn around and go, huh, I used to do that and I enjoy doing that. Because I think that adoption is interesting, particularly if you're running things kind of side by side. You're right. There are elements that, and certainly AI is posing a risk threat to many knowledge workers, right? Researchers are certainly not excluded. There are, though, plenty of things that I've had to do in the 20 plus years I've been a researcher that I really wish I didn't have to.

1:11:44Right. And I'll quote a client from just this week. It was in a room of PhD researchers. That guy's building charts in PowerPoint, right? I don't want him building charts in PowerPoint. I want him advising the business on, you know, strategic investment. And so you're right. There are certain things we want to shed ourselves of and never look back on. Sure. And there are other things where we're going to, you know, dig our heels in the ground and say, this is mine, damn it. And I've been, you know, built my reputation doing this. So the researchers, believe it or not, there was a lot more resistance last year.

1:12:19Same time last year, X4 last year. A lot more objection to how could this possibly be? And we were newer, right? One of the first coming to market with it. So there wasn't quite the groundswell in support that we have now. and said this year, fast forward, I walk into almost every single client conversation, not having to defend, not having to explain, not having to justify, but rather guide. Right. Can I do this? Can I use it for that? And have we thought about this? Do you have any, you know, case studies around that? Just, it's not, it's beyond curiosity. It's, I know I need to do this. Help me with what we've talked a lot about even today is there will always be study types audiences that have to be human right um and so the researcher's role now becomes a bit more of this um orchestrator of use consult this data source for that use this data source for this and that might be operational data that might be cx data that might be synthetic and it might be you know human crawl true and but i also think the um there's a just coming from wrong i hate There's a distinction here that this is not, when you build these synthetic panels, they build these models.

1:13:32Now, currently, you're deploying the one for a U.S., a U.S. consumer, but across different industries. And then there's, was it Ireland, the U.K., Australia, New Zealand, and Canada. So all the Anglophone countries get it rolled out over the course of this year. No, in a couple of weeks. Okay, in a couple of weeks. technically the same this year this year anyway other things coming later so very very soon but it's not a one and done thing because actually and I think we've seen this from other places where some of these big LLMs have started to create their own data and then actually the models collapse because you end up just emphasising looking at the wrong thing and it's not fresh But actually, it's my understanding that actually these models require maintenance and refreshing, and that's what keeps them kind of vital and relevant because the context window moves, right, as time progresses.

1:14:36Yeah. So we call it hydration. Yeah. So constantly, monthly, minimum, right? Hydration of net new batches of aggregated anonymized survey data and quickly on the horizon, non-survey data. So this requires more science and experimentation. I think it's what we've built is phenomenal. It's great. It meets our immediate need to come to us with survey question and want quant answer, right? If we start to introduce behavioral data, transaction data, news, trend, headlines, what might that additional context do representing and reflecting the human behavior? So just to be clear, the panel's more operating on a quantitative basis rather than a qualitative kind of basis.

1:15:22But is it plans to try and build in the right? Because that's where it gets super interesting. It does. And hard to validate. So, well, yeah. Right? So, the surveys have open ends and verbatims, right? So, that's the extent of the qual that we have. That's sort of the overlap, I guess. That's exactly right. There are models, research teams out there that have exclusively trained their models to qual data. So, we're talking like four-hour long interview transcripts, which is interesting, but I'd imagine impossible to hydrate. And expensive, right? Right. So there's a blind here where I'm not sure if you've seen in the XM Park, we are, and within Brad's or some of the product folks are down there demoing synthetic personas.

1:16:07So this is now, we've proven this model. This model meets my accuracy and validity expectations for question and answer in quant fashion. Now that it's done that, I can trust qual output from it, right? within the same use cases, within the same audiences, we're asking about dinner last night, you know, all of that still holds. And so that's what we're kind of demoing downstairs, really to get an understanding of what are these kind of next batch of questions and curiosities from clients? What are they going to use this for? What type of, is it marketing? Is it product? Is it, you know, what are we advancing?

1:16:44And validation is going to be hard because it's by design, generative AI can answer just about anything, right? And so there's a lot to learn on that front, but Qual is very much here. And I think I'll take the opportunity to build one other kind of extension of this model that we'll call Walled Garden. And so what'll happen now is, whether driven by Qualtrics or partnership with our clients, we'll start to specialize. It is general and it is wide and it is random, but wouldn't it be better to know how consumers interact or transact with automotive industry versus, you know, healthcare versus, you know, consumer packaged kits?

1:17:29That's one potential angle. The other is clients want to make it smarter to them and their business. Well, I was going to say that feels like an actual extension. You take a generalized kind of model and then you go, can we not tailor it to us? Exactly. And so we'll build an adapter on top of the model that ingests client data. We'll start with client research data, right? You know, what has Ford researched recently that could be really interesting to feed the model, their target population, how they think about, you know, their products and services. and now the model becomes something that I trust even more in both qual and quant fashion and I'm willing to democratize it within my organization to have all of my stakeholders you know access this awesome and so that's that's walled guard and that's kind of you know how we're thinking about advancing this incredible robust repository this thing that we've started how do we now extend it even further awesome and I know you've always got like a set of early adopters that's my final question let's say of earlier about this what are you seeing clients getting the most well dialing in on quickly so like what are they doing kind of how they how they kind of where are they starting with this yeah i'll give you kind of two examples my favorite is even when i meet the haters and the resistors right please consider it for just this pre-testing a design before you go and invest in time you know and budget with them with humans run it through synthetic see what distributions fall out what surprises you what might be missing what what what you know what might we change to this design so that's one that's my plea to everyone just try it for this it the experiments so far and the clients who are decrementing credits transacting it today tends to be more of what we talked about earlier the future focus concept test right yeah i've you know i got this new thing um do they like it would they pay for it that and idea screenings there's the session this week with Allstate was phenomenal.

1:19:30I loved, loved, loved it. It was his idea. And one that I've been preaching about for a year, where they had 25 different, let's just say, offers or, you know, mesquites that they wanted to test. You can't do that with humans. 25 is absurd, right? You'd end up breaking it down into five different cells. And so you can run 25 through synthetic. It doesn't get tired and reduce it down to here are the top 10 that we should then fill in the blank, take back to the team and call it good or, you know, advance to different types of research. And so those generally, we call it more like idea innovation, right?

1:20:06Yeah. Research innovation. Perfect. Well, thanks very much. Of course. That was, I mean, it's really fascinating because I must admit, and I've heard about some of the, I read, read about some of the synthetic data sort of talk and read some of the hyperbole kind of like that circulates some of that sort of stuff. it's i i had my concerns if it's like kind of models creating literally synthetic data you're like i'm going yeah that's not going to be right but if it's actually based on based on actual human insights that's being gathered and then it becomes a you can gather and use it as a predictive foundation to test kind of like scenarios and do the make that kind of easier then that just feels like a a really valuable thing that's exactly right and it's you That's why the name is not, appropriately.

1:20:58It's just not because it's real data. It is real predictive intelligence. And I think simulation kind of describes it best where we've studied and learned from humans, their behaviors, interests, attitudes, and beliefs. And we're simulating that for the purposes of market research and intelligence. I love it. Little by little, we'll get them. Thank you so much. Of course, my pleasure. Well, I feel like I possibly am going to be in danger of repeating myself. I mean, it was another great conversation. I love the insight that Ali gave into synthetic panels and how they work and their potential applications in customer insight and decision making.

1:21:37And overall, I do really hope that you enjoyed the series of conversations that I've had with Qualtrics execs here at their X4 event. See you next time and do tune in again. Thanks very much. Wow, what a great interview. I hope you enjoyed it. I know I did. Find out more about me and the work that I do at adrienswinsko.com. Do leave a review on your favorite podcast platform. And if you have any comments, feedback or questions about the podcast, then feel free to send me a message to podcast at adrienswinsko.com. And do tune in again. Thanks very much.