256 | How to build business strategy for the AI era, A detailed blueprint with Andrew Rabinovich, CTO and Head of AI + ML at Upwork

0:00Hello and welcome to the Leveraging AI podcast, the podcast that shares practical ethical ways to leverage AI to improve efficiency, grow your business, grow your business, and advance your career. This is Isar, Metis, your host, and I have a very unique episode for you today. As you probably know, the improvements in AI capabilities is putting a lot of question marks on the value that many companies are generating for their audience. For many companies, the risk is relatively low. Like if you're running a car body shop, then the chances that this impacts you are relatively low. For some others, the risk could be existential.

0:35Like if you're running a company that generates content for others, mostly written content, then you are facing some very serious risk for your future. So there's and there's the entire range, obviously, in between. So every single company is now need to be considering, OK, how is AI going to impact what I offer and the value that I provide? Now, dealing with such high level of uncertainty is not easy. and being able to learn from companies who have done steps in that direction and how they approached it and what steps they've actually taken in order to decide what to do is helpful for everyone.

1:10And this is exactly the conversation we're going to have today. Our guest today, Andrew Rabinovich, is the CTO and the head of AI for Upwork. Those of you who do not know Upwork, it's a company that existed for a very long while that I've used many times in the past that allows you to find contracted work on almost any topic you can imagine on the planet from anywhere in the world and resource talent from anywhere in the world. So it's a very helpful way to find people who are good at something, find what exactly they're good at and hire them to help you on that specific thing. Now, a company like that is in a very interesting scenario right now because a lot of the work that gets offered are things like creating logos for people or writing content for people or creating web design for people and so on.

1:59And a lot of that work could be done by AI right now. I don't think it's there yet, but it's definitely something I'm sure they have been considering. On the flip side, there's a lot of other work that it cannot replace right now. And that still require a lot of human intervention and efforts and capabilities in order to make it happen. But they're definitely in a very interesting junction as far as company. And that's why having that conversation with somebody from that company would be very interesting. So again, Andrew, as the CTO and the head of AI, was deeply involved in these kind of conversations and hence he's the perfect person to share with us the frameworks and the process that they went through in order to decide how to approach AI from a company strategy perspective.

2:41Like where are we going to be as a company with AI in the next few years in order to best serve basically both sides of their audience because they have the people like me who need contractors, but there's a, I don't know how many, but probably millions of contractors on the other side that could use AI in order to make their offering more attractive to people like me. And so a very interesting conversation to have. And since they are in that unique situation, and since they're already making steps in that direction, I know this is going to be a fascinating conversation. And so I'm really excited to welcome Andrew to Leveraging AI.

3:13Andrew, welcome to the show.

3:17In the next few years, AI technology will change our world dramatically. Whether you are a business executive trying to catapult your business forward, or just somebody who refuses to be left behind and want to advance your career, this is the show for you. I'm your host, Isar Maitis, a serial entrepreneur and an AI enthusiast. You'll hear invaluable practical tips from innovative business leaders, AI practitioners, and some of the brightest AI minds in our world today on how you can leverage AI in ethical ways to advance your career and grow your business.

3:58Thank you. Happy to be here. So really, let's dive right in. You kind of know what I just said, right? There's a company like yours that provides services that some of them AI can do today. Some of them might be able to do in a few years. Nobody really knows. How did you even approach this from a concept perspective? Like, what was the conversation? How was it initiated inside the company? What do you think about the future from a personal perspective? And then after we talk about this, we can dive into the actual process you guys took in order to address this. Sure. So to start, I've been working in machine learning, computer vision, and AI for about 25 years.

4:35So none of these things are binary to me as it didn't exist before and now it's here. It's an evolving process and gradually improving one. I joined Upwork exactly two years ago when my startup called Headroom that was focused on video conferencing collaboration platforms was acquired by Upwork. And the main reason why I chose to join Upwork is for the human talent that exists on the platform that you alluded to that really spans all of digital work that exists in the world. I think there's around 18 million of active freelancers on the platform across many, many categories of digital work. The way I think about AI is not whether it's shallow AI or general AI, vertical models or foundation models.

5:27I tend to think about AI from a human-centered perspective. So rather than thinking about building general intelligence, we think a lot about building human-centered AI. And I've done a lot of this at Google and at Magic Leap, at Headroom, and now at Upwork, where the big component of the experience comes from the humans themselves. The goal of human-centered AI is to not replace people, but to amplify them. And this comes in sort of two flavors. One, understanding that there are certain things that machines are good at and humans are not. And the second aspect of it is that when we look at certain skills and body of work that exists today, these are not all encompassing things that will be needed in the future.

6:15Right. If in the past we needed people who write content to be able to know grammar and be able to spell well, today these things are not necessary at all because spell checking and grammar corrections are at their best and you don't need to really worry about. So if you think about things like ChatGPT moving forward, you can basically, rather than having to write out entire paragraphs of content, you can list out a few ideas in bullet form and then say, go and translate that into prose. So to me, that's a very natural extension of a spellchecker. To me, this is just a next generation of a tool that didn't exist or existed with lower capability and now has expanded its scope and does not replace the tool user, but allows the tool user to augment themselves in such a way that they can tackle much broader concepts in the work that they're doing.

7:11Let me ask you a question before you continue, because I agree with you, by the way, 100 % with everything you said so far. In the courses that I teach, in the lectures that I give, there's a lecture that's called My Five Rules for Success in the AI Era. And one of them talks about, and it's all about like mindset changes. And one of them talks about what I call from profession to skill. Basically stuff that used to be professions that over the years, by the way, not just AI, became skills, like typing, right? So when I was a kid, my next-door neighbor was a typist. I don't know any typists that live today.

7:41Everybody knows how to type, so you don't need typists anymore. What I think that is happening, though, is that because of AI, more and more things that used to be profession are becoming skills that more or less anybody can acquire. So I'll use myself as a very good example. I understand software very well because I was a CEO of several different software companies, but I've never written a single line of code in my life. And now I'm generating really amazing applications for myself and for my clients. And so I don't have a skill of writing code. Anybody who knows how to write code will write code way better than me using these tools.

8:15But the fact that I have it as a skill right now allows me to do stuff that otherwise I would have hired somebody on Upwork to do. I'm literally now creating a really new, sophisticated website. And my previous website, I hired somebody on Upwork, which I'm not doing right now because AI is doing the work for me. So where do you see that trajectory from? I understand the human-centric aspect, and I agree with you. I wish that was the solution that everything was going for. But from my perspective, if I can talk to a machine and not have to go through long iterations and wait for responses and so on, and the machine can do that aspect of the work for me, I'm not going to hire a human.

8:50So I'm really interested because, again, you see that at ICM, you know, a size of one. That's not a very good way to make decisions. You look at, like you said, 18 million providers and probably 10x that or 100x that as far as the customers. Where do you see that trajectory right now? or do you think it's going in the future? That's a great question. If you think about these skills or trades as a closed set, then I think we would be in a little bit of trouble. If we look through sort of the history of industrialization and evolution, these skills are evolving, and the set of these skills is expanding more than ever.

9:32So the assumptions and the worldview that we have is that while certain skills are going to be replaced by machines, new and much more new skills will emerge that will be much more complex that machines can solve on their own. right if we think about people driving cars back in the day you had to like press the clutch shift the gears do all these complicated things today you sit in a tesla you barely have to like touch the steering wheel it kind of takes you there i don't know where you're based geographically but in san francisco you can like now take robotons yeah so way more is way more is cool because it's the most advanced and it's uh i've been closely related to it for a while but there are now self-driving cars in the city that don't even have a steering wheel and they can go backwards and forwards like a train.

10:25There's no front or back to the car. Oh, interesting. So this whole skill of having to drive, I would be surprised if my kids or their kids have that skill. And I would argue that it's fun to be a Formula One driver to drive as a skill. But today, like there's a sport of equestrian, right? There are people who ride horses for competition, but as a mode of transportation, that doesn't exist because it's just so antiquated and obsolete. So the same thing here. I was talking to a professor of computer science, famous university yesterday, and we almost at the same time said that this concept of software engineering as we know it today is a skill and it won't exist in 20 years, right?

11:11The fact that you never wrote a line of code and now you're essentially writing code, but you're doing it in English, not in Python or C plus or any other programming language, but English has become the language of coding means that this whole construct of writing these weird symbols in a certain sequence, difficult to comprehend for an average person, all that is going away, right? And that's going to be just become mainstream. As this happens, the ability of humans is only expanding because if in the past you couldn't do software engineering, now through the use of AI, you actually can. So your abilities have expanded.

11:49And we're seeing the same thing happening on Upwork. Talent or the freelancers that do all the 18 million, that do all this work, they are very openly saying they want certain parts of the workflows that they're involved in. to be automated because these are very mechanical, repetitive, and basically boring parts of their jobs that they don't want to do, but they have to do them because otherwise they don't get paid. They don't get paid for the fun and creative bits, but when it comes to like implementation, they're like, oh, hire someone else. No, you are the someone else who has to do it. Yeah, that's why I came to you in the first place.

12:26And that's why they're so happy. The possibility of replacing themselves in doing these laborious tasks can now actually be done. Now, there are two ways to go about this. Either the freelancers that you hire can go and try to replace essentially themselves through Vibe coding, coding agents, all these other systems, or they can identify systems inside Upwork that can help them do that so that they're really working in a platform as part of a framework, systematically figures out which parts of the work requires this high-level abstraction and guidance and evaluation, and which parts of the job can be done in a fairly autonomous way.

13:17So that's one bit. The other bit is that there's a lot of talk about agents or large language models taking over the world and making a serious economic disruption in the workforce. The reality is that work is actually very complicated. And if you can do a very small aspect of it in an automated way, it does not mean that you can do a task end-to-end. We recently published some findings and launched an internal product to the form of a benchmark that evaluates these AI systems. Rather than focusing on stale and academic benchmarks like solving SAT tests or even math olympiads, we take real jobs from the Upwork's marketplace that clients have paid real dollars for and freelancers have delivered the outcomes for those jobs and try to run them with the best-in-class agents across different categories.

14:22and not to our surprise, but many to some others, these agents don't perform that well. Depending on a particular agent, you get performance, again, depending on the category, around 20 to 40 % on the initial attempt of the agent to complete a task. Now, this is very consistent with the findings from other groups in the industry and academia. But what's very unique about this evaluation is that we went a step further and we said, how would the agent perform if it received multi-turned guidance from a human expert? So you give, I don't know, maybe you probably use cursor or something like that to build your website.

15:06You give it some prompt and it creates something. And three out of 10 times it gets it right, but seven out of 10 times it doesn't. typically you don't stop there whether you just have a conversation with chat gpt or any other thing you're like you did this part correct but this part needs a little bit more emphasis go and fix that and through this multi-turn interaction it turns out that agents improve by some ridiculous amounts of like 70 80 percent but what's and that's very interesting because no one's measured that before. But what is very interesting, not interesting, but surprising to see is that this multi-turn interaction between the human and the agent leads to a comparable output of a human doing the work alone, but it happens a thousand times faster.

15:56So it's as if, you know, in ancient Greece, you were a mathematician and you had to do some calculations and you would use like sticks and rocks to do it. And then I show up, I'm like, here's a calculator. you would arrive at essentially the same answer, but it would happen instantly as opposed to taking you six months to figure in the South, right? These agents are nothing but tools. These are statistical models that can't think, that don't have emotions. They don't have a goal, and they certainly don't understand how the world operates. There's no model of the world in these systems. It's just pattern matching systems.

16:29They match very, very large patterns, and they can't really think outside of distribution. So by providing these tools on the platform, we believe that it will make the job of a freelancer on Upwork much more effective and efficient as opposed to them having to use these tools off platform, which we know they already do. So this just streamlines their work. So a few things. First of all, I agree with you 100%. If again, we'll use my example of developing a website i've been working on it for if i had all the time together probably 24 hours by now uh so it definitely wasn't a give it a prompt and you get a website it's not what it is there's the level even even in the website that i'm developing that is not crazy complicated there's a lot of complexity uh you know there's different variations uh breakpoints so you have a large desktop and then a tablet and then a mobile device and then you have different types of motions and they like and you know engagements and stuff that needs to happen on the website So, you know, in backend and frontend, there's definitely more than one prompt.

17:31So I agree with that 100%. And like I said before, somebody who knows how to code would have done this even faster than me because some of the stuff that I struggle with would have done better. The question that I have, and then we're going to dive into kind of like really the process that happened inside of your company and what's the framework that you've used in order to decide what to do moving forward. The question that I have before that is because of the speed this is moving right now. Yes, right now, this one prompt, now I had to spend 24 hours to develop this website and I'm not done yet.

18:01So it's probably going to be 48 by the time I'm done. 48 hours of real work, meaning actual, you know, this will spread over a week, but actual work work, probably 48 hours. In a year, two years from now, five years from now, I will be able to describe in more detail what I want and say, oh, pay attention to these kind of breakpoints on these kind of devices and make sure that this interaction happens. And I want this to be available in the backend and it needs to connect to my CRM and all these kind of things. and it will just do it. And then my input goes down from 48 hours to maybe 40 minutes, which means the fact that I have the knowledge and I've been a CEO of several different software companies that kind of understand how software works, that won't be necessary because it will understand everything that I understand much better than I do.

18:45So I agree with you 100 % right now. There's zero disagreement on where we are right now on these agents. And even if we look at GDPVal, which is very similar, like it's the benchmark that OpenAI created, very similar to your idea. Like here's 44 professions, 90 whatever, I don't remember how many tasks, and the best models right now perform at around 40%. The best top of the line models, which means they do shitty work because more than half the time, they don't get it done properly. But six months ago, they did it at 20%. And so that's my question. Do you think that in a year, two years, five years, we get to a point where there's very little we can do better than the ai can do on its own so if you draw parallels in improvement between completing real world tasks and the performance of foundation models you see that the jump between pre-gpt2 models which sequence to sequence models existed for a very long time this This is not a new idea.

19:54Change between GPT-2 and GPT-3, 3 to 4, and now 4 to 5, you're starting to see it slow down. The reason why it's slowing down is twofold. A, because the algorithms are not improving. It's the same transformer architecture. It has improved a little bit when we added this concept of reinforcement learning from human back and then the reinforcement learning from verifiable feedback. But the algorithms are the same and the amount of data that you feed these algorithms is basically dried out, right? It's very, very difficult to generate data that doesn't exist already in such a way that it's net new compared to the existing distribution, right?

20:46That requires discovery. And that's kind of the bottleneck of evolution anyways. We don't know what we don't know. And if we knew it, we wouldn't need these things to do it for us. So that's kind of the issue. So I do agree with you that all these agents are powered by foundation models, whether general purpose or hyper-verticalized ones. And I'm a believer in the vertical AI over the general purpose stuff, because mixture of experts is still the path to bring it all together. I do believe that the quality of the tools that use these even not improving foundation models will improve. I don't think it'll improve 100x like you're suggesting.

21:30Maybe it'll improve twice. So instead of using 40 hours or 45 hours, it'll come down to like five or 10 hours that you'll have to do. And that will not even necessarily change from the performance of the models it'll change from yours your ability to figure out how to do it so this whole concept of prompt tuning or prompt engineering expands beyond like one line of questioning to this whole thing like i've built this app x many times i know where the pitfalls are so i'm just going to tell it in advance so it doesn't like screw up on its own many times it'll just like follow my instructions and that so that's how you get better at it it's like playing a sport or piano, whatever you try it for the first time.

22:11It's very difficult, but then you have habits. The biggest benefit I think in platforms like Upwork is that tasks that freelancers work on come from humans who have an ever evolving variation in need. so people don't if everybody kept asking for exactly the same website to be built then it would be very very difficult to leverage anything because we they just have a distribution and you sample kind of that the cool thing about upwork is that people are coming up with very new ideas all the time you know you can what used to be as you pointed out earlier people would come in and say i need a copyright or i need a logo or i need like a very basic e-commerce website now people are coming in and they're saying, I need to build software for medical diagnosis.

23:12Or I can, I imagine in the next, when self-driving cars become abundant and everybody owns one, you'll be able to go to Upwork and say, I need help programming my self-driving car because I want it to like not take highways or do this, like all kinds of things you can, because it's, it's basically a robot, right? Then you can program your Roomba, you can program your self-driving car, like whatever. So the set of asks or jobs that will be required to do on Upwork is going to evolve and it's evolving extremely rapidly. And the categories, we have internal discussions about how the categories of work are changing.

23:48And it used to be that 10 years ago, somebody wrote down a list of categories and they broke them down in some hierarchy. This is like web development, This is engineering. And this is like data science. Now these things evolve so much that we actually build algorithms that reclassify categories of work dynamically so that we actually know what it is that people work on. And then we can draw similarities across jobs so that every task is not some bespoke idea, but is actually a subset of a meta construct that we can think about and provide useful feedback. Fantastic. I love it. By the way, I love this concept of the reason this is not a finite or a zero-sum game is because the requirements keep on changing and hence the specialist keeps on improving what they know how to deliver because that's what people are asking for.

24:38Awesome. Let's go into the actual process that happened at Upwork, right? So you look, I don't know, a year ago, two years ago when you started, you're looking into the future saying, okay, this thing is coming. It's going to change a lot. How do you look at it from a company strategy perspective? Like, what were the steps? And it was one thing you already mentioned that I think every company should do is really evaluate on the best level you can. Again, you're a very capable software company with lots of data, but on the best way you can, how that is actually going to impact your work, right?

25:09You just said, you did the evaluation. You said, okay, here are the tasks that we are allowing people to get solved. How much I can actually do to really evaluate the risk as it is right now, maybe as it's going forward. But break down, like when you sit down for the first time, What are the things you were looking at? And then what were the steps after that? What was the framework that allowed you to decide how to approach AI from a strategic perspective? When we started looking at the Upwork experience end-to-end, we realized two things right away. First is that when people come to Upwork, they ultimately have an end goal that they want to achieve.

25:53And hiring someone to achieve that goal is really an intermediate step, right? You want to have a website that helps your business. You don't know how to build that website. So you come to Upwork, you hire someone, and then off-platform, they help you build that website. This is very, very complicated. In machine learning, we call the inverse of that end-to-end learning as opposed to trying, for instance, in self-driving cars, you don't train models to first detect stop signs and pedestrians. Then you train another model to take action. And then the fourth, the third model is to do planning. You do end-to-end learning where you're saying, I'm in point A, I need to get to point B and I can't hit anything along the way.

26:43Right? So that's like end-to-end learning. So in our case, we said, what is it that we need to do to make sure when a client comes into the platform and they say, I want to have an e-commerce website for my pastry shop. We don't say you need to hire a Java developer. We say, here's the e-commerce website that you need. So we go beyond matching and we actually deliver outcomes. Later, we can talk about how that actually happens, but that's like the first step in the process that we said we have to extend the experience so that people don't just find someone who can do things for them, but they can just get what they want.

27:20That's the first bit. The second bit is much more practical because traditionally clients on Upwork had to be very technical because when they came to the platform, they had to specify whom they want to hire. And if you want to do digital work, again, an example, you want to build a website, but you know how to bake cookies and you don't know anything about websites. The question is, what do you ask for? And how do you even know if you're hiring the right person? How can you evaluate if the candidate is correct? So in order to expand the set of clients who can use these services, we decided to move beyond an ability for someone to technically describe what they want and then being able to technically evaluate the results to basically turning this whole problem into a conversational dialogue between a client and Uma, which is Upwork's meta agent that drives all the experiences on the platform.

28:23So rather than coming in and specifying exactly who you want to hire and then looking at the results, clients now come in and they basically tell Uma what they want to build. And then Uma figures out who they need to hire to get that. Interesting. Okay, so what I hear you saying is two different things, and I love both of them. One is from the concept of a jobs-to-be-done perspective, right? Don't look at the steps. What is it that you're trying to achieve? Or from a company perspective, what is the goal that you are serving for your clients, right? If we try to generalize this, like why do people come to my company?