Turing Award Special: A Conversation with Jeffrey Ullman

0:00Jeffrey Ullman is a renowned computer scientist and professor emeritus at Stanford University, celebrated for his groundbreaking contributions to database systems, compilers, and algorithms. He co-authored influential texts like Principles of Database Systems and Compilers, Principles, Techniques, and Tools, both of which have profoundly shaped the field of computer science. Jeffrey received the 2020 Turing Award together with Alfred Ayo, quote, for Fundamental Algorithms and Theory, Underlying Programming Language Implementation, and for synthesizing these results and those of others in their highly influential books, which educated generations of computer scientists.

0:37In this episode, he joins Kevin Ball to talk about his life and career. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

1:17Hello, Software Engineering Daily listeners. I'm K-Ball, and it is my privilege and honor today to welcome Turing Award winner Jeffrey Ullman to the show. Hello, everyone, and thank you for inviting me to your esteemed podcast. Yeah, I'm excited to talk with you. I know you have such a kind of rich and varied background. How do you introduce yourself these days? Like, I know you mentioned a lot of the stuff you did was old hat. So what do you describe yourself as today? I'm basically retired over the hill, super numerary. You know, let's face it, folks, computer science is a young person's game.

1:58And I ain't that anymore. soon. That is fair, though. I think some of the things that you did have been shockingly long-lived in this field, which moves so quickly. I think when I mentioned to someone, oh, I'm interviewing Jeffrey Ullman, they immediately were like, oh, the Dragon Book, the Dragon Book. So I'd love to just spend a few minutes, I know that, once again, older, but what do you think made it so successful? It feels like this is something that first edition was decades and decades ago. And I talked to new computer science grads, and they're still talking about the Dragon Book. Yeah, well, I think the answer is in the name.

2:34It was the cool cover. And apparently, people were proud to be seen walking around campus with a book that had a picture of a dragon on it. And therefore, they enrolled in computer science. And that's probably the biggest contribution I've ever made in my life is getting some really bright kids who otherwise might have wasted their life in physics or something, you know, to go to CS. If we look at compilers, I know, you know, when you started, the types of languages we were compiling for were perhaps simpler than some of the things that we've had today. There've been a lot of developments. Are there any changes that stand out to you over the years in terms of language design and compilers that you found particularly interesting?

3:23Well, I think the biggest thing is that we're now compiling for parallel machines in some sense, whether it's an eight core processor or some massive supercomputer. There are issues which I guess the last version of the Dragon book began to address a little bit primarily because Monica Lamb, this is sort of her field, is parallel compiling. But that's something that when we wrote the first of the three editions, we didn't even really think was worth thinking about. And it probably wasn't at that time. That's 77, I guess. Yeah. 1977. Yeah. Well, and I think one of the threads that I've seen as I started researching your career in your work that very much shows in that book is kind of thinking about and connecting from sort of fundamental theoretical abstractions and thinking about the different layers of abstraction and then how you apply them to practical problems and sort of move from this very abstract theoretical approach into higher and higher levels of kind of context or implementation detail.

4:37I'm kind of curious, are there ways in which the abstractions we need have changed with that move towards more parallel computing? Well, let's see. Certainly, if you were compiling for a serial machine, you didn't need any of the abstractions that deal with concurrency control, for example. The map reduces, I guess, an interesting abstraction of parallel computation, things that wouldn't have made any sense back in the 1970s, certainly wouldn't have been useful. Yeah. MapReduce is a good sort of segue into, I saw some of your more recent work is really was tied into these large scale data mining and dealing with web scale and web scale.

5:21And you had a book in that space as well. So MapReduce is one example. What are the other key pieces that you think have gone into as we've scaled up and up and up into the current day? Well, I mean, it's pretty obvious that what has happened is getting all the data in the world together has given us this amazing power that we have. And, you know, I'm talking about these large language models. And I'm sure we know that there are, well, there's still some of the rough edges to be worked out. Let's put it that way. But obviously, it gives people power to do all sorts of marvelous things. This same thing was true, what, five to 10 years ago when the massive neural nets came along.

6:09And again, what made them work was not only the availability of, you know, very cheap, massive computing devices, but the availability of a massive amount of data. And this is, of course, something that people often dismiss is the importance of the data. It's not just the algorithm, it's the data as well. So, for example, I've heard people are now worrying that LLMs have reached their limit because we've already taken everything that's ever been written and there ain't no more. And that's an interesting, well, we'll see, I guess, in a couple of years, whether that turns out to be true, or whether, for example, there are ways to manufacture data that never really existed in the way, oh, let's say, if you're trying to do machine learning to recognize tumors, you can take an image of a tumor and you can enlarge it 10 % or rotate it 10 % and create additional useful data without actually having any more data.

7:18So anyway, that's an interesting question. Yeah. I think one of the key questions there is like, what makes it actually useful, right? Like in the image example, that's great because it allows you to sort of extrapolate things that look different to the computers, but that we as humans can validate are the same or equally valid. What does that look like in some of these other spaces? Yes. I have no idea. Yeah. It's just, it's an interesting question. I'm sure people are looking in this direction right now. Yeah. While we're talking about this, I think I saw something that you wrote about sort of the importance of dimensionality reduction when you start to deal with these very large scale data situations.

8:03And it was pondering if there was a connection there to what people are doing with distilling models in terms of like trying to pull out what are the relevant dimensions. I don't actually, I don't remember writing anything myself on that. Obviously, the Mind and the Data Sets book does talk about dimensionality reduction. It's often a useful tool in understanding data because there are various techniques for focusing on what's really important in some very complex, high-dimensional data. Yeah. Thinking about, once again, that sort of abstraction lens that I saw in this world of LLMs, and you highlighted there's some rough edges, there's some challenges.

8:45What are the key abstractions for us as software engineers in working with these things? Or are there abstractions that we are missing and maybe need to figure out? It's a good question. I don't think we have really said the last word about how you use them. One of the interesting things is Stanford, for example, never taught a course on how to Google stuff, right? How to create search queries. But we are now teaching a course in what's called prompt engineering. Probably many schools are doing it or will be. There seems to be the idea of a prompt, let's say, maybe needs to be abstracted or understood in some way.

9:29I have yet to see just a good set of principles being written down for how you use it. You know, I've been playing around with a little bit myself. One of the things I do in my old age is I do a lot of editorial work. So there's this journal that I work for where they have 300 editors and I'm the only one in computer science. And so a lot of stuff gets dumped on me. And I understand nothing about what I'm reading and there's nobody to hand it off to. So what I've been starting to use various LLMs, you know, ask it things like, you know, first defining what a good referee is, and then saying, and here's an abstract of the paper, suggest some people who would be good referees.

10:14Sometimes it actually gets it just right, and does introduce me to people I've never met, but who turn out to be good referees. And other times it just, you know, sometimes it'll say, well, this is too hard for me, but here's how you pick a referee. You read the literature and you, you know, basically completely useless comment. And frankly, I don't know what makes, you know, is it the difference in the detail in the abstract or just how I phrase the definition of what I want? So I don't know. There's a lot of very mysterious stuff, I think, that is still important. Yeah, well, and I think you're right.

10:56There's an opportunity here. I saw a tool that was essentially trying to compile prompts, more or less like a very simple compiler would, where it had sort of different layers, and you could see it sort of combining and lowering things down to different styles for different LLMs. but it didn't, I guess I could say, it didn't have any optimization, right? It was entirely focused on how do I go from kind of a high-level set of components to something that will work for Anthropic versus ChatGPT. But I think there is something there of the equivalent of a compiler that understands what are the right ways to prompt these different models and maps from intent to something useful.

11:36Well, yes. Good research topic. What can I tell you? It would be a good research topic. Speaking of research, then you mentioned recently you've been actively working on a different project. Do you want to tell us a little bit more about that? Well, let's see. Again, I'm trying to just sort of keep busy in my old age. I guess it was really about the turn of the millennium that I and a number of friends started a little company called Gradients. And it was designed to automate homeworks. And the architecture that we hit on was this. To the student, it looks like a series of short multiple choice questions where they're asked to solve a problem and then pick a correct answer from, say, four choices.

12:27But the difference between this and a routine multiple choice question was we felt that we wanted to use the homework to not only test, but to teach. And so if you got it wrong, we'd give you some sort of a hint and ask you to do this whole thing again. And the trouble is with multiple choice, well, if you guess A and it's wrong, then guess B the next time, and eventually you'll get the right answer without ever having to do anything. So the way we worked it out was we developed what we call root questions, which are questions that have more than one correct answer. Now, how does that work? You know, for example, if the question is, solve the equation 2x plus 5 equals 10.

13:19Okay. Solve for x. Now, that has only one correct answer. I think it's two and a half or something. So it looks like there's only one correct answer. But in fact, what we're asking the student to do is solve the problem. Okay. You're going to write down. And if you've got it right, it's x equals two and a half. And now the choices are not what is the value of X, but tell us something true about X. So for example, you can have a correct answer. X is not an integer, or X is less than three, or twice X is a prime. So there are any number of correct answers that are easy to recognize if you know what X is.

14:02And if you don't know what X is, you're just guessing. So, you know, we thought we would make a lot of money selling the service. We didn't, actually. Basically, people were willing to use it and still do use it. Our largest enrollment is from the University of Cairo, for example. So people still use it, but they wouldn't pay for it, basically. So I guess now almost two years ago, I gave a talk on this technology in Bangalore. And someone in the audience, I guess one of the original Infosys guys, got all excited about this and said we could use this to teach mathematics. And you can. If you go to the Gradient site, the materials that we have are really, well, computer science from Java programming, obviously compilers, operating systems, data mining.

15:01But I said, we can do this in mathematics. And so he actually raised some money and got a team together. And they're putting together where we've been working so far on ninth grade mathematics. And by the way, ninth grade mathematics in India is mostly stuff I learned in 10th and 11th grade. But that's another story, I guess. At any rate, we're hoping to field these materials that anybody in India or around the world, I guess, can use for free. And again, you know, the idea is it not only makes sure that students have learned the material, But where they're having trouble and give wrong answers, there's always an associated hint or explanation of some sort that we hope eventually gets them to get all the questions right.

15:53Yeah. I feel like this is a broad topic area that people are very excited about right now, which is like, how do we allow people to essentially scale education and go at their own pace and not depend so much on individual teachers? Okay, now that's an interesting point. You know, again, roughly the same time that we started Gradients, you know, these MOOCs were all the rage, and it was somehow assumed that these were going to replace universities. Yeah, and where did that go? Yeah, where did they go? Okay, well, it's a good question, and I think it's an important one to answer. Part of the problem is this.

16:32When, say, General Motors decides to replace assembly line workers by robots, nobody asks the assembly line workers whether they think the robots can do a good job or not. Okay. When you offer academia alternative ways of, you know, essentially introduce technology into the education process, teachers have traditionally had the ability to say, no, this is not acceptable. I don't want to use this. Keep it out of the classroom, basically. And this is, I can't think of too many other fields where that is true. But it turns out, I think it was a little bit worse than that. Not only was there obvious resistance on the part of people whose jobs are threatened, but it turns out that the MOOC doesn't really give you enough for 95 % of the students, that people need help.

17:32Yeah. And again, you can automate a little bit of that. And I think gradients does a little bit, but you're always going to need somebody to go to when you're stuck. So the MOOC cannot eliminate the need for teaching professionals. It can perhaps take some of the burden off of them, and thereby make them more productive, which, by the way, means we need fewer of them. Okay, that's life. It's true in every field, right? You know, I mean, I think we could do more to, you know, if instructors were willing to, let's say, play a MOOC, and then much the way they used to prescribe a textbook, right? I mean, the textbook did some of the job of teaching.

18:24The MOOC can do a little more of the job of teaching, but you still need somebody to conduct the class and in particular to handle all of the special cases that keep coming up. And then, of course, there's also the problem of validation. You can take a MOOC and have your smarter, older sister actually do the work for you and nobody really knows. So just because you've registered for something and then seemingly done the MOOC requires, it doesn't mean it was you. And, you know, OK, there's technology for you go to a special center and you get fingerprinted. But that's not really going to work. That's not a long term solution.

19:11You need people, instructors, to not only make sure that everybody kind of gets it and the people who are having trouble in a million different ways, they are helped through whatever problems they're having, but also just to validate that they have, in fact, learned the material. I think that's not going to go away. And I know there are technological solutions that sort of do it, but I can sort of understand why schools don't want to automate the validation of their own students. So I'd love to take this conversation about learning in a slightly different direction. So you talked a little bit about Stanford now offering a course in prompt definition or prompt engineering.

20:00Prompt engineering is apparently the term. It's the term. Yeah. We can argue about whether it's engineering or what it is. But you talked about that. We've talked a little bit about applying technology to help scale learning, both what you're doing at Gradients, a little bit what these massive online courses we're doing. I feel like we are in a time period where there's a lot of resistance to technology change. There's a lot of resistance to LLMs in different parts of the world. So I'm kind of curious how you think about how do we bring people along with the changes that are happening? Boy, it ain't easy.

20:39I think you just have to get new people. I mean, that is one solution, right? You wait for one generation to die off. Look, having aged, I see it in myself that it just becomes harder and harder. to adapt to new technology. And by the way, the field of HCI has been around for forever, really, you know, human-computer interaction. And I haven't seen anybody addressing the problem of sort of dealing with the old folks and the fact that I can't explain it, but you just... I've seen this as well. In my parents aging, my dad is in his 80s now, my mom has passed away, but the rate of change is just hard.

21:29And I feel like in the tech industry, we often change just to change and neglect the cost of that. But it's not just change for the sake of change. I mean, the fact is, for example, I had to learn to use and depend upon a cell phone. yeah and for example our 25 year old grandson was out visiting us and and he he likes to borrow a car and just drive around and so he did that and i have a nav system and i know how to use the nav system but he says well i just want to use google maps okay and he says well you need the right adapter and you can just plug your cell phone into your car, into my car.

22:16I've never known that, but apparently you can. These are the kinds of things, you know, I guess young people, they all know that, that that's how you navigate these days. I was visiting my son. I have a five-year-old granddaughter and I brought along a tablet and they're not allowed to play with these normally, but when grandma and grandpa show up, they... Yes, you get grandparent privileges. Yeah, right. And she wants to download an app to do drawing. And then the app has ads. And that gets you to basically... There's a strange ecosystem because it looks to me like all of these apps, all their ads are for other apps.

23:06And so that's basically an economy based on everyone taking in each other's laundry. So I'm just not sure exactly how that works. But this is what happens. So she's wandering from app to app. And I said, look, you got to get around the ads. And, you know, fudging around, I found I sort of found it. And 15 minutes later, I notice she's found her own way. And it's better than my way. And that's what five-year-olds do. Yeah, it's funny. it reminds me of when my son was that age, we would joke that somebody would hand him a device of their own for whatever to entertain him. And within minutes, they'd be saying, wait, how did you do that?

23:46What is that? Yeah, exactly. And I think a big question is, how do you make it possible to teach old folks who have, you know, I guess we have our own skills and our own ways of doing things that don't always work. And, you know, for example, how do you recognize what it is your customer, say me, is capable of doing or, you know, feels normal doing, I guess, or has the skill to do and use that to get them to navigate the device? I think this is really important. And it's a big blind spot, because as you said at the beginning of our conversation, tech is generally a young person's industry. And so the people making these decisions and thinking these things may not themselves be interacting with older folks very often at all.

24:43Even middle-aged folks, there's a big difference. Like I'm in my 40s. I naturally interact with most technology via text. My kids go to voice and video for everything. it blows my mind right i want to look up how to do something i'm writing in text i'm looking for an article on it they're looking for youtube and so we as an industry especially looking at you know our society aging like we do need to figure out like how do we incorporate the preferences needs and differences of older generations i'd love to see that have you seen anyone doing it well? Doing research in the area? Either research or even like some company managing to do a good job with their products on it?

25:29Well, I have not seen really good examples of the systems that adapt to, let's say, a variant mode of thinking about it. But I get the feeling it should be feasible. It should be feasible. Well, and I feel like I do wonder if this is a place where sort of modern machine learning LLM technology can help because it allows for what I've been describing to some people as more kind of intent-based user interaction paradigms where it's able to do some amount of interpretation. So if we can use that in a way so that we can try to infer intent, even if your natural mode of trying to interact is different than my natural mode is different than your grandchild's natural mode, like be able to figure out what you're trying to do and help you with it.

26:19Yes. I think the new LLM based technology, you know, as a simple case, it should be able to summarize a YouTube video. Yeah. And, you know, I think an interesting case that doesn't seem to yield to LLM technology would be. I mean, I have a map on my cell phone. I want to connect it to my car's screen. I don't know. Okay, it's in the manual, right? There's a 500-page manual in my glove compartment. Somewhere, it must tell you how to do that. But I don't even know that it's there, that it's an option. Another example, my son had to show me that my trunk has a button that lets me open the trunk from the outside without pressing the button that's in the car itself.

27:17I think all cars have this these days. Nobody told me that it was there. Yeah. There's sort of a dissemination of information problem, right? If you knew to asked the question you could have found the answer but how do we surface the questions that's exactly right now the car should say i notice you've been pressing the trump release button quite a bit did you know that you can yeah and that's not unreasonable and i'm not sure it's an llm problem but there is something there and it actually reminds me of kind of what you're doing with gradients, right? Of like, there are many correct answers here if you have sort of the underlying model.

28:00Can we build our technology to notice that there's a class of them that you are just systematically ignoring or unaware of and surface them to you? Anyway, that's what HCI people ought to be doing, I think. Yeah. You know, one of the things I love about talking with folks who are older and who have seen so much about this is you often have a much wider perspective than someone who's only seen 10 or 20 years in the industry. What else do you see as gaps in our current tech ecosystem and environment? Where should people be doing work that they're not? That's always the hard question to ask. You look at the great changes.

28:42Nobody realizes that we need it until somebody provides it. Give you an example. I think it was, the data is very important. I think it was like March of 1992. I was invited to DC for a discussion about what was then called the information superhighway. And it was some academics like me and heads of telecom companies and other industry types. And we wrote a report that came out of this. And the interesting thing is nobody mentioned the World Wide Web. Okay. I mean, there was some discussion of Gopher, I think. And, you know, the big question was would the information superhighway run along cables or telephone lines?

29:42Okay. You know, and here in Switzerland, the World Wide Web existed. And none of these 300 experts, you know, I take the blame for it as well, was aware that this was even an option. I mean, I remember several months later asking one of my colleagues, why do you always keep referring to things as colon slash slash? flash. You know, it just, nobody was even aware of the need for something like the World Wide Web. You know, I think you can probably say the same thing about cell phones, personal computers. You know, there's this famous quote from, I guess, the head of DEC at the time, you know, why would anybody want a computer on their desk?

30:32So the answer is, I cannot really tell you what the next or even guess what the next great thing is going to be. Certainly not. But you have already pointed to something that's missing right now, which is HCI research focused on older generations. I think it's a good research topic. I don't think it's going to change the world. You know, I think, I mean, if I had to guess, I would certainly have to think about quantum computing. I've always been sort of a skeptic about quantum computing. And apparently there are some applications. That is, you can simulate some things better using quantum phenomena.

31:17And by the way, I should say, it does look like quantum communication is real. You know, this spooky action at a distance story. Instantaneous communication. Mind-boggling. Yeah. That people are actually possibly able to do things of that nature. and that may become real. But what people have been expecting is that there are going to be these quantum computers with the right kind of qubits that will enable you to implement Shor's algorithm and break RSA codes and elliptical codes and things like that. I mean, that would certainly be a transition in our way of thinking if that were the case. I'm sort of dubious.