Computer scientists as rogue art historians

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:42Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing well, Daniel. How are you today? I'm actually super excited for this conversation because I don't know about you, but I've just been like swimming in generative text AI for like weeks and weeks. As have we all, I think. Yeah, this conversation feels like I can like come up for air and think more to both computer vision and generative image AI and other things like that, because we've privileged to have with us Amanda Waslewski, who is an art historian working in the digital humanities program at Uppsala University.

1:34And she's the author of a new book coming out in May, Computational Formalism, Art History and Machine Learning. Welcome, Amanda. Hi, thanks. Thanks for having me. Yeah, yeah. Like I say, I'm really excited about this. So I have to be honest, I was a little bit intimidated maybe because I don't know a lot about art history. But in looking at your book and also looking at your amazing research that you've been up to, like there's so much practicality in this, both in terms of like what is applicable to art historians and those working in that area, but also the things that you're talking about in terms of how we think about like machine learning and art and how those relate, and especially in light of generative things in recent years.

2:25So yeah, I'm super excited about this conversation. I'm wondering, you mentioned in the lead up to when we were talking pre-episode that your background is more on the art history side. Where did the art history and machine learning start to collide for you? I actually started out, well, I studied chemistry as an undergraduate briefly before kind of discovering art and art history. and was a practicing artist for many years before I went back to studying art history again. So I had a kind of, I guess I've never been formally trained, but I had a kind of sideline doing artwork that was based through using various digital technologies and certain kinds of programming.

3:14And I also kind of worked a little bit in web design and things like that. So I had a kind of background in computational things, both from a kind of art perspective and a professional perspective before I actually went into academia and academic art history. So I've always had those kind of interests in how art and technology co-aligned. And I kind of I came to this whole field or the kind of emerging image and AI field through older things like image databases and how they're sorted by metadata, textual metadata. So that was the kind of entry point. And then suddenly it seemed that, you know, more and more art collections or digital image collections were starting to use different computer vision techniques.

4:03And so that's kind of how I came at the field through, you know, the way that computer vision was increasingly being used to sort large image collections and image collections of art in kind of institutional contexts. It's interesting that you mentioned both elements of using machine learning to sort art, but also this background of people using textual metadata to describe art. And I know that you use this word formalism, which in my understanding has some history in the art world. But how standardized is the literature and research around how you describe the features of an artwork? that that's probably a very like naive way to ask that question as a person not in the field but like I imagine you know metadata to describe artwork is like you know artist Van Gogh you know medium like whatever what it seems like what you're talking about goes well beyond that could you kind of describe that space a little bit as a quick add-on can you also add just a little bit about like what is art history coming into that because a lot of folks on we probably have a lot of people that are doing machine learning, but not a lot of art history background.

5:17And some people may be wondering, including me a little bit about trying to understand what it is. So kind of working your way toward where Daniel was, but starting a little bit earlier for me. Well, one of the ideas of the book was actually to kind of in my own way, try to like bridge this gap because, you know, as I said, I don't have any formal training in any of these, like sort of the computer science side. But, you know, I've, you know, been in this kind of digital humanities milieu where it's a kind of combination of some computer science techniques with a kind of humanities focus in research.

5:53So, you know, I wanted with the book to both kind of introduce art history concepts to those people working in maybe computer vision, but also, you know, introduce people in art history to some of the things that are happening in computer vision. And so kind of trying to play both sides a little bit, but obviously from my own perspective in art history. And so art history is not a very old academic discipline at all. Its origins in the 19th century revolved around sort of practices of collecting antiquities, so ancient Greek and Roman artifacts. And that kind of collecting practice started to become a more sort of studied and systematic area, coalescing into like the first academic art history departments came about in the late 19th century.

6:43And back then, all academic sort of subject matter, the humanities included, kind of aspired to the scientific model in the same way that, you know, the natural scientists. So empiricism, taxonomy, these kind of things. So people at that point in time treated art objects kind of like specimens, like, you know, if they were studying plants and the kind of evolution of plants. And so early art historians studied art in much that same way. They sort of traced the evolution of art through time and through history. And so it was really focused on, you know, how the kind of superficial qualities of art change over time, rather than a kind of focus on other contextual things like, you know, the artist's biography or other kind of circumstantial things about the historical time period.

7:34But this has been a longstanding debate in the field pretty much since the beginning. So it goes both ways and often falls into two camps, the so-called formalists, who are the ones who just care about the kind of external appearance of images or works of art, and then the people who care about the other stuff, what the artist was thinking, what their intentions were, what their kind of historical context was, and all that sort of thing. So I'm kind of reaching back into that history of art history. One thing that kind of interested me in this area was I saw computer vision research, you know, so research that had no contact with the art history world, really, using data sets of artworks to answer computer science questions.

8:18So, you know, not answering art historical questions per se, but in the process, because they're using artworks, they're touching on things that are important to art historians or that art historians might be interested in. But I saw that there was this kind of call back to these formalist methodologies similar to what was happening in the late 19th and early 20th century. So I was interested in this kind of what I saw as like a revival of these taxonomies kind of matching to like really simple way or even sort of, you know, the kind of object recognition by finding different motifs or things like that.

8:55So, yeah, that was my, as having had training in art history and its methodologies, that was what kind of piqued my interest in what was happening in computer vision, because I saw it as kind of like rogue art history that was happening, like without art historians having any knowledge that it was happening. So I kind of wanted to like call attention to it on one hand for art historians, but on the other hand, call attention to some of the art historical issues that, you know, computer vision researchers may not have found or had access to. So I had that kind of that both directional interest for me.

9:33I think Daniel and I probably really like the rogue art historian designation. Who knew that machine learning practitioners would be kind of the pirates of the art history world in that sense? Yeah. I've seen a lot of good parallels or memes recently. I think one of my recent ones was like AI is like computer LSD. I think probably like rogue art historian is another good one. Um, so you mentioned that like machine learning people were integrating artworks like into their datasets or to like answer certain types of questions where those related to like, I can imagine like, Oh, if I have these different artworks in my dataset, maybe I can do image classification and classify like this is an artwork or maybe even more detail.

10:23like this is an artwork by a person or like in this time period or in this medium or something. But I could also imagine like artwork has objects in it, right? Like, can I recognize objects within an artwork or certain like features, that sort of thing? Is that the sort of questions that were being asked or what were these questions that you kind of started running across that you connected with the art history world. Yeah, so you hit on sort of two of the main areas that were being addressed. And I think from my reading of the literature, as I understand it, the computer vision literature, there was a kind of, you know, obviously object recognition in images has been a huge focus from the kind of the last 20 years plus because it has so many, you know, quotidian and nefarious applications.

11:14You know, you get lots of surveillance applications, but lots of like, you know, we open our phones with our face kind of applications. And the ability of, you know, a machine learning system to like recognize an object has obvious practical applications. And so I came across a lot of papers that said something along the lines of, well, recognizing objects in a photograph is a solved problem. So I think at a certain point in the last like 10 to 15 years, I kind of cover like a 15 year trajectory of this research in my book. Researchers kind of were looking for more difficult data sets to tackle.

11:52And one of those was art data sets because, sorry, to recognize an object in a kind of stylized painting would be something that would be slightly more difficult. Yeah. So, you know, we had these sort of object recognition activities that were happening. But from like my perspective in art history, it's not a very useful exercise. You know, I don't care really as an art historian if there are a bunch of dogs, if you can identify a dog in a painting. It's not that interesting as like a tool to use for my research. So simultaneously, there was a lot of research happening, which is, you know, the kind of categorization by style.

12:33And this was really interesting to me because this term style in art history is a really fraught term. It has a complicated history and art historians have fought a lot about, you know, what does style mean and how do we define it? And the categorization by style in this terms, you're looking at a kind of superficial quality and you're categorizing it by a known kind of textual label. I think it's interesting because, you know, this has now really important kind of knock-on effects in generative AI. Like if you open Dali and you see their kind of suggestion for the initial prompt they suggest you write, they say an impressionist oil painting of sunflowers in a purple vase.

13:18So right there in the generative AI platforms, you always have these quote unquote style markers. So I really wanted to sort of, I guess, unpack what style means for art history and what it might mean when we're suddenly applying things like Impressionist in the context of generative AI.

13:46Amanda, I love how you brought us along to understand both like this intersection of art history and machine learning and how like machine learning was sort of dipping into these formalism elements over time. You talked about like the prompts and Dali or something like that, like the style. When you're talking about now, like art historians kind of realizing how they can employ machine learning within art history, is that the sort of thing that they're thinking about? Like, like I could imagine if I take a bunch of artwork, you know, clustering image embeddings to like, look at the style of like, what is actually similar between all of these images and that sort of thing.

14:31That was kind of where my mind went when you were talking about style. But how have like practically art historians kind of been employing this once they realized that machine learning people were kind of like extracting some of these interesting features? Yeah, so exactly in the way that you just described, there is one of the sort of founding fathers of art history, Heinrich Wolfflin, who he pioneered that, you know, so art historians have always been kind of uh you know using tech for you know various uh you know teaching and or research purposes and he pioneered in the early 20th century the idea of having a double slide projector in an art history lecture so that you could compare to artwork it doesn't sound like much to us now but it was the idea that you could compare side by side in a lecture setting to artworks at once.

15:26And so you would kind of see, but, you know, the human eye is only able to sort of kind of take in so many comparisons at once. And so the way that these type of technologies have been used in art history context is exactly in this kind of mass comparison sense, you know, comparing many, many artworks, many, many more than could be possibly compared in a kind of one single view. So in kind of literary studies, they have something called distant reading, and there's a kind of corollary in our historical studies called distant viewing. And the idea is you get a kind of top-down, very faraway view of general patterns or general trends.

16:08And the hope was that you can kind of notice new things through looking from this distant point of view. But one of, you know, one of the things that, you know, is important in that is, again, you're looking primarily at visual characteristics. Can I ask a non-technical question? Just when you're doing that remote viewing and you're making those comparisons, like, just to give me a sense of the field, like what might be an example, like a typical example thing that you're trying to compare, aside from whether it's machine learning or entirely, you know, without technology in the process, just to give me a sense of a touchstone on what that is.

16:47In terms of what the point of comparison is? Yeah, yeah. I'm just kind of curious, just as a newbie to art history and learning from you as we go, I was just wondering what a momentary, aside from the machine learning side of it, what are some of the things you're trying to get to with it? So this is like the classic art history 101, something we call formal analysis or visual analysis, where the basic step of art history is, you know, first looking without jumping to context or content of an image or work to look at things like texture, line, shape, color, those sorts of basic building blocks of visual information.

17:29And once you've kind of understood that, you start to notice details. And I think it's a way of like looking very closely at an image or an artwork to sort of understand what that is doing visually, what the composition is doing. And then the next tool to add on to that is comparison. So once you understand kind of what's happening on a visual level, purely visual level, you start comparing it. And then you see, OK, so there's different things going on in this other artwork, maybe from the same time period or maybe from just after it. And so you kind of start to build an idea or narrative around, you know, how artworks change over time.

18:05So that's the kind of standard art history, like 101 skill that, you know, we start to cultivate. I'm sorry that I took you there, but I appreciate you doing it. It is helpful for me. Yeah, no, of course. No, I think it's important because it ties back into thinking about what we want to do if we want to use machine learning methods to perform those same tasks. We have to realize or recognize that machine vision doesn't understand images in the same way that we do as much as we might, you know, remove how we interpret content or context. The way we kind of dissect an image visually or the way we kind of analyze the visual properties is going to be very different in machine learning exercise.

18:52And the first way that that's different is that, you know, the vast majority of things we're dealing with are physical objects that have been digitized. So there's like a kind of layer of representation. They're photographs already. So there's already a difference between, say, looking at an artwork in person in a museum and looking at the kind of digital reproduction. I think it is important to sort of understand that foundation as well. So while you're talking about that and kind of the understanding, it's kind of like, I mean, my best parallel would be from the NLP world where like chat GPT or something does not understand user intent.

19:29Right. There's no understanding. Right. It can produce text, but we process language different than chat GPT does like as humans. And like you're saying, someone standing in a museum like processes that experience of standing in front of an artwork differently than a photograph, an intermediate representation differently than like a machine might like find features that are good for image classification or something like that. I'm wondering, because a lot of these computer vision models are so non-explainable or like there's an interpretability problem already, right, in terms of like, I might not know why an image was classified in this class with like a convolutional neural net or something like that.

20:19Is that a struggle for like taking this field forward in terms of applying machine learning in these contexts? Or are there ways to kind of extract some of those main features like you're talking about, like shape and color and line and other things like that? Yeah, I think that there's a lot of similar issues, actually, between the kind of text world and the image world in terms of this idea of what constitutes meaning or understanding. Are you guys familiar with the tank classifier problem? The tank classifier. I'm not. I'm sorry. I don't think I am. Although Chris knows about military vehicles, but I don't know about tanks.

20:57I don't think that's what we're talking about. It was a kind of apocryphal story that was passed around a lot in sort of machine learning circles. The story was, and actually it dates back to a kind of someone made this up as an example at some conference, I think, in like the 60s. But it became kind of passed around as like it actually happened. The story is that the U.S. military during the Cold War wanted to recognize in images the tanks. I do. Now that you go into it that way, I do remember this. Yes. Yeah. So like differentiate Soviet versus American tanks in images, but then ended up accidentally classifying the images by the background weather or environmental conditions.

21:45And that is the kind of thing that I think like really illustrates what we deal with when we're dealing with images because we understand things like background and foreground or the kind of subject and surround in a different way. We interpret those, the kind of illusionistic space of an image in a certain way that, you know, for a lot of kind of algorithmic classification, that surface is what we might call a kind of democratic surface. Like all areas initially are kind of treated the same. It has to be some kind of training to differentiate those. And of course, it's gotten very sophisticated where it is, we are able to sort of separate those things out a lot of the time.

22:29But of course, you still get lots of cases like in medical imaging. Like I read a few things about, you know, during COVID, they tried to classify, for instance, like COVID infected lungs versus healthy lungs. But they used a training set of like children's lung imagery. And so they accidentally classified by children versus adults, which seems like a very silly error to make. But so we get like issues like that, I think are really important because what it points to is that essentially we're dealing with like a two dimensional surface to interpret. But often those are two dimensional representations of a three dimensional space that we as kind of three dimensional beings intuitively understand when viewing an image like that or a photograph, for instance.

23:18whereas you know machine learning algorithms only know that we've kind of isolated a certain pattern of pixels to be a specific object and you know given lots of examples they're quite good at differentiating whatever object we've designated but still there's no kind of understanding of space it's not part of the understanding of images in that framework so I think that that's kind of one of these interesting examples of like, just because it successfully identifies something doesn't mean it understands what that thing is, like a dog in a photograph. Very good explanation. But I do feel on behalf of the defense industry, I should note that we are much better at identifying and classifying tanks today than we used to be.

24:03I don't know if I want to know how good you are. that might be something that I want to be ignorant of. I just feel the need to say that, you know. I have confidence that things have moved on significantly since the 60s. Someone should tell Vladimir Putin. That's all I'm saying. That's all the politics I'm inserting. I am really interested in all sorts of things about what we just discussed in terms of the the understanding elements and other things. But I'm intrigued by this in reading through some of the materials about your book and your work. You talk about how computer scientists often process these sort of like art image data sets or images that are part of their data sets without any real sort of understanding of art or art history.

24:59And you kind of one of the things you talk about in the book is how maybe there's an enrichment of like the data science and computer vision side by understanding more of the sort of humanistic issues and elements of the artwork and those sorts of things. Could you describe a little bit what you mean by that and how you think like, because we mostly talked about machine learning kind of enriching maybe art history or things that could be done there. What about the other side of that in terms of like things computer scientists could learn based on this kind of background and research on the digital humanities side?

25:36Yeah, I mean, I think one of the things that is really important to me is this idea that, you know, the assumption that accepted categories are in some way static or objective and unchanging can lead to really misleading findings. So for example, there was one study that I looked at where they were classifying paintings by artistic style. And they noted, the authors noted, that action painting was confused with abstract expressionism and said, oh, well, in future, we will be able to hopefully rectify this categorization error. But for an art historian, those are two kind of contextually specific style terms that two competing art critics came up with or groups of critics to, and they have a kind of ideological background.

26:34So there's a reason that some critics wanted to call this mid-century American art movement abstract expressionism, and some wanted to call it action painting. And neither term is really subservient to one another. And you don't need to necessarily understand the full kind of art historical picture to like, you know, say if you're using Dali and you want to make either an abstract expressionist or an action painting as a style, you probably get good results with both of those terms. But the kind of issue is that, you know, these are not stable categories. There's different style categories have very different kind of origins.

27:12They're inconsistent amongst each other. You know, some of them span a few centuries, some a decade, some are small groups of artists who all knew each other and worked together. Some are kind of catch-all terms or contextual terms. So I think people in computer science are like, great, I have a new data set to work with and here's the categories and I'm going to work with this and then see how effective it is, you know, categories. And that's fine because they're working on a problem that's different than necessarily what an art historian might work on. But the reason I kind of insert myself there is I'm like, hey, well, that is actually kind of an art historical problem that you're working on, but in a kind of way that doesn't understand that these terms are not fact, that they're not stable in the way that you can kind of like, once you insert something into a database, it becomes kind of solid in a way that it doesn't, when you're discussing it like I am, like I could talk for another 20 minutes about, you know, who came up with these terms and why and, you know, what their political beliefs might be and that sort of thing.

28:11Could you talk, maybe not for 20 minutes, but for some period of time, And I'm kind of curious because you've kind of posed this problem, you know, that's kind of brought by the data science is the way I'm seeing it. Whereas you're saying, you know, you may not have those categories, Craig. What are you proposing as a way of mitigating that in a way that is consistent with art history in terms of approach? Like how would you, you know, that has that kind of qualitative aspect? Yeah, I mean, I think like something I was talking about with a colleague who comes from a kind of computer science background is how do we bring together some of the, you know, concerns and interests of computer science with art history in a way that is kind of interesting to both sides.

28:55One of those things is, you know, for art historians, the context of the nuance of terms in a kind of qualitative way is important. But then how do you integrate that into a kind of data context is the question. And unfortunately, I don't have a really good answer, but I know, you know, there are researchers who are beginning to sort of combine different, well, text and image or different modalities of information together to try to create a sort of, you know, networks, bigger picture about, you know, how we might understand artworks beyond just a kind of textual category. So, of course, we can do a kind of like dispense with categories altogether and do a kind of purely visual kind of like unsupervised clustering type thing.

29:41But then what do we call those clusters or what do we call those collections? And that brings you right back to art history once again. So it's this kind of how to integrate all this sort of qualitative nuance within a data context is the big problem as I see it. And I think that's something that I still haven't found or heard a really good solution, but I've been talking about it with some of my colleagues. So maybe we'll come up with some bright idea in that area. Could that change depending on what question you're answering with a given training session? Like you could do, you could take different reinforcement learning approaches, but I would imagine that that might change the output.