
Accessible data visualization for blind/low-vision screen reader users. The episode argues that “alt text” for charts is often missing or insufficient because it can’t adapt to different user goals (overview patterns vs specific numbers). It introduces a co-designed system that makes charts keyboard-navigable with interactive, multi-level descriptions.
Jonathan Zong, assistant professor of information science at the University of Colorado Boulder; leads the Data and Design Group. He has a computer science background plus visual arts/graphic design training. Collaborators include Alan Lungard, Crystal Lee, and Daniel Hayesh (blind researcher at University College London).
Screen reader chart experiences should be comparable to sighted exploration by supporting structure, navigation, and customizable description (content, ordering, verbosity). Defaults in tools strongly shape perceived “objectivity,” and expressive/stylized charts raise additional accessibility questions.
VoiceOver/alt text failures (“image”/file names), Anscombe’s Quartet/Datasaurus Dozen (same summary stats, different shapes), “flatten the curve,” and a dinosaur-shaped dataset. The system OLLI (Open source library) outputs a tree-like, keyboard navigable chart.
Written by AI. May contain mistakes.
Jonathan discusses the importance of making data visualizations accessible for blind and low vision users.
“I'm currently an assistant professor of information science at the University of Colorado Boulder, where I lead a research group called the Data and Design Group.”
Exploration of the interdisciplinary nature of the Data and Design Group's work.
“where we're working with blind and low vision researchers and screen reader users to work together to design tools for working with data non-visually.”
Discussion of the challenges faced by screen reader users in accessing data visualizations.
“some of my interests in data visualization.”
Jonathan explains the significance and challenges of providing meaningful alt text for visual content.
“So there's also screen magnifiers for people who have partial vision.”
Exploration of how context affects the understanding of data visualizations and the creation of alt text.
“Well, I'm a big fan of a guy named Edward Tufte, who you might know, but I'll share a few details for listeners who don't, and you can chime in as well if you'd like.”
Jonathan shares his experiences and motivations for focusing on accessibility in data visualization.
“But I think also more recently, like people in the Viz community have kind of revisited some of those ideas, right?”
Discussion on the co-design process that Jonathan undertook with blind collaborators to improve accessibility.
“And then around the year 2020, pandemic happens.”
Learn about the value of collaboration between sighted designers and screen reader users.
“Because the theories and the frameworks we learn are also very primarily centered on graphical user interfaces.”
Discover how experiential aspects influence interface design and usability.
“I think that kind of encapsulates the both like the kind of social importance of having people with experiences of disability involved in the design of the things that they will then use.”
Understand the innovations and challenges in creating tactile data visualizations.
“zooming are kind of implicitly like, if not visual, at the very least, like very spatial.”
Learn about the key design dimensions that enhance screen reader experiences of visualizations.
“multimodal systems are kind of the way to go because they just they kind of just give everyone more options for understanding the data in different ways.”
Discussion on customizing data visualization descriptions for different user needs.
“So for someone who's maybe very new to navigating a system like this or new to data visualizations.”
Overview of Oli, an open-source JavaScript library designed for accessible data exploration.
“Oli is the open source library that we created kind of based on some of the things that we found in our research and some of our prototypes.”
Exploration of how developers can integrate Oli with existing chart visualizations.
“Often, you know, people who are writing alt text make choices about like what level of detail to convey the information at because you're always trading off like how long is this?”
Insights from a user study evaluating the effectiveness of accessibility prototypes.
“So if you're if you're a Python person, you can check out, you know, how to write Altair programs that output an Oli output in addition to a visualization.”
Discussion on challenges faced by screen reader users and the need for better interfaces.
“And we weren't really trying to see if it was better, for what definition of better, right?”
Examination of differing philosophies in data visualization and their impact on interpretation.
“But with screen reader interfaces there's so much chaos, right?”
Discussion on the potential of Oli and future aspirations for accessibility in data visualization.
“Should those descriptions be the same, you know?”
Learn about the challenges and innovations in making data visualizations accessible.
“And how those representations kind of facilitate communication.”
Discover opportunities for students interested in human-computer interaction research.
“Well, I am actively recruiting PhD students.”
Find out how to follow Jonathan Zong's work online and keep updated.
“You can also find me on social media these days.”

Get the gist, explore an idea or ask about something said.
On Data Skeptic, what are the key points of this episode?
Automatic transcript. May contain errors.
0:07Jonathan Zong: Welcome to another installment of Data Skeptic Human-Computer Interaction. I think most people take for granted how a chart works when you look at it as a glance, whether it's a bar chart or a line chart or a scatter plot. Take in the data, check out the axes, look for any indicators the chart creator put in there. Charts are a very visual language, and as the old adage says, a picture can be worth a thousand words. But now put yourself in the shoes of someone that needs or chooses to use a screen reader. Imagine this, you're scrolling down a website, reading an article, it refers to some data in a chart, and in most cases that chart will be announced as nothing more than an image or saying its file name.
0:47Jonathan Zong: Those of you familiar with HTML real deep probably know about the alt tag, a well-intentioned tool to provide a description of what an image is, often rarely filled out with anything useful. But even if someone takes the time to write a good alt tag for a chart, you're only getting that one perspective the author chose to tell you. When I look at a chart, I often see different things than other people. My guest today is Jonathan Zahn, an assistant professor of information science at the University of Colorado Boulder, where he leads the Data and Design Group. We talk about his paper, Rich Screen Reader Experiences for Accessible Data Visualization.
1:23Jonathan Zong: Jonathan and his collaborators have divined a rather novel system. It opens the door to finally provide people who need or choose to use a screen reader with a much better interactive experience of data in a graph itself. All that and more coming right up.
1:44Kyle Polich: I'm Jonathan Zong. I'm currently an assistant professor of information science at the University of Colorado Boulder, where I lead a research group called the Data and Design Group. I'm also a faculty associate at the Berkman Klein Center for Internet and Society at Harvard University.
2:00Jonathan Zong: And can you share some details on the Data and Design Group?
2:03Kyle Polich: The Data and Design Group is a research group that I started about a year ago at CU Boulder. And our interest is in kind of bringing an interdisciplinary perspective into, you know, understanding how the design of technology impacts how people experience it and how technology works in society. And a lot of our work has focused on accessibility in data visualization for blind and low vision screen reader users, where we're working with blind and low vision researchers and screen reader users to work together to design tools for working with data non-visually. And we also do research on things like data ethics and the way that people kind of respond to unwanted data collection and how people are kind of reacting to new changes in technology that are brought about by things like AI and things that are kind of rapidly changing the way that data collection happens and making it more pervasive in society.
3:03Jonathan Zong: And you mentioned it being an interdisciplinary group. Can you share a few details on what areas of scholarship or what field you draw from?
3:12Kyle Polich: So I think, you know, for a lot of these really important questions that people are facing with technology and society, we really need multiple kind of approaches to fully understand a lot of these problems. And HCI is kind of already an inherently interdisciplinary field. I'm coming from a background where I studied computer science, and I also have a background in visual arts and graphic design. I mean, I think that kind of naturally led me to some of my interests in data visualization. But I think we also have a lot to learn from people who are really good at understanding, you know, kind of the history of how technologies are proliferated in society, right?
3:51Kyle Polich: So these are fields like science and technology studies and anthropology and things like that.
3:56Jonathan Zong: Well, I haven't surveyed the listenership for this question, but I would guess most of them are sighted and not taking advantage of a lot of the accessibility features on their computers. So would you mind for what I presume is most listeners sharing a few details on maybe what are the challenges or what are the solutions available for people who need accessibility options?
4:17Kyle Polich: Most blind and low vision people who are using computers are using something called a screen reader, which is an assistive software that you can either download or sometimes it's built into operating systems. So for example, macOS has a screen reader called VoiceOver built in. And this is a program that kind of reads out content on the screen as text to speech. So there's a cursor that a user is controlling with keyboard commands, and it's kind of reading out whatever is highlighted with the cursor. And of course, many people are using multiple assistive technologies. So there's also screen magnifiers for people who have partial vision.
4:56Kyle Polich: There's also braille displays that people can plug in and are compatible with screen readers so that the text that would be read out of speech is also displayed tactilely. And so when we think about, you know, the way that screen readers work, it's reading textual content aloud. And so when you encounter image-based content, for example, whether that's, you know, memes you share on social media or data visualizations, which are kind of more complex images in some ways, The person who, you know, created the website or posted the post usually has to provide something that's called an alt text or alternative text, which is a description of the image that can be read out by the screen reader.
5:36Kyle Polich: And when that's not provided, usually, you know, you get something like image, image, or like a file name. And none of that's really informative for someone who's trying to kind of have a comparable experience of someone who's cited and looking at an image.
5:50Jonathan Zong: Well, broadly speaking, how often is that alt text provided? And what's provided is something useful to a person who wants the content?
5:58Kyle Polich: Yeah, that's a great question. Because I think, as you say, there's kind of the baseline, is there a description, which often there isn't. And, you know, I think even though some platforms are getting better about reminding people to add alt text to images that they share, a lot of the time, those are kind of skipped over. And even when there is a description, it's not always useful, right? I know like for a while, not so much anymore, but like there used to be kind of SEO hacking, right, where people would put information in the alt text that was meant to boost the search relevancy of whatever page it was on, but it wasn't actually describing the image, right?
6:34Kyle Polich: And so that would be like a very negative experience of an alt text. But sometimes, you know, even when people are well-intentioned, like thinking about accessibility, it's hard to know what information would be useful in an alt text, right? And I think part of it is there's a lot of guidance out there, but it's just kind of hard to write a single description that fits every situation, especially when you consider something like a data visualization. People read data visualizations in a lot of different ways. You might be looking for a high level pattern in the data. You might have a specific question that you're trying to answer.
7:08Kyle Polich: You might be just kind of trying to look up a specific number. Right. So I think what makes a description useful also depends on what a reader is trying to accomplish, which, you know, makes it very hard to kind of create these one size fits all accessibility solutions, which is part of the challenge.
7:24Jonathan Zong: Well, I'm a big fan of a guy named Edward Tufte, who you might know, but I'll share a few details for listeners who don't, and you can chime in as well if you'd like. One of his key concepts is the idea of chart junk. We want, if we're going to do a visualization, really boil it down to we want the reader to understand something from this. We want them to get, I don't know, the moral of the story or a sense of the distribution. but I could also imagine where modern like image recognition technology might look at an image and say it's a bar graph with red bars, which tells me frankly nothing about it.
7:57Jonathan Zong: You know, what are I guess some of the open challenges in getting things to work specifically in the data visualization world?
8:03Kyle Polich: Just to kind of like riff on, you know, some of Tufti's ideas a little bit, like I, you know, learned a lot from Tufti's books when I was learning data visualization, right? And I think he's been very influential in the field. But I think also more recently, like people in the Viz community have kind of revisited some of those ideas, right? Like chart junk. You know, I think a lot of artists and data journalists, right, have also been exploring how a lot of these decorative elements can actually enhance someone's understanding of what data is trying to communicate. There's a lot of information about like the subject matter of data visualization, for example, that can be conveyed through just graphical presentation of data beyond like, Is it just the simplest bar with a simple color scale?
8:46Kyle Polich: So I think when we think about data visualization as communication, it's usually situated within, you might find a data visualization in a news article or in a PowerPoint someone's giving. So there are a lot of things that are being communicated that's not just kind of raw numbers. And I think the way that translates to accessibility, too, is like my colleague Alan did a project where he was classifying all the different levels of content that when you write a description of a data visualization, what about the chart are you actually describing? And there's, you know, low level features like, oh, like what color is the bar, right?
9:25Kyle Polich: Like the example you just gave. There's things like what are the axis labels and things like that. But there's also higher level things like, are there statistical patterns or outliers? Or are there kind of perceptual things happening? Like, I don't know if you've ever seen the updated version of the Anscombe Quartet, Datasaurus Dozen. Do you know what I'm talking about?
9:45Jonathan Zong: I don't know the Datasaurus Dozen, but I know Anscombe's Quartet.
9:48Kyle Polich: Yeah, it's kind of a similar idea. So Anscombe's Quartet, like the idea that a data set could have like the same summary statistics, but visually be in very different shapes. The data source doesn't. It's just kind of another variation of this. But I don't know, it's funny and interesting because one of the visualizations is like the shape of a dinosaur. But, you know, the summary statistics are kind of normal. And, you know, for something like that, right, like, if you're only focused on describing the data, you might kind of miss out on, like, actually, it kind of matters that this chart is in the shape of a dinosaur.
10:18Kyle Polich: And then finally, you know, there's an even higher level of current events like social and political context, right? And especially with the way that data visualizations are kind of increasingly used to communicate really important information, right, about climate or elections or COVID, right? I think part of the challenge is like when we as people are writing these descriptions, do we have a clear sense of like who our audience is, what the message is that we're trying to convey, how things outside the visualization might be supporting or not supporting that message. Right. And, you know, that's before even we get to kind of how systems might automate something like this.
10:58Jonathan Zong: So it seems like a surprisingly difficult problem then to maybe like infer missing alt text for a data visualization. I'm sure we'll unpack that a little bit, but I guess I'm curious, and as best I can tell, you're a fully sighted person. How did you first take an interest in this challenge?
11:14Kyle Polich: I usually say, like, I think it's kind of right people, right place, right time. I'll talk about maybe the time first. I was doing my PhD at MIT in the MIT visualization group, advised by Arvind Satyana Ryan, and we were working on kind of tools for creating interactive visualizations and, you know, thinking about how to communicate data more effectively. And then around the year 2020, pandemic happens. And, you know, we were noticing, like, it felt like kind of data visualization was having this big moment, right, of being really central to how people were communicating around the pandemic. Like, if you think of flatten the curve, this is probably like one of the most well-known data visualizations, maybe most consequential recent visualizations, right?
11:57Kyle Polich: And we were kind of noticing the way that as we were relying more on visualizations to really, you know, cleanly and effectively communicate some of these ideas, we were also kind of experiencing this limitation of visualization, right? Which is that it's excluding people who aren't sighted. And, you know, I think there's this interesting tension that we were noticing. And around the same time, I was kind of collaborating with my lab mates and friends, Alan Lungard and Crystal Lee. And we had recently met my now friend and longtime collaborator, Daniel Hayesh, who is blind and is currently a researcher at University College London.
12:38Kyle Polich: And we all kind of just got to talking and we were thinking about, you know, some of these issues around accessibility that were specific to data visualizations, right? Some of the things I was just talking about, about why data visualizations might be, you know, because they're read differently often from other types of images like photographs or whatever, where we might be able to kind of think about, you know, revisiting how can we create, you know, better, more effectively communicative, maybe more comparable accessibility. experiences for screen reader users to the ones that sighted people often enjoy.
13:13Kyle Polich: And yeah, what this kind of turned into was this, I think, like six month long process where we were all kind of meeting on Zoom every week. And we were engaging in this method called a co-design with Daniel as the co-designer who is bringing not only expertise about assistive technology and kind of the experience of being a screen reader expert, using one every day, but also, yeah, more broadly, that lived experience of disability, right? And that understanding of what issues would be really important for screen reader users. And I think this was a really transformative experience in my research career, because I think, you know, we were like brainstorming in Zoom, and then like, we would pass around some prototypes of, you know, screen reader interfaces that we were making.
13:59Kyle Polich: And then like Daniel would try them, and then he would send back these really detailed emails, a lot of notes. And that was so valuable because as a sighted person, maybe I bring a lot of expertise in interface design and data visualization, but I don't have that experience of being a screen reader user and living as a blind person. So often when Daniel would respond with all this feedback on prototypes, there would be a lot of things that would really overturn my intuitions and also just kind of have really interesting kind of juxtapositions with kind of like conventional wisdom in HCI, right?
14:38Kyle Polich: Because the theories and the frameworks we learn are also very primarily centered on graphical user interfaces. So I think for me, it was that experience of kind of, you know, being really clear about what expertise I did and didn't have and kind of bringing that intellectual humility into a collaboration was really productive, I think, for the work. And, you know, Daniel and I have been collaborating on a lot of the follow-up work to the study that we'll probably spend the most time talking about in this conversation. But yeah, I think, you know, there's a disability activism slogan called nothing about us without us.
15:13Kyle Polich: I think that kind of encapsulates the both like the kind of social importance of having people with experiences of disability involved in the design of the things that they will then use. And also just kind of the methodological importance of like being able to validate your designs and your, you know, hypotheses with someone who would know.
15:33Jonathan Zong: Are there any key insights from that co-design process you'd want to highlight?
15:38Kyle Polich: You know, one of the things that really struck me was like the way that Daniel talked about like some of the experiential aspects of using some of the prototypes that we were developing. So for example, when you're looking at, you know, a graphical interface, you can kind of see all the things that exist on the page, right, and are available. And then you can kind of like get this really quick overview of everything and then maybe, you know, pick things to focus on based on the information hierarchy, right? Like if you're on a page with like a very clear header structure, you can kind of like skim the headers and then like find the thing that you actually want to focus on and things like that.
16:15Kyle Polich: And this like overview detail idea is often not present in a lot of screen reader interfaces, at least by default, where because things are being read aloud, there's kind of implicitly you have to like linearize the interface into like an order that you can kind of arrow key navigate through. So, you know, in some of our prototypes, for example, we would try these different like ways of organizing things. You would try more like hierarchical structures, like more like a tree than a list. And Daniel had really interesting observations about like when you have like a finite list of something, right?
16:48Kyle Polich: He kind of compared it to being in a house where like you are in like this bounded room and you kind of have these boundaries and know where the doors are. And like, so you can kind of like mentally divide the space into like meaningful sub areas. Whereas, you know, if you just kind of gave everything in like an unstructured list, you would have to do more work to build that model in your head, but not have it kind of be reflected in the interface. And then as another example, we were testing out some of these interactions where you would maybe like navigate to a different part of the chart and then you would maybe press the down arrow to kind of increase the level of detail and it would break down the part of the chart that you were highlighting into maybe like sub intervals along the axis or something like that, for example.
17:33Kyle Polich: And I remember, you know, in the paper, Daniel wrote a bit about how this felt kind of in his head metaphorically, like he was zooming into the data, which I felt was really interesting because it made me think about how a lot of our kind of interaction, like data exploration metaphors, like panning and zooming are kind of implicitly like, if not visual, at the very least, like very spatial. So kind of being able to give some of those affordances of like spatiality in these screen reader interfaces, I think was a really cool thing to discover.
18:05Jonathan Zong: Well, I think we're getting close to the design dimensions that are one of the main contributions of the paper. I want to get to those. But I guess before we delve right in, I know you're working on really the screen reader technology, but just out of curiosity, is there any interest maybe in other academics or in the community or with Daniel with people like that in something more tactile? You know, could there be a graph that I'm touching? Yeah.
18:30Kyle Polich: And we actually have done some follow-up work on tactile graphics. we made this prototype system called Tactile Vega Light, which is an extension of Vega Light, which is a common visualization toolkit in JavaScript. And maybe some people know the kind of Python equivalent, which is Altair. And yeah, with tactile graphics, it's kind of this whole other thing, right? Where I think a lot of the theme of the work in accessibility is this idea of modality differences. The idea that just our senses kind of all work differently from each other, right? And the challenge in tactile charts is kind of, you know, we're thinking about either charts that are printed out on embossed paper with an embosser, or, you know, some people are 3D printing tactile charts, or there's also kind of these new refreshable tactile displays where it's kind of a pin array, and the pins kind of raise and lower programmatically, and you can convey pictures that way.
19:30Kyle Polich: The resolution is really low, and like some of the refresh rates are kind kind of slow. So they're getting there. And I think part of the challenge is like the way that we see things is just very different from the way that we, you know, understand things through touch. I saw this metaphor someone made of like using a tactile map, for example, is kind of akin to like if a sighted person were like looking through a small tube at a map, right? Because you're kind of feeling locally with your hand or hands. And so you're kind of piecing together the big picture from like the parts, right? Whereas visualization, you often kind of get the big picture, at least implicitly, like more immediately, and you can kind of break it down into parts.
20:12Kyle Polich: And so with Tactile Vagalite, we were kind of thinking about, you know, how do we help people create tactile charts more easily? And we were doing a lot of research on some of the best practices that folks like the Braille Authority of North America have put out, like people who publish tactile books. And it turns out that even when you follow kind of all of the best practice guidelines, you can create these charts, but there's still more or less visual forms that have just been kind of printed out tactilely, right? And when you think about a bar chart, for example, bar charts are very useful because like we have visual perception that's good at perceiving like relative length against a baseline or like things like area, right?
20:50Kyle Polich: But like tactilely, if you're feeling a bar chart, really what you're interested in is just like the ends of the bar, right? And so there's all this like space within the bar that does something visually, but is actually not that informative tactilely. So I think there's a lot of exciting work to think about, you know, how do we create these tactile first experiences of data? But yeah, lots of people are working on this, not just me. So I know, like, for example, Kim Marriott at Monash is doing a lot of work with, like, refreshable tactile displays. Joo Young-so at Illinois has done a lot of cool work with Braille displays and data, right?
21:25Kyle Polich: Yeah, I think a lot of the work that we've done with screen reader interfaces kind of also gives us a starting point to like think about maybe analogies to other modalities and I've also done some work with multimodal systems that kind of combine like visualization and sonification and text and because all these senses are kind of work differently and are kind of doing different work I'm increasingly of the opinion that like multimodal systems are kind of the way to go because they just they kind of just give everyone more options for understanding the data in different ways. And even if you're fully cited, a lot of people have told me that they also like having text or like something to listen to alongside the chart.
22:04Jonathan Zong: Well, feel free to correct me if I'm wrong, but I guess in my reading, the paper's core contributions in terms of the design dimensions are those three, the structure, the navigation, and the description. Could you give some loose definitions on those and how they work towards your goal?
22:20Kyle Polich: So the goal of this project, you know, we set out to kind of think about what would screen reader experiences of visualizations be like that were kind of more comparable to the way that sighted readers are reading visualizations. We ended up framing our contribution as a set of design dimensions of screen reader interfaces for visualization. We were trying to think about like, what are the things that someone, a designer would have to consider if they were making any type of data visualization interface, right? So we made a bunch of prototypes and we were kind of almost trying to like explore this design space of all the different ways that you could make screen reader versions of charts.
23:01Kyle Polich: We settled on a few that we liked the most and kind of built those into OLLI, but we recognized that, you know, there's a lot of different ways to do this, right? So what we were trying to do was create this framework that would be helpful to people who are trying to think about this systematically so that they can kind of make decisions about what's appropriate for their system. And so our three design dimensions that we identified through this prototyping process and this process of, you know, co-design and reflecting on our prototypes and exploring different variations, our design dimensions were structure, navigation, and description.
23:31Kyle Polich: I can kind of talk about each of these one at a time. So starting with structure, by structure we mean, you know, how do you represent a data visualization in a way that can be traversed by a screen reader? I mentioned before, right, there's this idea that you have to kind of like linearize things into things that are like selectable by a cursor, and then there will be a description read out, right? So we were kind of thinking about structure as like, yeah, like what is the data structure that a screen reader user is kind of traversing? You can arrange things as a list, but that might be really annoying because, you know, you might like if your scatterplot has like 500 dots on it, and you have to go through those one by one in the list.
24:09Kyle Polich: Like that's pretty frustrating. And so we explored other kinds of structures like trees and graphs. With trees, we could add some hierarchy so we could like group things together. You know, part of it is also deciding what the nodes represent. Where we ended up, for example, was maybe like there's a root node that's kind of an overall summary of what the chart is. So like, is this a scatterplot or a bar chart or what is it? And like, what are the axes and things like that? So for example, if someone is running into this on a web page that's like a news article or something and they see this chart, they can kind of get an idea of what it is through like a very quick overview.
24:45Kyle Polich: And if they're like, whatever, I don't care about this chart, they can just go to the next thing, right? They don't have to like go through every single scatterplot data point just to get past this chart. But if you want to go kind of down one level in the tree hierarchy structure, then we had nodes that represented the axes and legends. So you could get more information about the x-axis is this field. It has values from this to this. And then you could break those even further down into, okay, this has like these categories or these are the intervals along this axis. And then finally, at the very bottom level, like, you know, the leaf nodes of the tree, you could see the individual data points.
25:22Kyle Polich: And so this gives people more of like an option to kind of control what level of detail that they're exploring at and also, you know, maybe like skip over parts of the chart that they're not as interested in so that they can save time. So that was structure. Second dimension is navigation. There's the obvious way, which is, you know, moving along the structure itself. So like for the tree we had up was like up levels of the tree and down was down levels of the tree. And then like left and right is like siblings on the same level. But there are also other ways we could imagine. So for example, spatial navigation, if maybe up and down mean like towards the top and bottom of the chart rather than up and down the abstract structure, right?
26:02Kyle Polich: Maybe for a scatterplot, that might be more interesting and important so that people can kind of understand, you know, spatial layout of the data and how it's presented visually. And then, you know, we were also thinking about, oh, like maybe people kind of already know that there's a thing that they're looking for in the chart and they just want to find it, but they don't want to figure out like what sequence of arrow keys gets them there. Right. They don't have to like search for it. So we prototyped something we were calling targeted navigation, where you kind of just use menus to kind of directly jump to a place.
26:34Kyle Polich: And then maybe you can explore locally with the arrow keys once you've got there. Finally, we were thinking about description, and we kind of broke this down into three parts. One is the content. So I was talking a bit about this earlier, but what is it about the chart that you're actually describing, whether that's low-level features like encodings and colors and stuff like that, or higher-level features like the context of a chart? If you have a stock chart that includes 2008, you might want to say something about the financial crisis, right? Yeah, so there's what are you actually describing?
27:07Kyle Polich: there's kind of what order are you putting the information in? Because in addition to data about the chart, there's probably also kind of like useful navigation and wayfinding related information that you might want to have. So for example, if you have a list of categories, right, it might be helpful to have like one of two, two of two, right? And, you know, often this type of thing is like built into screen readers, but, you know, not always, or sometimes people have it turned off, but also, you know, like depending on what you're trying to do, right? if you're trying to find the third item in a list, having the index at the front is probably very useful.
27:43Kyle Polich: But if you're trying to find a specific category, for example, like if you have to listen to, you know, one of seven, two of seven, three of seven, and you're just really just trying to listen for what the category is, maybe it's better for the category to be upfront, right? And then finally, we were thinking about verbosity of descriptions. So how many words are you actually using to convey an idea, right? So for someone who's maybe very new to navigating a system like this or new to data visualizations. Maybe the description could be like access with values from 50 to 70. And then for someone who's like used the system a bunch of times, kind of knows a lot about what they're expecting each description to say, right?
28:22Kyle Polich: It could just be like access 50 to 70 and that would be equally fine, right? For that user. So we did some follow-up work to kind of developing interfaces for people to actually customize these descriptions and like kind of change Change what information is actually included, change the ordering of things, change the verbosity level. I think that helps enable kind of the flexibility to adapt a description to what a reader or data analyst is actually trying to do with the data.
28:50Jonathan Zong: Well, I'm hoping we can zoom in on Oli a little bit. Can you remind the users what it is and maybe comment on who the audience or the users of it are?
28:59Kyle Polich: Oli is the open source library that we created kind of based on some of the things that we found in our research and some of our prototypes. And it's a JavaScript library that you can kind of, you know, NPM install and then you can include it in a web page. And you can basically either give it an existing visualization specification that's in an existing format like Vega Lite, or you can kind of manually specify in Oli, you know, a data set and fields and things like that. And it will output a keyboard navigable kind of tree-like structure with all these descriptions at varying levels of detail and kind of convenient key bindings for people to move around.
29:40Kyle Polich: And I think, you know, the audience, there's of course like the developer audience and the user audience. And, you know, we're really intending these to be useful for screen reader users who want to kind of explore data on their own, right? And, you know, when you have an alt text, you kind of just have to, you know, accept the choices that whoever wrote the alt text made about what information was important to include, maybe what conclusions they decided to include or not include. Often, you know, people who are writing alt text make choices about like what level of detail to convey the information at because you're always trading off like how long is this?
30:15Kyle Polich: Like how much do I have to listen to versus kind of getting the point across? And so with Oli, we're kind of hoping that it's helpful for screen reader users who kind of want to do that independent, like self-guided data exploration where you can, you know, just like a sighted reader visualization would maybe looking, look at the overarching patterns, drill down into individual parts of the visualization, right? And then kind of piece together conclusions for yourself or at least verify, you know, things that are being described using the underlying data. And then on the developer side, yeah, I think, you know, we're hoping that it's kind of lowering the barrier to create baseline accessible experiences, right?
30:55Kyle Polich: Because accessibility is hard. Like, I recognize that, you know, I had to do a whole PhD to kind of know how to, like, you know, write a good alt text or whatever, right? And so I'm very like, like, accessibility is very important. And like, we can't make excuses for like, not making things accessible. But I also recognize that it's super hard, right? So I think that's why like building toolkits like this that people can kind of drop in can, I think, maybe like raise the quality of like the baseline experience of visualizations. Right. And we've, you know, we have this JavaScript library. We've also built Oli as a render in Altair, actually.
31:31Kyle Polich: So if you're if you're a Python person, you can check out, you know, how to write Altair programs that output an Oli output in addition to a visualization.
31:40Jonathan Zong: So I'm kind of wondering if, you know, let's imagine I was a software developer at like an online newspaper and we do a lot of charts, but they're, you know, rasterized charts. Yet I want to make them more accessible. Can I just NPM install Oli and then, you know, put Oli tags around my images and everything's fixed? Or do I go through and then do some sort of process of annotating every image so it has the Oli format?
32:05Kyle Polich: So if you have existing charts that you made either in a different visualization library that's not Vigalite, or you just have an image that, you know, you have a designer made it in Illustrator, right, you can still use OLLI. One is you can just write an OLLI specification where you include the data set, you kind of include the encodings, right, where you map the fields to, you know, what would be the visual properties in your chart or even just kind of generic axes and legends. And, you know, you can add additional information about how certain things ought to be described. And that will generate an OLLI output kind of without, you know, an input visualization.
32:45Kyle Polich: So you can put that parallel to your existing PNG or whatever, or in addition to any alt text you may have already written. If there are other existing libraries that you use to make visualizations, I mentioned earlier that OLLI is extensible. And so I think part of what I'm hoping to do, you know, in the near future is actually do some more research where we're collaborating with folks like data journalists and trying to understand their existing workflows and, you know, what kind of challenges they face with accessibility and maybe also just like tech infrastructure, right? Like, what does it take to get a package approved for use in your publication or whatever?
33:23Kyle Polich: And yeah, think about kind of bringing these tools closer to where people are actually facing some of these challenges of like, we're on a deadline, but we need to make things accessible. And, you know, we know it's important. And yet, like the resources aren't always there. Right. So I think, yeah, the more we can think about kind of those real world sites of use, the better it will be.
33:42Jonathan Zong: Can you share some details on the feedback you got from participants in your study when they got to experience Ali and its outputs firsthand?
33:50Kyle Polich: When we were doing this research, we, you know, we were doing co-design with Daniel. But of course, you know, we knew that we couldn't just design things amongst ourselves without making sure that we were making things that worked for more people and weren't just kind of reflecting the opinions of people within our group, right? So we ended up doing this user study evaluation with 13 blind and low vision participants, where we took a couple example prototypes that worked in different ways and showed them to people to kind of compare them with both their baseline experience of the ideal accessible visualization that has an alt text and maybe has the data table attached versus some of our prototypes.
34:29Kyle Polich: And we weren't really trying to see if it was better, for what definition of better, right? But I think we were really trying to understand how some of the choices that we were exploring in the prototypes were affecting the way that people thought about the data, right? So we were kind of trying to understand the influence of our design decisions on people's experience and understanding of the data. And I think we heard a lot of interesting things. So like people said things like, I really like having the ability to scroll through at a higher level and then drill down deeper if that's of interest.
35:02Kyle Polich: So that was kind of telling us that our idea of like adding some hierarchy and this idea of multiple levels of detail was pretty helpful for people to have that kind of drill down exploration strategy that, you know, a lot of sighted readers of visualizations are also using. And some people said things like, you know, it gives me different ways of thinking. I'm thinking more in spatial terms, right? And I think that's also another advantage of having something you can kind of interact with and navigate around rather than a static alt text that just kind of blasts a paragraph at you, is that we can start to develop some of those spatial metaphors, Kind of like Daniel was saying with like the zooming metaphor.
35:40Kyle Polich: But I think we also, you know, very understandably heard things from people that were kind of more cautious, right? Someone was like, you know, I think this might be understandable to someone who's done this before, but I don't, you know, really do well with these charts unless they're converted back into tables. And what that reflects is that people who are using screen readers every day have very kind of personalized workflows that they're used to. And there's also kind of conventions and, you know, what common practices are that we have to kind of understand. Right. And so we can imagine maybe a situation where, you know, you have to put up a lot of upfront effort to learn how to use an interface like Oli.
36:20Kyle Polich: Maybe after you do that, it could be worth it for like an advanced user who really wants to understand the data. But also maybe for many people, it's like not worth that effort. Right. And I think that's a totally understandable reaction for someone to have. So I think, you know, we can think about how to make these things more learnable, obviously. But I think there's a broader point about, you know, we have very established user experience conventions for graphical interfaces. there's like common sets of components and most websites kind of have one of a few pretty standard layouts, right? But with screen reader interfaces there's so much chaos, right?
36:54Kyle Polich: I don't think that users have really come to expect like a consistent experience from app to app or like website to website. And so I think that's another reason why reusable toolkits like Oli that people can kind of drop into their own websites could make a difference is kind of establishing more of that consistent design language around screen reader interfaces.
37:14Jonathan Zong: So I know there are different perspectives in the data visualization world. I'll maybe, I don't know, steel man or straw man, two of them, but one is to say that every choice I make from my axis and my line thickness is a decision I make and I'm narrating my chart. And maybe there's another that says I'm really just trying to give you the unopinionated data and you were to draw your own conclusions from it. Do philosophies like this get involved in Ali in any way?
37:41Kyle Polich: Absolutely. I think that's a really good kind of encapsulation of the two poles of this conversation. I think increasingly, like I'm seeing a lot of really cool work. My former colleague, Amy Fox, who's at UCSD now, did really interesting work, for example, where she did a study where they basically took a bunch of charts from a bunch of different places on the internet, made in a bunch of different tools, like some from, you know, Reddit, some from Tumblr, some from kind of more like data artist circles, right? redacted all the information that was about what the data was about. So like no titles, no like labels or anything.
38:19Kyle Polich: And then just showed people the charts. Even for the most simplest like chart junk free visualizations, people were instantly kind of looking at the graphic design of the chart and like making assumptions about the person who made the chart or like what tools they used, right? For example, I don't know. Oh, this looks like it's kind of the default settings in matplotlib. So like a scientist must have made this. But like we have no idea, right? And then vice versa, there's the work of folks like Nadia Bremer and like Stephanie Pasovich, right, who illustrate these really like evocative and kind of communicative data visualizations that are very stylized, but very effective.
39:00Kyle Polich: Like they've won a lot of like journalism awards, right? And then, you know, with all the information redacted, people will be like, oh, like it looks like someone hand drew this. So they probably didn't know what they were doing or whatever, right? So I guess what I'm trying to say is that I think regardless of whether or not we think we're objectively conveying the data on the receiver end, like when people read things, they will bring their own subjectivity to it, right? And kind of impute subjectivity onto you, even if you don't intend it, right? Through just the choices that you make. Yeah, your question about how this translates to accessibility and Oli is like really interesting.
39:32Kyle Polich: And yeah, I would love to think about this more. But I think my initial instinct is like, yeah, we have made this kind of very templatized, standardized tool, right? And I think we've made a lot of default decisions about like what the structure should be and like what, you know, the default orders of descriptions should be and like what should be included. And I think defaults are so powerful, right? We've given people the ability to customize things, but just based on what we know about how things work in HCI, most people never touch the defaults. And so I think there's an element of, yeah, those decisions have a lot of power over interpretation and the way that people kind of perceive the authoritativeness or objectivity of something.
40:14Kyle Polich: I also think that, you know, I would love to think more about how do we make accessible the more expressive parts of data visualization, right? You have this really intricately illustrated visualization that, you know, was handcrafted by an artist versus a plot that is the default matplotlib output. Should those descriptions be the same, you know? And I think there's a broader idea in accessibility that, like, for so long, screen reader interfaces have been very purely efficiency focused. I think there's very good reason for that. You know, a lot of what the BLV community felt was very important is, you know, being able to like be part of conversations in the workplace and like, you know, be on a team and be able to do their work at the same pace as anyone else.
40:57Kyle Polich: Right. And with the same efficiency. but I think what that means is there's like a huge opportunity to think about kind of the expressiveness and like the aesthetics of screen reader interfaces and you know how do we enjoy things like art and poetry like through screen readers right and I know there's lots of other folks thinking about this too there's a project called all text as poetry which is kind of as the name describes right kind of trying to get people to think about descriptions beyond kind of the utilitarian aspect of it. And yeah, I just think that's a very rich area to think about kind of, you know, before we mentioned the levels of like, is there even a description?
41:37Kyle Polich: Is the description useful? I think the level after that is, is the description like enjoyable to read, right? I'll just say that if people are interested in learning more, you know, my research group's website has some of our more recent work that's kind of built on Oli. So I mentioned before, like the customizable description project, the tactile charts project. We've also done some work with how do we make tools that enable blind and low vision users to create their own visualizations and not just kind of consume visualizations made by other people. More recently, we've done some work kind of collaborating with an oceanography lab led by a blind scientist where we were kind of looking at the ways that they use kind of multi-sensory data representations like charts and descriptions and like, you know, 3D printed tactile maps and things like that to kind of collaborate and do science, right?
42:28Kyle Polich: And how those representations kind of facilitate communication. So yeah, I think there's like a whole wide world out there in data visualization accessibility.
42:38Jonathan Zong: Well, Oli is a relatively new project and growing project. Do you have any idea about aspirations you have for it in the next couple of years or maybe what a fully deployed project looks like where all of our accessibility needs are solved?
42:55Kyle Polich: Yeah, I mean, I'm still super committed to kind of like building on Oli, both as like a thing that's a useful instrument for us to do research on and kind of learn more about what's, you know, what are good experiences for screen reader users, but also just as a tool that people can use. So, you know, for example, some of the things we have coming down the pipe, there's a student that's working with me who's really interested in kind of accessible mapping, and thinking about how geospatial visualizations pose a lot of unique challenges around what regions are adjacent to other regions and what politically and culturally defined ways of grouping geography are and how we can apply similar approaches in OLLI to maps.
43:37Kyle Polich: I think longer term, I'm also hoping that the approach that we've kind of set out with in Oli that's really building off of Vega Lite and kind of other declarative specifications is, you know, how do we kind of think about broadening this from data visualization and think about kind of improving the accessibility of like web applications in general? And, you know, we have a lot of like existing very important standards around that. WCAG, the Web Content Accessibility Guidelines, and ARIA, which is the markup language where you annotate HTML with additional accessibility metadata. But I think the really nice thing that we've learned in the visualization community from grammar-based approaches like VegaLite is that when you have declarative specification where you're kind of writing visualizations with high-level concepts like fields and encodings rather than rendering-level concepts like you might write in something imperative like D3, you get all these kind of nice properties.
44:36Kyle Polich: you have more kind of guarantees about like if a program compiles, it'll also produce like a good output, right? You also get nice like compositional ideas where like if you put together two subcharts like in a layer or whatever, you just automatically get this other type of chart. And I think there's a lot we could learn from those almost like programming language design ideas to this process of like making the web accessible because, you know, So guidelines are guidelines, but for a developer who is trying to put these into practice, it's so like situational. It's so easy to mess up, right? And we're all kind of reinventing it from scratch every time because there's a lack of consistency.
45:17Kyle Polich: So yeah, I'm kind of longer term thinking about these ideas of structure, navigation, description, where visualization could, I don't know, maybe possibly be like this special case for us to study things that might be applicable more broadly to like graphical user interfaces in general.
45:36Jonathan Zong: What are some opportunities for grad students or maybe an ambitious undergrad in that area?
45:41Kyle Polich: Ah, yes. Well, I am actively recruiting PhD students. And that could be, you know, if you're an undergrad who's really interested in doing this kind of human computer interaction research. Or that could be, you know, if you've like worked for a while and you kind of want to take the skills that you've developed in industry towards some of these problems that kind of unfold over like a longer time, right, where we get to kind of go deep into questions and, you know, work with communities that, you know, you might not have time to work with in like a product cycle and industry, right? That's definitely, you know, an opportunity to work together.
46:18Jonathan Zong: Yeah. Where can listeners follow you online?
46:21Kyle Polich: Yes. So these days, well, first of all, you can go to either my personal website, which is jonathanzong.com, or my research group's website, which is data-and-design.org to, you know, get updates about the work that we're doing. You can also find me on social media these days. I'm mostly on Blue Sky and LinkedIn. And I should just come up if you search my name.
46:49Jonathan Zong: we'll have links to all the above in the show notes for listeners to follow up Jonathan thank you so much for taking the time to come on and share your work thank you so much yeah and thank you for the interesting questions
Kyle speaks with Jonathan Zong about making data visualizations accessible to blind and low-vision users through screen readers, interactive tools like Olli, and inclusive design approaches that enable independent data exploration.