Chart Junk: A Magnet for Misguided Research

Review of the Research Study “What Makes a Visualization Memorable?”
Michelle Borkin, et. al. (Harvard School of Engineering and Applied Sciences and MIT)

No topic within the field of data visualization has created more heated debate over the years than that of “chart junk.” This is perhaps because, when Edward Tufte first introduced the concept, he did so provocatively, inviting a heated response. Ever since, this debate has not only flourished without signs of cessation, but it has generated some of the least substantive and defensible claims in the field. I’ve contributed to this debate many times, always trying to rein it back into the realm of science. Whenever a research study that appears to defend the usefulness of chart junk is published, the Web immediately comes alive with silly chatter, consisting mostly of chest thumping: “Ha, ha! Take that!” The latest study of this ilk was presented this week at the annual IEEE VisWeek Conference by Michelle Borkin, et. al. (students and faculty at Harvard and MIT), titled “What Makes a Visualization Memorable?” Yeah, you guessed it, apparently it’s chart junk.

When I last attended VisWeek in 2011, my favorite research study was presented by this same researcher, Michelle Borkin. Her study produced a brilliant, life-saving visualization of the coronary arteries that could be used by medical doctors to diagnose plaque build-up that indicates heart disease. It was elegant in its simplicity and clarity. Borkin’s latest study, however, does not resemble her previous work in the least. Here’s the paper’s abstract in full:

An ongoing debate in the Visualization community concerns the role that visualization types play in data understanding. In human cognition, understanding and memorability are intertwined. As a first step towards being able to ask questions about impact and effectiveness, here we ask: “What makes a visualization memorable?” We ran the largest scale visualization study to date using 2,070 single-panel visualizations, categorized with visualization type (e.g., bar chart, line graph, etc.), collected from news media sites, government reports, scientific journals, and infographic sources. Each visualization was annotated with additional attributes, including ratings for data-ink ratios and visual densities. Using Amazon’s Mechanical Turk, we collected memorability scores for hundreds of these visualizations, and discovered that observers are consistent in which visualizations they find memorable and forgettable. We find intuitive results (e.g., attributes like color and the inclusion of a human recognizable object enhance memorability) and less intuitive results (e.g., common graphs are less memorable than unique visualization types). Altogether our findings suggest that quantifying memorability is a general metric of the utility of information, an essential step towards how to design effective visualizations.

The authors collected a large set of data visualizations from the Web. Each visualization was coded by the research team for various characteristics (type of visualization, number of colors, data-ink ratio, the presence of pictograms, etc.) During a test session, subjects were shown one data visualization at a time for one second each, followed by a 1.4 second period of blank screen before the next visualization would appear. Each session displayed approximately 120 visualizations. The test was set up as a game with the objective of clicking whenever a visualization that appeared previously appeared a second time. A particular visualization never appeared more than twice. Hits (the subject indicted correctly that the visualization had appeared previously) and false hits (the subject incorrectly indicated that a visualization had previously appeared when it hadn’t) were both scored, but misses were not. The study’s objective was to determine which of the characteristics that were coded caused visualizations to be most memorable.

Any form of presentation, be it a book, speech, lecture, infographic, news story, or research paper, to name but a few, should be judged on how well it achieves the author’s objectives and the degree to which those objectives are worthwhile. A research paper in particular should be judged by how well it does what the authors claim and how useful its findings are to the field of study. This study does not actually do what it claims. What it actually demonstrates is quite different from the authors’ claims and does not qualify as new information.

The title of this study, “What Makes a Visualization Memorable?,” is misleading. It doesn’t demonstrate what makes a visualization memorable. A more accurate title might be: “When visualizations are presented for one second each in a long series, what visual elements or attributes most enable people to remember that they’ve seen it if it appears a second time?” That’s a mouthful and not a particularly great title, but it accurately describes what the study was actually designed to test. The study did not determine what makes a visualization memorable, but what visual elements or attributes included in the visualization would be noticed when viewed for only a second and then recognized when seen again. A data visualization contains content. Its purpose is to communicate that content. A visualization is not memorable unless its content is memorable. Merely knowing that you saw something a minute or two ago does not contribute in any obvious way to data visualization. And, more fundamentally, remembering something about the design of a visualization is nothing but a distraction. Ultimately, only the content matters; the design should disappear.

When an image appears before your eyes for only a second and then disappears, what actually goes on in your brain perceptually and cognitively? When the image is a visualization, you don’t have time to even begin making sense of it. At best, what happens in that brief moment is that something catches your eye that can be stored as a distinct memory. When the task that is being tested is your ability to recall if you’ve seen the image before when it’s flashed in front of your eyes a second time, then it’s necessary that the memory differentiates the image from the others that are being presented. If a clean and simple bar graph appears, there is nothing unique, no differentiator, from which to form a distinct memory. At best in that single second that you view it the concept “bar graph” forms in your brain, but you’re seeing many bar graphs and nothing about them is being recorded to differentiate them. If you see something with a profusion of colors, that colorful image is imprinted, which can serve as a distinct memory for near-term recall. If you see a novel form of display, a representation of that novelty can be retained. If you see a diagram that forms a distinct shape, it can be temporarily retained. What I’m describing is sometimes called stickiness. Something sticks because something about it stood out as memorable. That something rarely has anything to do with the content of the visualization.

Visualizations cannot be read and understood in a second. Flashing a graph in front of someone’s eyes for a second tells us nothing useful about the graphical communication, with one possible exception: the ability to grab attention. Knowing this can be useful when you are displaying information in a context that requires that you first catch viewers’ eyes to get them to look, such as in a newspaper or on a public-facing website. This potential use of immediate stickiness, however, was not mentioned in the study.

So, when the authors of this study made the following claim, they were mistaken:

Altogether our findings suggest that quantifying memorability is a general metric of the utility of information, an essential step towards determining how to design effective visualizations.

Whether the assertion is true or not, this study did not test it. They went on to say:

Clearly, a more memorable visualization is not necessarily a more comprehensible one. However, knowing what makes a visualization memorable is a step towards answering higher level questions like “What makes a visualization engaging?” or “What makes a visualization effective?”.

Although the first sentence is true, what follows is pure conjecture. The authors seemed to wake up toward the end of the paper when they stated:

We do not want just any part of the visualization to stick (e.g., chart junk), but rather we want the most important relevant aspects of the data or trend the author is trying to convey to stick.

Yes, this statement is absolutely true. Unfortunately, this study does not address this aspect of stickiness at all. Sanity prevailed when they further stated:

We also hope to show in future work that memorability — i.e., treating visualizations as scenes — does not necessarily translate to an understanding of the visualizations themselves. Nor does excessive visual clutter aid comprehension of the actual information in the visualization (and may instead interfere with it).

If they do go on to show this in the future, they will have succeeded in exposing the uselessness of this paper. If only this realization had encouraged them to forego the publication of this study and quickly move on to the next.

If we reframed this study as potentially useful for immediately catching the reader’s eye and that alone, the following findings might have some use:

Not surprisingly, attributes such as color and the inclusion of a human recognizable object enhance memorability. And similar to previous studies we found that visualizations with low data-to-ink ratios and high visual densities (i.e., more chart junk and “clutter”) were more memorable than minimal, “clean” visualizations.

More surprisingly, we found that unique visualization types (pictoral [sic], grid/matrix, trees and networks, and diagrams) had significantly higher memorability scores than common graphs (circles, area, points, bars, and lines). It appears that novel and unexpected visualizations can be better remembered than the visualizations with limited variability that we are exposed to since elementary school.

As I mentioned in the beginning, however, these are not new findings. It’s interesting that finding described in the second paragraph above contradicted the authors’ expectations. They assumed that familiar visualizations, such as bar and line graphs, would be more memorable than novel visualizations. We’ve known for some time that novelty is sticky. The wonderful book by brothers Chip and Dan Heath, Made to Stick, made a big deal of this.

The one part of this study that I found most interesting and informative was a section that wasn’t actually relevant to the study. The authors quantified the number of times particular types of visualization appeared in four particular venues: scientific publications, infographics, all news media, and government and world organization. I found it interesting to note that news media of all types use bar and line graphs extensively, but infographics seldom include them. It was also interesting that tables supposedly appear much more often in infographics than in scientific publications, which doesn’t actually ring true to my experience.

A few other problems with the study are worth mentioning:

  1. The authors created a new taxonomy for categorizing visualizations that wasn’t actually useful for the task at hand. When revealed for only a second, there is nothing that we could reliably conclude about the comparative memorability the visualization types defined by their taxonomy. Because their taxonomy did not define visualization types as homogenous groups, comparisons made between them are meaningless. For example, grouping all graphs together that show distributions (histograms, box plots, frequency polygons, strip plots, tallies, stem-and-leaf plots, etc.) is not useful for determining the relative memorability of visualization types.
  2. They described bars (rectangles) and lines (contours) as “not natural,” but diagrams, radial plots, and heat maps as “more natural” and thus more memorable. From the perspective of visual perception, however, few shapes are more natural than rectangles and contours, which represent much of our world.
  3. I found it interesting that the racial mix of participants in the experiment (41.7% Caucasian, 37.5% South Asian, 4.2% African, 4.2% East Asian, 1.1% Hispanic, and 11.3% other/unreported) was considered by the authors to be “sampled fairly from the Mechanical Turk worker population.” When did Mechanical Turk become the population that matters? Wouldn’t it be more useful to have a fair sample of the general population? A 37.5% proportion of South Asians is not at all representative of the population in the United States in particular or the world in general, nor are 4.2% African and 1.1% Hispanic representative.

I’ve yet to see a useful study about chart junk in the last decade or so. Perhaps there’s something about the controversial nature of the debate and the provocative nature of claims that chart junk is useful (e.g., the possibility of knocking Tufte and Few down a notch or two) that shifts researchers from System 2 thinking (slow and rational) into System 1 (fast and emotional). Despite the flaws in this study, just like the others that have preceded it, dozens of future studies will cite it as credible and people will make outlandish claims based on it, which has already begun in the media.

Take care,

18 Comments on “Chart Junk: A Magnet for Misguided Research”


By Colin Michael. October 18th, 2013 at 12:42 pm

Now I know which book I should write this weekend: “Big Data Visualization in One Second”. Very sure to be a million seller by lunch on Tuesday.

By Stephen Few. October 18th, 2013 at 1:07 pm

What stands out in the first second of viewing a data visualization? That’s right, it’s often the junk. The fact that it’s memorable should encourage us to avoid it. A worthwhile question is, “Can we get the information itself to stand out immediately in a way that clarifies rather than obscures?” Elements or attributes of a data visualization that accomplish this wouldn’t be chart junk, would they? A study that examined this would be useful. We already know, based on years of perceptual studies, many of the elements and attributes that obscure data. Let’s stop beating this dead horse. I’m sure that there are many visual elements and attributes that could be used for good to give data a clear voice. Why isn’t the infovis research community focusing on this? Is it not sexy enough? Is it not provocative enough? Perhaps it’s harder to do. Anyone out there who’s interested in doing well-designed research into what actually works so we can make things better, let me know and I’ll gladly help. Let’s work together to turn this around.

By Matthew Clapp. October 18th, 2013 at 3:11 pm

This is a great topic. At my job, we are just beginning to have an appreciation for good data visualization. There is a confusion between things that glitter (chart junk) and insight. I’ve tested people to see which charts they prefer and time and time again, they choose the chart with the most junk. When tested on comprehension, those that prefer junk, tend not to understand even the simplest visualizations, while those that prefer 1:1 data to pixel, have much better comprehension.

By Jim Wahl. October 19th, 2013 at 1:07 pm

Another interesting example of memorable vs effective is Hans Rosling’s famous TED Talk (one of his first, famous talks, anyway).

Rosling gave the keynote at Tableau’s Europe conference this year. Upon reflecting on the viz (20 years in the making), he lamented that while everyone knows about this viz, few can recall some of the key messages.

Sweden apparently does a survey every year, asking citizens how they want their tax monies spent. At some point they started including knowledge questions. “So you agree with foreign aid to developing nations. What do you know about these countries?”

One question is the average number of children per woman, a key message in Rosling’s viz. He showed the number of correct survey responses the years prior and the year during which the YouTube of his viz was the “more popular than Lady Gaga” in Sweden. You know the answer. No significant change.

By Stephen Few. October 19th, 2013 at 1:30 pm

Jim,

Yes, even though Rosling’s 2006 TED presentation was extremely engaging and many things stuck in the memories of those in attendance, such as the fact that bubbles moved around (something no one there had seen before) and that Rosling’s patter was a bit like an announcer at a horse race, most of Rosling’s message was forgotten. Getting the message to stick is the goal, and nothing that doesn’t feature the message itself can make that happen. Even the best data communicators, including Rosling, can improve. To do so, we must find better ways to engage people with information in meaningful and memorable ways.

By Michelle Borkin. October 20th, 2013 at 10:14 am

Dear Stephen,

I certainly do wish that you had a similar reaction to this work as you had to my 2011 paper. Your reference to the Heath brothers (who credit Malcolm Gladwell with the “stickiness” moniker) seems quite appropriate as they talk of the need for memorability in communicating important ideas. They use “stickiness” to describe the memorable end of the “spectrum of memorability”.

The formal research study done by my fellow researchers and me simply aims to understand what aspects of a visualization move it to the memorable end of the spectrum of memorability. I am hopeful that over time and in light of follow-on research on visualization effectiveness you will see the value of this work.

Sincerely,
Michelle Borkin

By Stephen Few. October 20th, 2013 at 11:19 am

Michelle,

Thanks for responding. It would be helpful, however, if you would address the specific points that I made in my critique. Doing so could help all of us learn something. You’ll find that I am willing to revise my opinion if you can show that I’ve erred.

I have argued that your study does not actually do what you claim. Your study does not reveal anything about what makes a visualization memorable. It does not in any way help us design visualizations more effectively, nor serve as an “essential step” in the process. It’s hard to imagine how the study, as designed, could have possibly done so. At most, your study suggests that particular visual elements and attributes catch our attention immediately when a visualization appears for only a second. That doesn’t make the visualization memorable. It makes a particular element or attribute exhibited in the visualization memorable. Nothing about the information contained in the visualization is remembered. In what sense is it useful for something about a visualization that isn’t information to be remembered? More fundamentally, how could an artificial activity that allows a visualization to be seen for only a second apply to actual uses of visualization, which are meant to be read and interpreted, not perceived in an instant?

I believe that you derived some benefit personally by doing this work. I hope so. I don’t believe that the study has contributed anything useful to the field of data visualization. Nevertheless, we can learn something useful by critiquing the work to understand why it failed. There is no shame in a research failure unless we fail to recognize and learn from it. On the other hand, as we’ve observed in recent years, it is harmful to claim results that are unfounded. Some of the most cited infovis studies of the last few years have been some of the worst. Knowing that your intentions are good, I’m sure that you don’t want to be responsible for another false promotion of chart junk as useful. Now that your study has been released into the wild, you have a chance to prevent this problem, at least in part, by more clearly stating what your study actually reveals, if anything, and what it doesn’t.

By Cole Nussbaumer. October 21st, 2013 at 11:53 am

Stephen, thank you for you clear articulation of the issues in this research and its “findings.” It’s sad to me that those who don’t know better will cite studies like this as a way to exempt themselves from creating visualizations that impart information in a straightforward manner.

You’ve been an inspiration to me when it comes to effective data visualization. It’s clear empirically that reducing clutter in a graph can help improve the communication of information and that building a compelling story and narrative around this helps the information to stick. It’s curious to me that there isn’t better research out there to help prove this point. For now, I guess we can continue to prove it through example.

By Dale Lehman. October 22nd, 2013 at 9:05 am

A very similar issue arises in the standard guidance about creating Powerpoint slides. As Tufte points out, the advice that no more than 7 points should be made on a single slide was a misunderstanding of the research - which actually found that people cannot digest more than 7 UNRELATED ideas on a single slide. So, if a presentation has no real organization then the number of ideas on each slide must be limited. Similarly, it appears that this study has demonstrated what makes visuals memorable, in the absence of anything else that would make the presentation memorable or useful. Thanks for taking this on, as I think your criticism is right on target.

By Jon Peltier. October 22nd, 2013 at 2:55 pm

I think the important thing isn’t whether a visualization itself is memorable. What’s important is whether the message is memorable.

It’s easy to find examples of memorable charts with forgotten messages. Rosling’s presentation itself is memorable, as is the famous chart showing a cartoon leg clad in fishnet stockings. But in neither case it is easy to remember what was the message in the graphics.

Some charts I’ve seen recently about the global temperature anomaly aren’t themselves memorable (they’re simple line charts, after all), but the message is.

Menard’s infographic showing Napoleon’s ill-fated invasion of Russia is memorable and has a memorable message. More recently Nate Silver’s election maps and their messages are both memorable. These are the exceptions.

By Phil Gonzalez. October 25th, 2013 at 2:11 pm

At the risk of being flippant, a train wreck is certainly more memorable than a routine Monday morning car ride to work … however, you would never want your visualization to be considered a train wreck. :)

As many of you have already stated, having a visualization be memorable may actually work against remembering what the message of the visualization actually was.

By Neil Barrett. October 28th, 2013 at 6:38 am

In these cash strapped times, faculty are expected to be prolific and be cited often, because that’s perceived to be useful (even when it isn’t). Sensationalist articles, delivered piecemeal are necessary to one’s career.

It certainly isn’t invalid to probe what people remember as a step along the way to figuring out whether well designed graphics are actually better. Though it would be preferable if they’d have bundled some useful stuff with practical application along with it.

It’s still not proven, though, that ‘better’ information design is actually better in the real world. If a school were to implement the winning solution of the dashboard design competition, would performance improve, or would the niceties of visualisation just be swamped by intuition or ability to undertake complex reasoning? Take cycle plots — by providing a focus on an historic trend, it forces the person seeing the world to see it through that lens of that graph, perhaps blinding them to other, valid, interpretations.

In my experience, his has happened with KPIs, where people manage the indicator rather than the thing it is meant to be indicative of (using the sickness KPI to rewrite sickness policies rather than tackle overworking, morale, etc.).

By Kris Erickson. October 29th, 2013 at 10:31 am

It’s probably far easier to make something ‘look interesting’ than to make it actually ‘be interesting.’

By Klaas Vandenberghe. October 30th, 2013 at 4:17 am

Stephen

While I agree with your review, I wonder about this line in one of your comments: “Getting the message to stick is the goal, and nothing that doesn’t feature the message itself can make that happen.”
Isn’t that in contradiction with all mnemonic techniques? Memorization champions like Wang Feng, Dominic O’Brien,… almost always add useless information, be it in picture or in words: they add a fantasy scene or a story to remember numbers or other facts, and when the facts need to be reproduced, they remove the junk again.
Isn’t it possible that a meaningless, flashy ‘key’ could trigger our memory to remember the meaningful?

By Stephen Few. October 30th, 2013 at 9:56 am

Klaas,

Your question reminds me of an experience that I had when I was young. I produced a radio show on a large station in Orange County, California. I had arranged for a renowned memory expert and former pro basketball player named Jerry Lucas to be our guest, but when the time arrived for the show to begin, he wasn’t there. Why? Because he forgot. Although he had great mnemonic tricks for remembering people’s names and other facts, he either didn’t have any tricks for remembering events or simply failed to apply them.

If something that isn’t data could be added to a chart that would help the message of the chart to be remembered, it would not in my opinion be chart junk. Anything that is useful as a means of informing and serving any other goals of a chart, such as motivating people to care and take action, is not chart junk. If the visual component that’s added is itself remembered, but it doesn’t help us understand and remember the chart’s message, then its memorability isn’t useful, so it qualifies as junk.

The techniques of memory experts, such as those that you mentioned, take a great deal of work to master. They associate mnemonic devices that work for them to information to help them remember it. This does not mean that when we design charts we could associate a visual mnemonic device with each of the facts in the chart to make the its message memorable. Mnemonic devices that work for us because of practice wouldn’t work for others. People might remember the silly images, but would not associate them with the facts. Also, of course, a chart filled with silly images wouldn’t be readable as a chart. And finally, could you imagine the time that it would take to create charts in this way?

By Jennifer Teves. November 21st, 2013 at 3:11 pm

I could not agree more with this statement!
“When an image appears before your eyes for only a second and then disappears, what actually goes on in your brain perceptually and cognitively? When the image is a visualization, you don’t have time to even begin making sense of it.”

By Thomas Considine. December 3rd, 2013 at 5:26 pm

I agree that 1-second displays are not the same as letting someone view the visualization. I think Borkin’s study confounds memorability and regnizability, and then goes off headlong inferring about one from her test of the other. Really, remembering information and recognizing a flashy image are two distinct types of cognitive process.

Also, I’m pretty sure the the “mnemonist” techniques are not a form of excelling at something we all do, but rather are used by people with an unusual ability (often synesthetic) which allows them to recall a series of items by the extraordinary technique of laying them across a visual space or a story. I think I remember that Luria’s “M” once explained an omission by saying he had accidentally placed it in the shadow of a larger object, and therefore missed it when he returned!

Both of these performances - recognizing slides after a brief presentation, and hanging serial recall on top of a different sense - are more on the order of “tricks” rather than offering a glimpse at normal cognition.

By btown. January 27th, 2014 at 11:17 am

I agree that this study cannot be taken seriously. By their logic, the most “memorable” data visualizations would presumably all contain images of sex and violence.