VisWeek – What Constitutes the Best Research?
I’ve often complained that much information visualization research is poorly designed, produces visualizations that don’t work in the real world, or is squandered on things that don’t matter. To some degree this sad state of affairs is encouraged by the wayward values of many in the community. A glaring example of this is the “InfoVis Best Paper” award that was given at this year’s conference (and many in past years as well). Despite the obvious technical talent that went into developing the “Context-Preserving Visual Links” that won this year’s award, these visual links are almost entirely useless for practical purposes.
Context-preserving visual links are lines that connect items in a visualization or set of related visualizations to highlight those items and thus make them easier to find, and do so in a way that minimally occludes other information. There are many ways that items can be highlighted. The best methods apply visual attributes to those items that we perceive preattentively, causing them to pop out in the display. This approach highlights items without adding meaningless visual content to the display. As you can see in the following example of context-preserving visual links, items are highlighted by the addition of lines to connect them.
The lines are the most salient objects in the display, yet they mean nothing. Drawing someone’s attention to visual content that is meaningless undermines the effectiveness of a visualization. They direct attention where it isn’t needed. Even worse in this case, they suggest meanings that don’t actually exist by forming paths, which usually suggest meaningful routes through data. Also, lines that connect items suggest that those items are somehow related, but this is not always the case.
My intention here is not to devalue the talents of these researchers, and certainly not to discourage them, but to bemoan the fact their obvious talents were misdirected. What a shame. Why did no one recognize the dysfunctionality of the end result and warn them before all of this effort was…I won’t say wasted, because they certainly learned a great deal in the process, but rather “misapplied,” leading to a result that can’t be meaningfully applied to information visualization.
The fact that this work was given the “Best Paper” award indicates a fundamental problem in the information visualization research community: because so many in the community are focused on the creation of technology—a computer engineering task—they lose sight of the purpose of information visualization, which is to help people think more effectively about data, resulting in better understanding, better decisions, and ultimately a better world. Except in those rare instances when context-preserving visual links represent meaningful paths that connect items that are in fact related, they are useless. The fact that they connect items in a way that minimally occludes other items in the visualization is a significant technical achievement, but one that undermines use of the data.
Technical achievement should be rewarded, but not technical achievement alone. More important criteria for judging the merits of research are the degree to which it actually works and the degree to which it does something that actually matters. Information visualization research must be approached from this more holistic perspective. Those who direct students’ efforts should help them develop this perspective. Those who award prizes for work in the field should use them to motivate research that works and matters. Anything less is failure.
Take care,
8 Comments on “VisWeek – What Constitutes the Best Research?”
amen. My opinion has always been that there are 2 problems being solved here which may require different skill sets. The first is a communication problem, the second an engineering problem. You can’t solve the latter (or even know what it is) until you solve the first.
We (the authors of Context-Preserving Visual Links) would like to respond. (Note: The authors actually posted a link to their site, but to keep the discussion here in this blog, I’ve provided the text of their response below.)
Context-Preserving Visual Links
Our paper “Context Preserving Visual Links” has recently received the best paper award at the IEEE InfoVis conference, of which we are very proud. Such an award comes with a great deal of visibility for our work, and naturally also criticism. While most of it was very positive, there are also those who think visual links are a bad idea.
In particular, a fairly popular blog by Stephen Few (here), has taken our paper as an
example of how the field of information visualization has a “fundamental problem”. While we leave the general comments on the state of the community for others to discuss (e.g., this article), we would like to respond to the issues raised concerning our paper.
But first, let me briefly introduce visual links: The idea of visual link is to connect related items of information using curves or shapes. This differentiates them from traditional highlights, which are more subtle. This idea has been fairly popular in the InfoVis community (e.g., the original VisLinks paper, or Bubble Sets by Collins et. al, or this year’s LineSets by Alper et. al.).
Stephen Few states that “the best methods [for highlighting] apply visual attributes […] that we perceive preattentively, causing them to pop out in the display.”
Yes, we totally agree. And if you are in a situation, where you can utilize pre-attentive features they are certainly the best choice. But those situations are rare. (If you want to brush-up your knowledge on perception in visualization and pre-attentive features we highly recommend this site). Pre-attentive features don’t work in conjunction, and if you use for example any color in your visualization, you are very likely to run into situations where conjunction is a problem, when you use color to highlight.
Stepen Few continues by writing:
“The lines are the most salient objects in the display, yet they mean nothing. Drawing someones attention to visual content that is meaningless undermines the effectiveness of a visualization. They direct attention where it isn’t needed. Even worse in this case, they suggest meanings that don’t actually exist by forming paths, which usually suggest meaningful routes through data. Also, lines that connect items suggest that those items are somehow related, but this is not always the case.”
And this is where Mr. Few didn’t get our intentions, and didn’t see the real use case (which is clearly described in the paper). Visual Links always should be interactively activated, and of course, they should connect only items that are related (what would be the point of connecting unrelated items?).
You might find examples in the paper where highlighted items seem to be unrelated, but this is because this paper deals _only_ with the intelligent routing, and assumes meaningful connections. We’ve previously published an article at GI (which by the way also won a best paper award), where we show how visual links can be interactively invoked between applications, and of course we link only related information. You can check out the video for this paper here.
We also discuss in the current paper, that visual links can be problematic when the path can suggest meaningful routes, as would be the case when using the visual links on top of maps. That’s simply a limitation which you have to accept if you want to use visual links.
Mr Few. continues:
“Except in those rare instances when context-preserving visual links represent meaningful paths that connect items that are in fact related, they are useless.”
Links between unrelated items are useless, that’s no surprise. Links between related items however, have many benefits. We have proven, that they perform better than highlighting in real-world situations. They also have the potential to guide users to content out of their current field of view.
But probably it’s best if you judge for yourself, by looking here
The Context-Preserving Visual Links authors.
Hi,
Mr Few replaced our link with a paste of the contents here, which is ok for us. However, we still would like to make the link to the original available to readers of this blog: http://alexlexvisual.blogspot.com/2011/10/context-preserving-visual-links.html
Asides from stating the source, the original contains several links which were lost in the paste here.
The Context-Preserving Visual Links authors.
Hello Context-Preserving Visual Links Guys,
I appreciate your response. Before addressing your points, I want to say again that the purpose of my comments was not to belittle your work, which exhibited fine technical achievement, but to raise concerns about the priorities of the infovis research community in general, which is less concerned with perceptual effectiveness and application in the real world than I and many others in the community believe it ought to be.
You agree that preattentive visual attributes alone would highlight items more effectively, but argue that they cannot be used in the cases that you address. That is not the case. For instance, in the example from your work that I featured in this blog, imagine that the lines are erased and only your color-coded enclosures around the items remain. Or, imagine that everything except the items that you wish to highlight are dimmed enough to cause the selected items to pop out.
The fact that visual links should only be activated interactively does not eliminate the problems. Even if they only appear when they’re turned on, and can then be turned off when they’re no longer needed, they still highlight items through the use of links that add meaningless non-data ink to the display, and that non-data ink stands out as the most salient items in the display, which is a powerful and unnecessary distraction.
Even if all the highlighted items are somehow related, which wasn’t apparent to me from the examples that you provided, the links still aren’t necessary. The technique of brushing and linking is used to highlight items in multiple coordinated views in a way that allows them to stand out without losing the context of the whole to which they belong. This appears to share a common purpose with your examples of visual links. Should a visualization tool such as Tableau use your link lines rather than the dimming of non-selected content to highlight items that are selected? The methods that Tableau and other tools that support brushing and linking use manage to highlight items and show their connections without the use of unnecessary and highly salient link lines.
What I’m saying is that I find it hard to imagine many cases, if any, when your link lines would provide the best solution for highlighting items. Perhaps there are other use cases, however, when your method for finding a non-obstructive path would more usefully apply. That might be worth exploring.
Apparently, the questions that I’ve raised in this blog regarding the priorities of the information visualization research community as reflected in the awards process have led to a great deal of discussion in the community. Is the process broken and in need of repair? I hope this discussion is useful.
As someone who works closely with data visualization practitioners out there in the world, I like to expose them to fine work that is generated by the research community. I also like to encourage commercial software vendors in the infovis space to attend VisWeek to learn of innovations that can be incorporated into their products. When we select a best paper, we’re saying that this is the best that information visualization research has produced this year. Have we selected the research paper that represents the best that we have to offer? Have we selected an example that illustrates the kind of work that we believe is most valuable and should serve as a model for future work? From my perspective, in these respects we’ve failed.
Much good work has been presented this year. In another blog article later today, I’ll showcase one that I think is worthy of an award, but didn’t even receive honorable mention.
It seems to me that the result of this research is a somewhat programmatically interesting anti-collision algorithm.
I have to share Stephen’s concerns about the application of this as presented, and especially about
1) the way that the lines appear to denote meaning by their connections when they do not, and
2) the way that the lines become the most visually important aspect on the page
In looking through the image gallery, these concerns were exacerbated rather than alleviated.
In the vast majority of the images presented, the lines offer absolutely nothing to help explain the relation between the linked items, and add a huge distraction.
For example: https://picasaweb.google.com/111174118588927995482/Visual_links?authkey=Gv1sRgCLLGwr2c1NDcag#5668279639164288962
This image is clearly an improvement on the previous, where the lines cut across the text.
But what benefit can I possibly get from seeing lines linking these highlighted terms??
As I follow a line from one word to another, or from one word to another line, I am simply left confused as to why the lines are there.
Examples like this: https://picasaweb.google.com/111174118588927995482/Visual_links?authkey=Gv1sRgCLLGwr2c1NDcag#5668281912071133586
I would consider outright dangerous.
I can’t help but agree that the work is technically interesting and robust but of little use and potentially much harm in practical application – at least in terms of how the uses are presented here.
While the solution is rather cute, I am worried that the user study in the paper is a little biased. Authors asked participants to count the number of highlighted items by showing them static images with links overlaid on top. If we are just interested in a count, wouldn’t showing the total count somewhere in the corner answer this question better? Why is counting things so much more important than making sense of the data? Is counting a valid proxy to understanding the relationships between data items from various screens? Maybe, authors should have come up with a different task to justify the usefulness of the display.
Another concern is related to a number of items the authors chose to highlight: authors selected images with 5-12 highlighted items only. Just searching for the word “paper” on this page turns up 16 matches, “vis” is used 62 times, and the ubiquitous “the” occurs more than 200 times. Would anyone be able to accurately count so many items with either of the highlighting techniques? Maybe authors could comment on the scalability and limitations of their method.
Going through the images selected for the user study, I am increasingly concerned about the size of the highlighting boxes the authors compare against. Wouldn’t authors agree that making the regular highlighting box slightly bigger, i.e. as salient as the links, improve user’s ability to find the objects on a page? It is a standard practice with the mouse pointer – it is hard to find on a screenshot, so in videos and screenshots it is highlighted by a big yellow circle (http://www.techsmith.com/camtasia-features.html – Cursor Effects).
Lastly, I would be careful calling photos 10-12 a success (https://picasaweb.google.com/111174118588927995482/Visual_links?authkey=Gv1sRgCLLGwr2c1NDcag#5668281719625342770): a KEGG pathway appears to have another network layered on top of it suggesting that metabolites are related in some fashion rather than just highlighting them. I would imagine that networks in general are not a good application for the context links in their current implementation.
Stephen, your points are excellent & timely. It strikes me that people are developing expensive, over-engineered tools and then looking for applications. A bit of historical knowledge would help: the biggest advances in visualisation have been through people with ideas looking for an efficient way to communicate them – Descartes, Playfair, Snow, Nightingale, for instance.
Given that technical ‘solutions’ are now so complex, perhaps it’s no longer possible for an individual to solve his own problem. But all these means is that the technical people must stay close to the problem by working with the problem-owner/s. Fit-for-purpose products will be the result.