Feature Lists Make Us Comfortable, but Sometimes Make Us Dumb
A few days ago I noticed a blog posted by Boris Evelson of Forrester Research titled “How to Differentiate Advanced Data Visualization Solutions.” Forrester is one of the leading IT research and advice companies. Along with its larger rival, Gartner, these companies serve as trusted advisers to thousands of organizations, helping them make decisions about all aspects of information technology. Although it’s convenient for Chief Information Officers (CIOs) to subscribe to a single service for all the advice they need, is this approach reliable? It depends on whether we actually get advice from someone who has the expertise we’re missing. Far too often when relying on these services, however, we get advice from people whose range of topics is too broad to manage knowledgeably. We sometimes find ourselves being advised by someone who understands less about the topic than we do. If you’re looking for advice about data visualization products, based on what I read in Forrester’s blog, I suggest that you look elsewhere.
Evelson provided a list of features that he believes we should look for when shopping for an advanced data visualization solution. Unfortunately, his list looks as if it was constructed by visiting the websites of several vendors that claim to offer data visualization solutions and then collating the features that they offer. I expect more from a service that people pay good money to for advice. We can’t trust most vendors that sell data visualization software to tell us what we should expect from a good product. It is in their interest to promote the features that they offer, and only those features, whether they’re worthwhile or not. In fact, most vendors that offer so-called data visualization solutions know little about data visualization.
Another problem with Evelson’s advice is that it isn’t clear what he means by “advanced data visualization solutions.” What distinguishes advanced solutions from the others? Of the few features on his list that actually characterize an effective data visualization solution (most of his list misses the mark, as I’ll show in a moment), none go beyond the basic functionality that should exist in every data visualization solution, not just those that are “advanced.”
Evelson has offered the kind of analysis and advice that we get from people who dabble in data visualization, rather than those who have taken the time to develop, not just shallow talking points, but an understanding of what’s really needed and what really works.
Let’s take a look at each feature on Evelson’s list in the order presented and evaluate it’s worth.
Feature #1: “If it’s a thin client does it have Web2.0 RIA (Rich Internet Application) functionality (Flash, Flex, Silverlight, etc)?”
Response: This is a feature that only an IT guy with myopia could appreciate, not someone who actually analyzes and presents data. When evaluating software, we care about functionality and usability, not about the specific technology that delivers it. If we’re exploring and analyzing data via the Web, what matters is that interactions are smooth, efficient, easy, and seamless. How this is accomplished technically doesn’t matter.
Feature #2: “In addition to standard bar, column, line, pie charts, etc how many other chart types does the vendor offer? Some advanced examples include heat maps, bubble charts, funnel graphs, histograms, pareto chats, spider / radar diagrams, and others?”
Response: So it’s the number of chart types that matters? What constitutes a chart type? Do useless chart types count? This is a lot like giving high marks to the software programs with the most lines of programming code, as if that were a measure of quality and usefulness. What matters is that a data visualization solution supports the types of charts that do what we need and that they work really well. Many data visualization products could be dramatically improved by removing many of the silly charts that they offer rather than by adding more to the collection.
Feature #3: “Can the data be visualized via gadgets/widgets like temperature gauges, clocks, meters, street lights, etc?”
Response: Is Evelson serious? Should vendors get points for providing silly, dysfunctional display gadgets? Most of the gauges, clocks, meters, and street lights that many so-called data visualization products provide are worthless. Anyone who understands data visualization knows this to be true. This is what Evelson looks for in “advanced” data visualization solutions?
Feature #4: “Can you mash up your data with geospatial data and perform analysis based on visualisation of maps, routes, architectural layouts, etc?”
Response: While the ability to view and interact with data geo-spatially is critical, most of the “mash-ups” that vendors enable are horribly designed, and thus of little use. Throwing quantitative data onto a Google map doesn’t qualify as effective data visualization. Google maps (and other similar services) were not designed as platforms for quantitative display, but instead as sources for directions (“How do I get from here to there?”). Good geo-spatial data visualization uses maps that are designed to feature quantitative data only within the context of geo-spatial information that adds meaning to the data. What’s also important is that geo-spatial displays can be combined on the screen simultaneously with other forms of data visualization (for example, bar graphs, line graphs, tables, and so on) to provide a fuller view of the data than geography alone.
Feature #5: “Can you have multiple dynamically linked visualization panels? It’s close to impossible to analyze more than 3 dimensions (xyz) on a single panel. So when you need to analyze >3 dimensions you need multiple panels, each with 1-3 dimensions, all dynamically linked so that you can see how changing one affects another.”
Response: This is probably the clearest description on Evelson’s list of a feature that is actually useful and indeed critical. Whether the separate views of the data set appear in separate panels or not isn’t important however. What’s important is the ability visualize the data in multiple ways–that is, from multiple perspectives on the screen at once. Only then can we construct a comprehensive view and spot relationships, which would be impossible if we were forced to examine each view independently, one at a time.
Feature #6: “Animations. Clicking through 100s of time periods to perform time series analysis may be impractical. So can you animate/automate that time period journey / analysis?”
Response: So far, researchers have only found a limited role for animation in data visualization, especially for data analysis. When Hans Rosling of GapMinder uses bubble plots to tell a story, such as the correlation between literacy and fertility throughout the world and how it has changed through time, bubbles (one per country) that move to display change through time work because he is narrating–telling us where to look and what it means. Research has shown, however, that these same animated bubble plots are of limited use for data analysis. We simply cannot watch all those bubbles as they follow their independent trajectories through the plot. To compare the paths that two bubbles have taken through time by means of animation, we must mark the path with trails that provide static representations of the bubbles’ journeys. Too many software vendors are providing animations that are nothing more than cute tricks to entertain, rather than useful visualizations. We should run from any vendor that has actually taken the time to make the pointers on their silly gas gauges wobble back and forth for several seconds until they eventually stop moving and point to the value that we need.
Feature #7:Â “3 dimensional charts. Can you have a 3rd dimension, such as a size of a bubble on an XY axis?”
Response: Simply asking a vendor if his products support 3-D displays is the wrong question. 3-D pie charts, bar graphs, and line graphs are almost never useful. Most implementations of 3D in so-called data visualization products are either entirely gratuitous and thus distracting, or far too difficult to read. The example that Evelson gave, however–the ability to add a third quantitative variable to a scatterplot by allowing the data points to vary in size to represent a third quantitative variable–is actually useful, assuming the vendor designs this feature properly. That’s a big assumption.
Feature #8:Â “Can you have microcharts (aka trellis) — a two dimensional chart embedded in each row or cell on a grid?”
Response: Evelson is onto something here, but he seems a bit confused about the terms. “Microcharts” is the name of an Excel add-in product from Bonavista Systems. A microchart is a small chart, such as a sparkline or a bullet graph, which conveys rich information in a small amount of space, such as in a single spreadsheet cell. A “trellis” display, what Edward Tufte has been calling “small multiples” for many years, is something quite different. It is a series of charts that breaks a data set into logical subsets, each with the same quantitative scale, arranged within eye span on a single screen or page, for the purpose of making comparisons between the charts. For example, if the correlation between the number of sales contacts and sales revenues for 500 customers and 20 separate products would be too cluttered and complex if displayed in a single scatterplot, we might be able to solve this problem by creating a trellis display of 20 scatterplots, one per product.
Feature #9: “Can you do contextual or gestural (not instrumented, not pushing buttons, or clicking on tabs) manipulation of visualization objects, as in video games or iPhone like interface?”
Response: Evelson might be getting at something useful here, but he hasn’t distinguished the gratuitous video game-like interactions that have become all too common in many so-called data visualization products from useful interactions that are needed to uncover meanings that live in our data, which only a few products actually support. For data exploration and analysis, it’s quite useful to interact with visualizations of data directly to change the nature of the display in pursuit of meaning, such as to sort or filter data. For instance, rather than using a separate control or dialog box to remove outliers in a scatterplot, it’s useful to be able to grab them with the mouse (or with your finger on a touch screen) and simple throw them away.
Feature #10: “Is the data that is being analyzed
a) Pulled on demand from source applications?
b) Stored in an intermediary DBMS
c) Stored in memory? This last one has a distinct advantage of being much more flexible. For example, you can instantaneously reuse element as a fact or a dimension, or you can build aggregates or hierarchies on the fly.”
Response: What really matters is not where the information is stored, but how easily, flexibly, and rapidly we can access and interact with the data that we need. How this is accomplished technically needn’t concern us as long as it works.
Feature #11: “Is there a BAM-like operational monitoring functionality where data can be fed into the visualization in real time?”
Response: When real-time data updates are needed, this is a useful feature, but few data visualization solutions require real-time updates.
Feature #12: “In addition to historical analysis, does visualization incorporate predictive analytics components?”
Response: This is indeed useful, but what many vendors call “predictive analytics” are neither predictive nor analytical. Rather than simply asking vendors if they support predictive analytics (you will never get a “No” answer to this question), we should questions such as: “Can the software be used to build effective predictive models (that is, those that are statistically robust) that allow us to not only determine the probability of particular results under particular conditions, but also to see, understand, and therefore reason about the interactions between variables that contribute to that result?”
Feature #13: “Portal integration. If you have to deliver these visualizations via a portal (SharePoint, etc) do these tools have out of the box portal integration or do you need to customize.”
Response: Generic portal integration isn’t important. If you use a particular portal product and you need the analytics tools to integrate with it, then this specific requirement might be useful to you. This should not, however, be a reason to reject an otherwise effective data visualization solution. There are so few good solutions to choose from today, don’t let someone in your IT department turn away the one that’s useful to you because it doesn’t integrate neatly into your organization’s portal.
At the end of his list of features, Evelson asked, “What did I miss?” I appreciate his openness to suggestions. More than what he missed, however, I’m concerned about the features that he included that are either unimportant or that in some cases actually undermine data visualization.
Fundamentally, Evelson missed the opportunity to assess the effectiveness of data visualization solutions. Lists of features–even good ones–fail to do this. Another fundamental problem is that his list lumps all data visualization solutions together, as if every purpose for which data visualization might be used requires the same functionality. This is far from the truth. Uses of visualization for monitoring, analysis, or communication, although they share much in common, require many distinct features as well. When shopping for data visualization software, you must first know what you plan to accomplish with it and then determine the features that are specifically required for that purpose. Unless you’re planning to use a single tool for all purposes, you won’t need everything that a data visualization solution could possibly offer.
Evelson is but one of many people that organizations erroneously trust for critical advice. Regarding data visualization, he lacks the expertise that’s required and legitimately expected. Anyone who sets himself up as an adviser—especially one that organizations pay for dearly—ought to develop deep expertise in the subject matter. Before we can shop effectively for technology, we must first shop effectively for reliable sources of advice.
Take care,
34 Comments on “Feature Lists Make Us Comfortable, but Sometimes Make Us Dumb”
Wow! Yeah, I’d characterize that list as a miss too. It seems clear from his CV on Forrester that he’s experience lies in the BI Tech and process world and not so much in the arena of visualisation. If sheer number of chart types were an important measure wouldn’t Excel win? ;-)
Clint,
I suspect, given the fact that Evelson wrote this blog, that Forrester Research doesn’t have anyone who knows more about data visualization than Evelson does. Given the importance of business intelligence (BI) and the centrality of data visualization to BI effectiveness, this is a sad state of affairs. Although I don’t know for sure, I doubt that Gartner has anyone who understand data visualization any better than Evelson. If these large IT research and advice organizations, given their influence, would recruit and develop expertise in data visualization, perhaps the tipping point that separates today’s paucity of good data visualization tools and practices from the culture of analytics we hope for and desperately need would be reached much sooner.
Stephen,
Agreed. It seems to me that most formally trained visualists (?good term?) are either in Academia or at places like IBM and NYTimes. So doesn’t the question become how can they be enticed into a place like Forrester (or stepping down from that, a BI team in a business)? My guess would be that many analysts who strive for solid visualisation in their work as part of a BI or other analytics teams are self-taught enthusiasts like me – rather than trained specialists.
Although you might argue that it’s an easier step for a well-trained enthusiast to make (into an industry Analyst role) than for someone who is formally trained and specialized in information visualisation. This presupposes that Forrester or Gartner or whichever recognizes the need for such expertise.
Steve
I have a lot of respect for your work and often refer my clients to you when I think they require deeper dives into data visualization capabilities and best practices that I can provide.
The point that you seem to be missing is that unlike smaller organizations and individual analysts, that use blogs as the main venue for publishing their research, Forrester’s main body of research is syndicated and available on subscription only basis. Forrester analysts use blogs mainly to stimulate discussions, solicit feedback and link to our main body of reasearch. After all, it’s business.
My other point is that you are data visualization deep subject matter expert, while I am a BI generalist. Data visualizatin is but one of dozen features in an enterprise BI implementation. Rather then criticizing my comments, I respectfully suggest that you take more of a “fill in the blanks” approach, and concentrate on adding value to our resepective clients. I would hope that our respective organizations can make this a win-win scenario and collaborate. FYI, it is an a strictly enforced policy at Forrester Research never to criticize our peers and competitors. It’s a loose-loose.
I sincerely hope in the future we can engage in more constructive dialogue based on mutual respect and individual strengths.
Best Regards
Boris Evelson
Principal Analyst, Business Intelligence
Forreste Research
Boris,
I appreciate your response, but find it lacking, both in reason and responsibility. Are you seriously suggesting that Forrester doesn’t need to give good advice in its blogs because, after all, people aren’t paying for them? Regardless of your intentions when writing a blog, you represent an organization that is a trusted adviser, a repository of “expertise.†You list “data visualization†as one of your areas of research coverage. If you had given useful advice, I could have responded in the “fill in the blanks†manner that you would have preferred, but your advice wasn’t merely lacking in thoroughness, to which I could have appended additional information, it was downright misleading. As a result, I was left with only one viable response: I had to take out my red pen and strike through much of what you wrote.
Forrester Research and all those who set themselves up as advisers are responsible for delivering what they advertise: trustworthy and expert information. Your list of advanced data visualization features was at best free marketing for software vendors. Unfortunately, the people who read your list don’t understand it as such. They expect it to be the result of objective research, built on a foundation of expertise. If you lack expertise in data visualization, you shouldn’t include it in your list of research coverage, and you shouldn’t write about it in your blog. That is, unless you preface your comments with a statement to clarify your lack of expertise on the topic, so readers know to take it for what it is.
Does anyone at Forrester Research have expertise in data visualization? Four people at Forrester, including you, are listed as working in this area. Research papers written by Forrester on the topic range in price from $199 to $499. Would a customer in need of advice get his money’s worth if he paid for one of these? If Forrester lacks expertise in data visualization, this field shouldn’t appear on the list. Organizations and individuals should only claim expertise that they possess. To do otherwise is false advertising.
To offer bad advice is to do harm to your readers and customers. It is my job to provide useful information and to expose misinformation. As a person on whom people rely for expert advice, it is your job now to either correct what you’ve written in your blog or to defend it. If the purpose of your blog is to encourage useful discussion, I have provided you with an opportunity.
Steve
Steve,
Very interesting post. Here are minor comments on terminology. Bill Cleveland introduced trellis displays. A quote from his 1996 paper in the Journal of Computational and Graphical Statistics follows:
“The salient visual aspect of Trellis display is a three-way rectangular array of panels with columns, rows, and pages.” Note that a trellis display does not need to be on a single page. In your example, you would probably need more pages if you had many more than 20 products.
I have heard vendors other than Bonavista use the term microchart to refer to any sparkline or other small chart.
Regards,
Naomi
Steve
Visualization is just one small – important, but small – part of an overall complex business intelligence enterprise application. Forrester defines business intelligence as a journey that raw data takes from being in a meaningless state (bits and bytes) to becoming usful, actionable and insightful information. That journey involves multiple architectural components and process steps (I typically cite over 30), complex governance and organizatonal structures.
In evaluating these business intelligence solutions Forrester publishes at least several “Waves” (ETL, DW and BI, are just some most relevant ones) that cover these 30+ architectural components at a certain level of detail.
For example, my latest BI Wave has over 150 criteria covering 12 leading BI vendors. It contains 1800(150×12) Excel cells that are filled out with the description of each product capability in a particular area, another 150 cells with very detailed scale/score explanation, plus another 1800 cells with the actual scores, based on evaluation of product capability rated against the scale. This process is very transparent (clients and vendor can see every single cell) and vendors push back tremendously on just about every single cell. It’s a process that takes several months to complete.
If you do not believe that the results of such an evaluation, and analysis, and advice to our clients based on such results, are not usefull, well, then I challenge you to find any similar, transparent, detailed evaluation of BI solutions, by any other analyst firm, including your own.
I spent over 25+ years in the BI market as a hands on practitioner and management consultant. I built over 100 these BI vendor evaluations for my former employers and clients. I know that these prototypes and evaluations cost tens of thousands of dollars. If Forrester can deliver them to our clients for $499, and you think that is not good advice, and advice not worth the money, well, with all due respect, I think you are seriously misled about what typical IT clients challenges, pain points and requirements are.
Visualization capabilities of these BI vendors take about 3 or 4 out of 150 criteria. Regardless of what you say, I strongly believe that high level summarization I published in my blog is the right level of detail for evaluating visualization capabilities of larger BI solutions. I hope you realize (and advise your clients accordingly) that no data visualisation solution can be usefull unless the data, metrics and processes that feed it are properly constructed, implemented, managed and governed.
I respectfully disagree with your assessment and your points of view. I also respectfully suggest that you actually read some of our Waves (I’ll be more than happy to send you a complementary copy)
before passing judgement on whether it is usefull advice. I actually red your book before I passed judgement (positive) on your work.
So, I respectfully suggest that we only continue this dialogue after you read content of every single cell in my BI tool evaluations (with same level of attention I gave the content of your book), offer any other venue that a potential BI client can turn to to find similar analysis and advice.
Boris
Boris,
You wrote 527 words in your response, but completely failed to address my points. We’re talking about “data visualization.†That’s what your blog post was about. The fact that there are other aspects of BI is not in question. Whether Forrester provides expert analysis and advice about these other aspects of BI is not of interest to me. Your only response to my case against your list of data visualization features was, “Regardless of what you say, I strongly believe that high level summarization I published in my blog is the right level of detail for evaluating visualization capabilities of larger BI solutions.†What does this mean? The “right level of detailâ€? I haven’t questioned the level of information you’ve provided; I’ve questioned its accuracy and usefulness. Also, you did not present your features’ list as a means to evaluate the “visualization capabilities of larger BI solutions,†as you now claim. You presented it as a means to evaluate the capabilities of “advanced data visualization solutions.â€
You wrote, “I respectfully disagree with your assessment and your points of view,†yet you haven’t actually responded to any of my points. Your entire response was written to counter an argument that I haven’t made. Whether Forrester in general or you in particular produce worthwhile research and advice on other aspects of IT falls outside of the case that I’ve made. We’re talking about “data visualization.†The only judgment that I’ve made is that your list of features for evaluating the worth of data visualization solutions is “bad advice†and that it indicates a lack of expertise in the area. I can at best only speculate that other areas of research and advice done by Forrester are similarly of poor quality and therefore a bad investment. Regarding Forrester and you, I have made a clear argument that organizations or individuals that set themselves up as expert advisers should not claim expertise that they lack and certainly shouldn’t charge money for it. Regarding your offer to send me review copies of reports that Forrester has produced, if you’ve produced anything regarding data visualization, I’d love to see it. I will gladly review and respond to it in my blog. If it’s worthwhile, I’ll say so without reservation.
Whether you are a good researcher and analyst in other areas and in general is something I haven’t addressed. I’m not out to get you; I’m trying to provide people with useful information about data visualization, which includes warning them away from bad advice, which you’ve given. However, your response to my critique exhibits several fundamental thinking errors. You’re not being a good analyst when you write 527 words to counter arguments that I haven’t made, while completely ignoring the arguments that I have made. I have no doubt that your intentions are good. The soundness of your research, analysis, and advice is what I question. Business intelligence cannot be achieved without clear and rational thinking.
Steve
Naomi,
Thanks for the useful additions to this discussion. I’m familiar with the term “trellis” chart and its early use by William Cleveland. I used the term “small multiples” in its place because it is a term that is familiar to more people, due to the popularity of Tufte’s work. Regarding “Microcharts,” it is true that this name for one of Bonavista Systems’ products has since been picked up by others and used as a generic term for small charts, especially those that fit into the cells of a spreadsheet. My point regarding “Microcharts” in response to Evelson’s blog, of course, was to clarify that microcharts are not the same as trellis charts, small multiples, or visual crosstabs, as he suggested.
Steve
“Forrester is one of the leading IT research and advice companies.” That statement alone makes me cringe for many of the reasons noted…Forrester now has a habit of buying has-been thought leaders to lead “fresh out of school with no business experience” researchers. And worst of all, their data is dirty…I’ve seen how they distribute their surveys and who answers them. They do nothing to validate their findings (anyone can ‘say’ anything). And they refuse to include or listen to consultants who can speak about the things they’ve witnessed firsthand inside of organizations.
Indeed, they don’t even run their own company by the principles they espouse to others.
Steve
If you really had meant what you said – about providing valuable advice to our clients – your blog would not have contained offensive terms like “dumb”, “shallow”, “myopia”, “silly”, “dysfunctional”, “confused”, “worthless”, “organizations erroneously trust”. That’s 8 valuable real estate plots in your blog where you could’ve provided additional helpful advice to our mutual client audience.
If you want to keep the dialogue going, sure, but let’s keep it professional, not offensive, and concentrate 100% on adding value to our clients. Let’s leave criticism of our peers and colleagues to trashy blog sites that true professionals stay away from.
I once again (for the third time) reach out to you and offer to respect each other’s work and make it a win-win scenario. When you feel that visualization is just a tip of the iceberg of your clients BI issues and challenges, send them to me. When I feel that a client has most of their BI environment in order, but lacks deep visualization expertise, I’ll send them to you.
Boris
Boris,
The terms that I used–“dumb,” “shallow,” etc.–were clear, accurate, and intentional. There is nothing unprofessional about stating the truth in clear terms. When something is dumb, our clients expect us to call it what it is.
You have yet to respond to any of my points, except emotionally, never factually. Do you object to the substance of what I’ve said? If so, then make your case. Explain how something that I called dumb is in fact not dumb. What the people who rely on us appreciate is useful information. What you provided in your blog was not useful–in fact, it was misleading and harmful. If people relied on your advice, they would make “dumb” choices. If you disagree, then stop hiding behind the excuse that I have offended your sense of professionalism and make your case.
While focusing on the provocative nature of my word choices, you are missing how much more provocative your advice was to someone like me who understands and cares about data visualization. To give bad advice is much more provocative than calling something dumb. Giving bad advice is what qualifies as unprofessional.
Steve
My goodness, Boris, get out of the shark pool. A few points:
1) I don’t think Stephen Few has an “organization”, whom Forrester can win-win-win with, so much as a “mission” which is tougher to negotiate with.
2) Your argument is maddeningly imprecise. For instance, let’s consider feature 7: “3 dimensional charts. Can you have a 3rd dimension, such as a size of a bubble on an XY axis”. You seem to confuse 3D charts which most frequently display 2 dimensions of data (poorly), with means of displaying multiple dimensions at once. Wasteful 3D is a hot-button issue in this area.
3) The unexamined assumption of your piece seems to be that we should look for a monolithic provider of business intelligence. A more modern infrastructure for analytics (like Google’s) involves strong core technologies and data apis, but benefits from decentralization and lack of vendor lock-in.
4) Belittling the importance of data visualization on this blog doesn’t go very far. If you can’t communicate accurately in the “last mile” of data’s journey from measurement to the eye of a decision-maker, you’re doomed.
Chris
[Sidebar]
Chris, interesting note on “Last Mile”. What I’m wondering is while visualisation is (and always be) critical to the last mile, do you feel like it is growing in importance pre- last mile? In other words, visualisation becoming a more important tool in the tool belt for the analysis as well as presentation?
[/Sidebar]
This is a fun one to watch. What has become difficult for those of us who buy these sorts of tools is that our users read Forrester and Gartner and make us buy stuff that is recommended. When looking for specialty tools like “data visualization”, it’s wrong to include these capabilities in general BI tool evaluations. They simply do not suffice.
If you strip away all the name calling and other hyperbole, I think Stephen is on to something here. Until the big research vendors develop deep expertise in specialty products, I think they are doing their customers a disservice by rating things that they don’t truly understand.
Also, if it’s in a blog, it needs to be true. Otherwise, it’s just hurting your credibility. Boy, I miss Howard Dresner.
Hello Steve Krandel,
It’s been ages since our paths last crossed. I see that you’re still at Intuit. I hope that all is well with you.
I’m curious–what qualifies as “name calling and hyperbole”? I don’t believe I’ve done either.
Stephen,
Thanks for the straight-shooting – it’s disappointing how this often becomes misconstrued as a personal attack, when clearly it’s not. I tend to call a spade a spade, and appreciate it when other’s do likewise!
Boris,
I’d be interested in seeing your response to Stephen’s issues regarding your data visualation information advice. However, I’ve no interest in reading your responses to issues that were never raised.
This dialogue is reminiscent of a major impediment in converting data to information. Staying on course and understanding the real aims of information management often falls by the wayside in favour of the vagaries of personalities and vested interests.
Ryan
Having read through this discourse, I would dearly like to know whether Boris thinks that Stephen’s comments are valid, or feels that they are invalid. I am sure that he could offer counter arguments without breaching Forrester’s code of conduct (unless Forrester define criticism in its purest sense, that of commenting on someone or someone’s work) if he believes that he was absolutely correct. In all of his responses, all I see is Boris acting like a politician, evading the point, and trying to shift the discussion to a subject that he feels more comforbale with. Even if applied to “visualization capabilities of larger BI solutionsâ€, those might be the features frequently seen, they are not what clients should be considering as the most important IMO.
Stephen’s ‘objective’ comments are most welcome in what has become a tech and gadget focused BI industry. My clients range in size from 10 person shops to 10,000 person shops and what has become clear to me in 2 decades of visualization work is that there is no meaning without ‘context’.
The assessment of BI capabilities should always be performed in a meaningful context, of which the technical is an important part, but not the end result that clients are seeking. At the end of the day they are looking for cost effective solutions that help them analyse and understand their business.
We should all strive to be ambassadors of our industry, offering insight and clarity to our clients … not overwhelming them with technical detail and cosmetic distractions.
Thanks for the blog! Quite entertaining.
I doubt you’ll find many people who will disagree with you here Stephen – we’ve already self-selected somewhat by reading your blog. I certainly agree with your sentiments – when I worked for a software company I was always disappointed with the focus on technology/robustness/features that the analysts had rather than a focus on how well the software met the end-users’ needs.
Understanding needs requires so much more thought than comparing features. While these features are important, they should come after an assessment of the end-user experience.
Stephen and other Data Visualization Professionals
It is so tiresome to read about speeds and feeds – the techno-babble of research blogs from the over priced subscription services. The world of data visualization while small in the world of a BI generalist is one that has emerged with business users as a huge differentiator. The cultural adoption phase to VI from BI is a train that left the station 10 years ago but some failed to board it
Stephen,
I’ve been following your blog and work for sometime now and always apprecaite your spot-on observations, directness, and honesty, which is sometimes hard for business people to state let alone accept these days.
As a DW/BI practioner for many years, I’ve become increasingly frustrated with the lack of data visulation capabilities in the tools I was using. At the time, I didn’t realize that’s what to call it, but your work has provided a focus to what IS missing and what BI vendors aren’t addressing.
Also, having utlized both previously mentioned research offerrings, I can personally speak to the cursory analysis that is too often provided. I don’t know if this is a function of the analyst or of the organization directing the analyst to “review” knowledge areas that they have little direct experience in or with, but your criticisms of are spot-on.
I’ve definitely jumped on the VI train that Larry refers to and only wish I had found it earlier!
I learned today, thanks to thoughtful comments in his blog by Ronald Damhof, that my critique of Boris Evelson’s data visualization features list has generated a large number of comments in Twitter – many by “thought leaders” who support Evelson’s position. Because I don’t tweet, I was entirely unaware of these comments. (I don’t tweet because I find that it isn’t an appropriate venue for thoughtful discussion.) I haven’t taken the time to track down these comments to find out who’s saying what. I’m confident, however, that no one who understands data visualization has defended Evelson’s list. I suspect that those “thought leaders” who have rallied in support of Evelson and Forrester have not actually responded with substance to my critique, but are merely offended that anyone would dare speak out publicly against advice given by one of their own. I find it noteworthy, but not surprising, that those who find it offensive when someone presents an expert, rational critique of misinformation and does so openly, are often the same people who have no problem making flippant, disparaging personal remarks in Twitter. Is this an example of thought leadership?
If you disagree with my critique of Evelson’s list, please make your case openly and with substance. If all you can do is whisper disapproval from the shadows, you’re not contributing usefully to the business intelligence industry and you’re not living up to your role as a thought leader.
Steve,
I wrote a blog about this exchange at http://www.b-eye-network.com/blogs/raden/ and there were quite a few comments on Twitter, all largely positive. I suppose I’m in a different network because I don’t recall any being supportive of Boris. Having said that, I am supportive of Boris as an individual and a professional, but the analyst firm system is the real culprit. In addition to Boris, there are many fine and conscientious analysts in these firms, but they are subverted by the business plan under which they labor.
Serious question from a bit of an outsider (and certainly not a ‘thought leader’: In a perfect world, since Boris admits that Stephen has ‘deep-level expertise’ on the topic of Visual Business Intelligence, why would he not turn to him to add insight when writing his blog posting? I think Forrester would gain credibility if they looked to a larger pool of thinkers, particularly on highly specialized niche subjects.
In fairness to Boris, some of the criticisms of his posting by Stephen do come across as a bit personal in nature and could probably be toned down a bit. That being said, I agree that Boris is not really and has not directly addressed the points made by Stephen.
Dear Data Visualization,
Thanks for weighing in on this. Regarding the personal nature of my critique, I think it’s important to clarify what we mean when we say that comments are “personal in nature” and to distinguish the ways in which a critique should be personal and the ways it should not. Prior to a Google Alert bringing Evelson’s advice about advanced data visualization to my attention, although his name seemed vaguely familiar, I didn’t know him or his work. I was not seeking opportunities to criticize him, so in that sense my critique was not personal. On the other hand, we should all recognize that anytime someone states an opinion and gives advice publicly, especially as an “expert”, it is a personal matter. A person is trying to exercise influence over others, and in so doing, is personally responsible for the results. When harmful advice is given, a person is responsible for doing harm. This doesn’t mean that the person is bad, but it does mean that he gave bad advice and should take responsibility for doing all he can to correct it.
I feel strongly about this, because especially today when anyone can speak his mind electronically to a worldwide audience, people often do so cavalierly, with little concern for the effects of their words. But the quality and veracity of information matters, and the only way to oppose misinformation is to question and call to account those who spread it. I take responsibility for every word that I utter. If I err, I expect to be corrected. Whenever this happens, I hope that I will always be big enough to admit my error, apologize for it, and make amends by replacing the prior misinformation with the truth. I expect the same behavior from others.
Here are some thoughts…
Feature #1: “If it’s a thin client does it have Web2.0 RIA (Rich Internet Application) functionality (Flash, Flex, Silverlight, etc)?â€
Response: This is a feature that only a IT guy with myopia could appreciate, not someone who actually analyzes and presents data. When evaluating software, we care about functionality and usability, not about the specific technology that delivers it. If we’re exploring and analyzing data via the Web, what matters is that interactions are smooth, efficient, easy, and seamless. How this is accomplished technically doesn’t matter.
My Response: Although I do not yet have reached the myopia stage, I am sure you would care if you had to shell out thousands (if not hundreds of thousands) of dollars more to achieve smooth, efficient, easy and seamless visualization if the chosen technology is of a particular kind not readily available such as Web2.0 (hate to jump on the 2.0 bandwagon but what the heck). I assume that you, as most of those who analyze and present data, use computers and related technology to do so, which makes the point very relevant as the point of the blog was “How to Differentiate Advanced Data Visualization Solutions” (at least from a title perspective).
Feature #3: “Can the data be visualized via gadgets/widgets like temperature gauges, clocks, meters, street lights, etc?â€
Response: Is Evelson serious? Should vendors get points for providing silly, dysfunctional display gadgets? Most of the gauges, clocks, meters, and street lights that many so-called data visualization products provide are worthless. Anyone who understands data visualization knows this to be true. This is what Evelson looks for in “advanced†data visualization solutions?
My Response: If simplicity in data visualization is one of the ultimate objectives (as per your interview and several book statements), why is a simple gauge or a two way (+/-) visualization a “silly, dysfunctional display gadget?” Is there anything more simple than knowing, for example, if the rate of production is within estabilished parameters? Or if sales are performing within budgeted values? Conveying the message in a way that your audience can easily digest it and understand mustbe one of the most important mandates of a good data analyst or?
Feature #10: “Is the data that is being analyzed
a) Pulled on demand from source applications?
b) Stored in an intermediary DBMS
c) Stored in memory? This last one has a distinct advantage of being much more flexible. For example, you can instantaneously reuse element as a fact or a dimension, or you can build aggregates or hierarchies on the fly.â€
Response: What really matters is not where the information is stored, but how easily, flexibly, and rapidly we can access and interact with the data that we need. How this is accomplished technically needn’t concern us as long as it works.
My Response: Again, separation of technology and puristic statements such as “easily, flexibly, and rapidly” must be put to an end. It sounds like a software vendor which states that their software can do such things but not telling us how. I do not understand how separating data analysis from technology can one truly continue to grow in this field. Perhaps detailing 3 methods of data storage is too detailed for the purpose of differentiating advanced data visualization solutions but nonetheless, it is critical information to understand at least the basic architecture of a tool that enables us to perform data visualization better.
It would be similar to a doctor having a deep, theoretical and scientific knowledge about a patient but unable to understand the compounds behind a specific medicine, their side effects, dosage, duration and effectiveness on such patient.
Anyway just some comments on this interesting blog. Although I found indeed your response to a simple (perhaps misleading) blog by Forrester a bit aggressive in nature and sometimes contradictory, I admire your eagerness to point it out and to correct it as I am sure you will appreciate my attempt (or perhaps not).
Telmo,
Thanks for contributing to the discussion.
I don’t understand your point regarding Feature #1. Of course I would care if I had to shell out a lot more money to achieve the desired level of performance. If Web 2.0 provided the same functionality for less money, then I would purchase a product with Web 2.0. My point is that it is the performance and functionality that I care about, not the underlying technology that is used to achieve it. When people who are shopping for good data visualization software narrow their list to only those products that use particular technologies, they risk the elimination of products that might be much better and less expensive. I have personally seen IT departments block the purchase of the best products because they didn’t include some unnecessary technology that IT deemed important, resulting in the purchase of really bad products simply because they happened to include the technology that was on IT’s checklist of requirements.
Regarding Feature #3, the “gadgets/widgets like temperature gauges, clocks, meters, street lights, etc†that many so-called visualization products include are notoriously ineffective. To list these as desired features for “advanced data visualization†is absurd. Simplicity is indeed useful in data visualization, but these gauges, etc., are not simple, they are simplistic. They don’t display information simply, they display it poorly. Displaying information in a way that your audience can easily digest and understand is indeed the goal, but these gadgets almost never manage to do this. They are typically much more difficult and slower to read than other means of display, and they usually take up a great deal of space to say little when an alternative form of display could say much more in less space and still be easier and faster to digest and understand.
Regarding Feature #10, we should never lose sight of our fundamental requirements, which are not particularly technologies but the ability to do something in particular more effectively. Why must we “put to an end†our desire that data be “easily, flexibly, and rapidly†accessible, which is fundamentally what we need, unless we tie it to a particular technology. Technologies come and go, but our data sensemaking needs remain pretty much the same. Should I narrow my list of products to those that use an in-memory model if one that uses a hybrid in-memory/on-demand model performs better? Again, my point is that when shopping for a data visualization product, we should begin by understanding our needs and then build a list of the features, functions, levels of performance, and quality of operation that are required to meet those needs. When developing those lists, we turn to experts to learn more than we already know about what distinguishes good products from bad. We then examine products to determine whether they do what we require. We don’t accept the vendors’ word for it, we test the product ourselves. We don’t use a checklist of technologies (Does it use an in-memory model? Does it use Web 2.0 RIA?), we put our own hands on the products and put them through their paces with data sets of our own, making sure that they can do what matters to us and do it well.
After thinking about this dust-up for some time, I posted my comments on my Better BI blog. Here they are:
I’ve been thinking about this, and why I’m troubled by the nature of the conversation.
First off, I wholeheartedly agree with Stephen’s overall analysis of the contents of Boris’ Forrester blog in terms of the blog’s value in assessing the data visualization abilities of various products.
I applaud Stephen for pointing out the abrogation of professional responsibility by Boris and Forrester in publishing and promoting analyses which are in the main actively damaging in that they propagate misconceptions about their nominal topic area.
It’s distressing that many people have been critical of Stephen for providing the extremely valuable service of pointing out that the Emperor has no clothes. That many of these criticisms have come from Big-BI vendors is no surprise, as they can be expected to attack anyone who exposes their flaws and points out their shortcomings. It’s disheartening to see the same sentiments parroted by those who are actually suffering from these very same problems with the various products, but this simply attests to the success of the Big-BI vendors, and their promoters, in establishing the framing of the dicussion about the BI environment, of which data visualization is (in their view) a small part.
The big fly in the BI soup is that BI has become framed in the media and in the minds of most people solely in terms of very large, complicated, complex, expensive, and difficult to install and get operational data warehouse-based commercial products.
BI has become the fiefdom of large software companies whose motivation is to sell larger and more expensive products.
They are aided in this by organizations like Forrester and Gartner whose revenue is derived from providing analyses of the products in the areas they review. It’s in their interest to collude, if even through harmonic reinforcement, with the large BI vendors in promulgating the idea that Big-BI products -are- the way BI is done. The larger, more complex, and more expensive the products, the more “value” the analysts’ products-reports, quadrants, capability analyses, etc,-appear to be and the more revenue they can derive from them.
On one level, it’s understandable that Boris’ blog enumerated the feature set that’s been created as the desirable characteristics of commercial BI tools; these are the features that the Big-BI vendors have been promoting, are surfaced as Good Things in their products, and are therefore necessarily going to be prominent in the lists of features provided in Vendor representations, and by the majority of non-expert IT people who are passive market followers.
On another, more meaningful level, and this is where I think Stephen rightly criticizes Boris and Forrester, passing off bad information (and Boris’ list of features really does qualify here) as informed, expert analysis and advice really does come up short of professional standards, and does real harm in that it continues to reinforce harmful ideas that limit the effectiveness of delivering high quality information to people who need to make decisions.
I’ve been working in BI for twenty five years. Early in my career I was lucky enough to work for one of the companies that pioneered the field of BI. Our product was a specialized reporting technology that let us essentially sidestep the entrenched Data Processing environments then controlling things and get information into the heads and minds of business decision makers, often in hours instead of the weeks and months it took the DP shops to bring their big machinery to bear on even simple reporting requests.
BI has become the modern DP. The environment is ruled by the commercial interests of big technology companies.
Data visualization is, in this environment, a small backwater of little interest to the Big-BI-invested parties.
The products that provide high quality visualization capabilities are the early mammals in this world. They are more nimble, agile, and provide tremendous value that Big-BI tools do not.
To sum up, the crux of the larger issue here is whether one considers the delivery of high quality information or the installation of complex, expensive big machinery to be the point of BI. If the former, use the good tools and observe the principle “All BI is Local”, your users and clients will appreciate the value you deliver. If the former, spend a lot of time and money while delivering little if any information to the business decision makers; your users and clients will be impatient and frustrated.
Better yet, use the best modern tools where they provide their real value, and use the Big-BI tools where they contribute to improving the delivery of information, not impede it.
I couldn’t agree more with the issue that the BI field has been overtaken by “BI solution vendors” and in the process taken a different shape, for the worse in my opinion, by attempting to complicate matters with respect to visualization by adding superfluous gadgets, bells and whistles.
Chris, to your point this statement is, at least to me, very true:
“The big fly in the BI soup is that BI has become framed in the media and in the minds of most people solely in terms of very large, complicated, complex, expensive, and difficult to install and get operational data warehouse-based commercial products.”
However, please don’t mistake my previous entry as an attack on Stephen, in fact I admired his criticism of the status quo. I merely pointed out two factors:
1. There are better ways to critique someones work.
2. I do not agree with the separation of technology and data visualization.
Seeing that Stephen has conducted many workshops with BI solution vendors, I assume by his involvement in influencing those organizations, that he does neither.
Just like BI can be implemented by the delivery of clear, concise, simple data visualization assets, technology can also be simple and easy to implement. But without proper, innovative, solid, fast, cheap technology supporting BI data visualization processes, do not fool yourself into thinking that you can deliver information to any organization in a manner that will be useful or needed.
Perhaps my definition of BI is not the same as others, but if you can’t bring the data for the analyst to work with it, what good is data visualization?
If you can’t associate one fact with another, or on the fly align two metrics which at first seem unrelated, why would you bother visualizing data?
If you can’t discover new “golden nuggets of data” and all you need is a standard set of data visulization components that represent data, in perhaps a clearer way, but in no way give you any insight apart from what you already have in a table (or multiple tables) format, why purchase expensive BI tools – instead use Excel?
Again my point is, computer technology together with data visualization (if selected, implemented and used correctly in both areas) are pre-requisites for true BI. Anything short of that and it will either fail due to inability of users to comprehend the data being presented or it will fail because the data is not delivered in a timely, correct way.
Now, I am sure that even Stephen uses a fast, cheap, application to produce those amazing charts. Even perhaps a Web 2.0 application?
Microsoft internal memo, released as part of the Comes v. Microsoft trial:
“Analysts sell out – that’s their business model… But they are very concerned that they never look like they are selling out, so that makes them very prickly to work with.â€
I don’t know who actually takes analyst shills like Gartner or Forrester seriously. They are still considered a good marketing channel for software vendors, so there must be a reservoir of gullible and clueless CIOs somewhere out there who make purchasing decisions based on payola.
This can bee seens as a part of a larger issue: IT as a whole has begun to re-invent itself, to overcome its position as a provider of tools with many features to something that really solves problems that business stakeholders and human users face in their daily work.
In BI, this concerns having the right information at the right time and in the right format at hand, along with the right capabilities to analyse the data and to finally make business decisions. The “old” IT industry represented by analysts, IT departments and vendors, is still focussing on their feature-lists and in its majority just doesn’t see this shift yet.
I see this in every BI project we do, but also in projects involving other kinds of information systems or software. We as User Experience specialists are still aliens to IT people. Having a mixed background myself (Information Systems and Communication Design, which is a completely different way to think), I can see why, and yet I hope this shift will continue and result in a much more useful IT.
Milan
I am a fan after I read your book on data visualisation. I work in business intelligence. So, what according to you, should be three best qualities of BI software as far as data representation and visualisation is concerned?
Sid,
For analytical purposes, data visualization tools should only support useful visualizations (i.e., chart types) that are well designed to display data effectively (i.e., what’s going on in the data should be easy to see and understand, as well as accurately rendered), and every way in which the analyst might need to interact with the data to make sense of it (e.g., change the type of chart, sort the values, filter values, add and remove variables, express the values in other terms, break the data into a series of small graphs, create new calculated variables, group values in an ad hoc fashion, view data from multiple perspectives in multiple visualizations simultaneously, etc.) should be available and so easy to accomplish that the stream of analytical reasoning is not interrupted. Data analysis involves thinking. Tools that we use for data analysis should help us think more effectively. The primary activity of data sensemaking is comparison. Visualizations of and interactions with data should support rich comparisons. It is important to focus on the goals of data visualization before trying to list the features that support those goals.