Boris, speak only of what you know

Boris Evelson of Forrester Research has been singing off-key about data visualization recently and he doesn’t seem to realize that he’s tone deaf on this topic. Have you ever noticed that when people become recognized as experts in a particular field, they sometimes think this magically grants them expertise in other fields as well? Expertise requires study and years of practice, practice, practice. I’m particularly sensitive to this tendency when BI generalists give opinions about data visualization without taking time to understand it. This bothers me because people put their trust in “experts” and make costly decisions based on their opinions. My ire was most recently raised when reading statements by Evelson about data visualization as quoted in an InfoWorld article by Chris Kanaracus.

Evelson and I exchanged strong words back in 2009 when he deigned to list the features of “advanced data visualization” in his blog. His list was nonsense and I said so. Long after the dust settled, Evelson contacted me to ask if I’d be willing to advise him on matters related to data visualization. He should have asked my advice before his interview with Kanaracus.

Here’s the section of the article “Tableau BI visualization tools with user-centric design” (InfoWorld, January 18, 2012) that cites Evelson’s opinion:

Until the in-memory addition, Tableau wasn’t necessarily something a company already invested in a BI platform from SAP or Oracle would need, according to Forrester Research vice president Boris Evelson. “These days all of the other vendors have perfectly fine data visualization capabilities,” he said. “Now they let you do this in-memory, which very often is what the business users want. They don’t want to be restricted to the underlying database structure.”

At the same time, Tableau and its competitors need to further differentiate themselves. Microsoft is pushing PowerPivot as an extension of Excel with not much of a learning curve, while Spotfire features integration with Tibco’s middleware stack and offers advanced analytic capabilities, he said.

However, “whatever [Tableau] is doing, they’re doing it right,” as Forrester client interest in the company has jumped significantly of late, Evelson added.

I can imagine the mixed feelings of Tableau’s leaders when reading Evelson’s words: grateful he said that “they’re doing it right” but cringing to have these words spoken by someone who doesn’t actually understand what they’re doing and what makes it right.

Tableau did not suddenly become relevant to organizations with big BI product stacks when they introduced in-memory data handling. All along, these organizations have needed what good data visualization vendors like Tableau and their kin have been providing—effective ways to explore and analyze data—because the big BI vendors haven’t provided it and still don’t, which brings us to Evelson’s most naïve and potentially harmful statement: “These days all of the other vendors have perfectly fine data visualization capabilities.” After I read this, my wife mistook my convulsions as a seizure. Evelson’s statement couldn’t be more wrong. To date, none of the big BI software companies support data visualization in a manner that is “perfectly fine” or even reasonably adequate. They allow you to view data in graphs, but do so in embarrassingly inadequate ways. This inadequacy is especially apparent when we narrow our focus to exploratory data analysis, which requires meaningful and rapid interaction with data. Neither PowerPivot from Microsoft, Business Objects Explorer, nor any of the other attempts that I’ve seen by big BI vendors to enable exploratory data analysis have advanced past kindergarten. To draw on my Biblical roots for a moment, good visual analysis products such as Tableau, Tibco Spotfire, and SAS JMP lead people who have previously stumbled around in the dark using clunky BI products to exclaim “I was blind but now I see.”

Finally, back to our friend Boris Evelson. The best experts in any field are the people who started out as and continue to be the best students. When we stop being students, our expertise ceases to grow. When people recognize you as an expert and begin to hang on your every word adoringly, it’s tempting to wear that mantle with pride, refusing to ever again assume the role of student. If Evelson wants to express useful opinions about data visualization, he’s got some learnin’ to do. This is true of many BI thought leaders. Until then, they should stick to what they know.

Take care,

8 Comments on “Boris, speak only of what you know”

By Jim Jarrett. January 31st, 2012 at 12:28 pm

Recently read “Thinking, Fast and Slow” by Daniel Kahneman. He cites research that has shown that the most famous/visible/lauded pundits are also the most wrong. He specifically talks about stock pickers and political analysts, but I bet the results extend into other domains as well.

By George S. January 31st, 2012 at 1:55 pm

The problem with the so called “experts” from companies like Forrester, IDC, Gartner, etc is that they are all paid experts. Vendors pay to be reviewed, have white papers written, customized research, and even include their statements in the press releases and “industry articles”. On the other end buyers (organizations) pay them to access their research.

Not sure which side provides the most revenue, but from reading several of these experts I would guess the vendor side. Even in this example Boris seems to favor the big pocket companies Oracle and SAP, when new entrants like Tableau and Spotfire were clearly winners 3-4 years back.

Sadly the SEC (and govt) does not review these analysts like it reviews the stock analysts – and lets them talk junk to the public and misinform those looking for expert opinion.

By Mike R.. January 31st, 2012 at 8:38 pm

The ‘adequate’ BI vendors remind me of Plato’s Allegory of the Cave; those guys are still stuck looking at shadows, thinking that they’re reality.

By Steven N. February 1st, 2012 at 11:37 pm

Something one of my college professors said still resonates with me. He told my class that “many self-proclaimed experts you’ll encounter just happened read a couple of good articles”. In my experience, I haven’t found that to be too far from the truth.

You are right about major BI tools. Having spent several years of hair pulling with Cognos, I can tell you that anyone who *cares* about visualization hates the tool. I once sat in the popular “Best Practices for Report Design” seminar that is held annually at the Cognos forum and had to try very hard not to allow my eyes to roll to the back of my head. And this is in a presentation where you’ll hear the name Stephen Few dropped several times.

By Boris Evelson. March 11th, 2012 at 7:37 am

I invite all critics to reach out to me and take a look at our objective and detailed research on data visualization capabilities of leading BI vendors. If some of our evaluation criteria or evaluation results are factually incorrect, I’ll be happy to adjust accordingly. Untill then the objective results of our evaluations and client surveys will continue to form our opinions and advice. Not emotions.

By Stephen Few. March 11th, 2012 at 9:56 am


You’re welcome to post any of your research regarding data visualization here. I and others will critique it. What you’ve said publicly makes it clear that you know little about data visualization. Don’t expect us to pay a huge fee to see the work that you’ve done for Forrester Research.

By “research,” do you mean interviews and the results of questionnaires? If so, this is not true research into data visualization – what works, what doesn’t, and why – but a reflection of what users (mostly those in management) think they need. Useful research in any field begins with expertise in that field and seeks to extend what is known. Questionnaires won’t help you understand what people actually need or what will best serve their needs.

Twice in this blog I’ve critiqued your statements about data visualization. Rather than referring us to work that isn’t available to the public, respond to the content of my critique. If my critique is inaccurate, show us how.

By Yves de Montcheuil. March 13th, 2012 at 5:46 am

What ever happened to this myth of the Tier One Analyst, whose wisdom was never openly criticized (but made teeth grind in the boardroom and marketing office)?

Forrester seems to be coming under heavy fire lately. Boris Evelson gets criticized by Stephen Few in this post. Jim Kobielus took heat from me (guilty!) for his Hadoop Wave, where he is comparing elephants to kettles while forgetting many other options. Noel Yahanna and Rob Karel drew criticism from Vincent McBurney for desacralizing IBM in the ETL Wave.

And Forrester analysts are becoming even more stretched than before. They have lost some good people lately. They need to backfill, quickly. Or they won’t be able to cover all the domains that their clients expect them to. And the result will be just that – sloppy results, which receive open criticism.


By Vincent McBurney. March 15th, 2012 at 4:31 pm

My issue with the recent Forrester Wave reports is that they do not seem to have the depth of the Gartner Quadrants and therefore I do not trust the findings or summaries. For example they provide scores for vendors but do not include a list of strengths and weaknesses. They also include some company financial measures that seem bizarre and I agree with Stephen’s warning about an analyst making comments and rankings on a topic outside of their expertise.

Most concerning was a Forrester Wave that included a blatant vendor marketing statement. You need due diligence from analysts to check on claims made by vendors and to avoid some of the exaggeration that can come from marketing materials and briefs that analysts use to learn about products. I think we can expect a report to get carried away if it is sponsored by a vendor but when it is supposedly a comparison of vendors it needs to be independent.