Hollow Guidance from Forrester for $2,500 or Useful Guidance for Free – The Choice is Yours

Forrester has released a new report titled “Advanced Data Visualization (ADV) Platforms” by Boris Evelson and Noel Yuhanna, which you may purchase for a mere $2,500. I suppose that some reports might actually be worth the price of 50 books on the topic (at $50 each), but I’m certain that this report isn’t on that list. How can I be certain without reading it? (No, I didn’t pony up $2,500.) Because, based on his previous work, I know that Boris Evelson understands little about data visualization and is misinformed in many respects, and because the abstract that Forrester’s made available to tempt us into buying it reveals that it’s fundamentally flawed. Here’s what it says:

Enterprises find advanced data visualization (ADV) platforms to be essential tools that enable them to monitor business, find patterns, and take action to avoid threats and snatch opportunities. In Forrester’s 29-criteria evaluation of ADV vendors, we found that Tableau Software, IBM, Information Builders, SAS, SAP, Tibco Software, and Oracle led the pack due to the breadth of their ADV business intelligence (BI) functionality offerings. Microsoft, MicroStrategy, Actuate, QlikTech, Panorama Software, SpagoBI, Jaspersoft, and Pentaho were close on the heels of the Leaders, also offering solid functionality to enable business users to effectively visualize and analyze their enterprise data.”
[emphasis mine]

No one with even a modicum of expertise in data visualization would place IBM, Information Builders, SAP, and Oracle in the list of advance data visualization leaders along with Tableau Software, Tibco Spotfire, and SAS, nor would they leave good vendors such as Panopticon and Advizor Solutions off the list. The chasm that separates these data visualization providers is huge. Lists and claims like these are typical in reports by Forrester and Gartner. There’s a reason for this. Do you know how software vendors get onto Gartner’s annual BI magic quadrant? They pay for the privilege. When vendors that deserve notice are missing from the list, it’s because they haven’t paid the fee. When they suddenly appear in the magic quadrant although they were never there before, it isn’t because they’ve improved in a significant way, but because they paid the fee for the first time. I don’t know exactly how Forrester works, but I suspect that it’s the same. (Anyone who works for Forrester is welcome to tell us differently by responding to this blog.)

Even though I haven’t read the report, I did get a glimpse into it by reading Evelson’s recent blog post about it. What does Forrester mean by advanced data visualization?

How is Advanced Data Visualization (ADV) is [sic] different from earlier generations of data visualizations? Many corporations have effectively used — and will continue to use — traditional business graphics, such as bar charts and pie charts. At the next level, modern technologies have enabled the use of more dynamic and interactive business graphics, such as real-time dashboards and charts that update automatically as the data changes…Now, through ADV, potential exists for nontraditional and more visually rich approaches, especially in regard to more complex (i.e., thousands of dimensions or attributes) or larger (i.e., billions of rows) data sets, to reveal insights not possible through conventional means. Forrester differentiates ADV from static graphs and charts along six capabilities, as follows:

  1. Dynamic data content.
  2. Visual querying.
  3. Multiple-dimension, linked visualization.
  4. Animated visualization.
  5. Personalization.
  6. Business-actionable alerts.

What becomes immediately clear to anyone with experience in data visualization is the fact that by “advanced” Forrester is referring to features that have existed for quite awhile. Tableau, Tibco Spotfire, SAS (the product JMP in particular), Panopticon, and Advizor Analyst have always supported these features, with the exception of animation, which is only useful for some visual storytelling, not for data exploration and analysis, and “business-actionable alerts” (don’t you just love the jargon?), which are useful for performance monitoring. The bar is set fairly low. Forrester is just talking about basic information visualization as defined by Card, Mackinlay, and Shneiderman in 1999 as “…the use of computer-supported interactive visual representations of abstract data to amplify cognition.”

But wait, there’s more.

Navigating the ADV landscape requires evaluating significantly more features than the six key ADV capabilities described in the previous section. In our latest research, Forrester identified numerous functional and technical capabilities businesses need to architect, design, build, and implement ADV applications. Forrester recommends starting your evaluation of ADV platforms by defining your requirements for the following functionality:

  • Types of graphs, charts and other visualizations.
  • Tufte’s microcharts.
  • Cockpit gauges. [Seriously?]
  • Visual query.
  • Visual exploration. [Duh!]
  • Geospatial representations. [This is just one of those useful forms of data visualization that belongs among the "type of graphs, charts and other visualizations" that they referred to in their first point above.]
  • Modes of interaction.
  • Storyboarding fit for client and boardroom-level presentations.
  • Data latency.
  • Data granularity based on your requirement.

The points in the list above — those that are useful — are well known and have been for ages. Granted, they are not well known to everyone in the business intelligence community, but should these folks really pay $2,500 for this report to learn about them when excellent books and many free resources are available, which were written by actual experts in the field?

Typical of many in the field of business intelligence, Forrester’s team is obsessed with the technology rather than the skills and activities that inform effective data visualization, especially of the advanced variety. This fact is revealed by the following section of Evelson’s blog:

Forrester identified eight categories of ADV technical architecture capabilities through posing the following questions:

  • What analytical engines does the ADV platform support? How does it access and process data?
  • Is there an intermediate storage platform?
  • How is the in-memory data model managed?.
  • What types of data can the ADV platform analyze?
  • Does the ADV platform support write-backs?
  • What platform/technology is the ADV output based on?
  • What, if any, ADV-specific programming language is used?
  • What are the ADV platform’s integration capabilities?

Most of these are the wrong questions. If either of the two authors of this report and any of the seven contributors had any real experience with data visualization or had ever worked as data analysts, they would realize this. Read the biographies of the contributors to this report and you will discover in each case someone who is entrenched in an old and myopic BI mindset. To understand the potential of data visualization, especially advanced data visualization, you must actually study it and do it. In many respects, an extensive background in BI, especially at an expert level, works against one’s ability to understand the interaction with data that is needed for sensemaking. It is a human activity that involves our eyes and brains. It can be supported wonderfully by well-designed technologies, but never driven by them. To date, none of the software vendors on Forrester’s list, except for the three that I singled out as worthy, have developed better than rudimentary data visualization tools, many of which are truly awful.

If you want to learn about data visualization, especially what to look for in a well-designed tool, read the article in my most recent Visual Business Intelligence Newsletter titled “Criteria for Evaluating Visual EDA Tools“. You will not only learn more than you would by reading Forrester’s new report, but you will also save yourself $2,500. Now that’s a deal. If you’ve already paid for Forrester’s report, demand your money back. Make them accountable for delivering a level of expertise that warrants the high price.

Take care,

33 Comments on “Hollow Guidance from Forrester for $2,500 or Useful Guidance for Free – The Choice is Yours”


By Rebecca Croucher. July 20th, 2012 at 9:11 am

Great article. I initially started to read your article because I was surprised that someone would have the nerve to stand up to Forrester and I had just looked at the $2,500 price tag on that report yesterday and thought it was too much. Then when I started to read your article it became clear that you were well versed in ADV and stood behind your subject line with facts. I’m usually most interested in what and how they evaluate, so your points and article were priceless, thanks.

By Andrew. July 20th, 2012 at 10:39 am

‘“business-actionable alerts” (don’t you just love the jargon?)’

Just as with their understanding of data visualization, appearance is everything. It doesn’t matter if the data make sense if they look good, and it doesn’t matter if the words mean anything if they sound professional.

Also it’s redundant; shouldn’t *all* alerts be actionable? If I get an alert and it doesn’t require that I take action, then what’s the point? Such an alert is only wasting my time and training me to ignore alerts.

By Stephen Few. July 20th, 2012 at 11:07 am

Something that has concerned me about Forrester’s claims is the fact that Evelson’s blog about the report has been picked up and republished (in part) by many online BI publications. These publications repeat anything that comes from Forrester or Gartner as Gospel. As we have been reminded many times, if you say something often enough, people will believe it is true. Most people in the U.S. probably still believe that there were weapons of mass destruction in Iraq and an absurdly large number believe that Obama was not born in the U.S. The lie that SAP, IBM, Information Builders, and Oracle are leaders in advanced data visualization today, although absurd to anyone who understands the space, will become an accepted truth in the minds of the larger BI community. People who spread such lies should be ashamed of themselves. Their lies cause great harm. They may not lead to a war, as did the lies regarding Iraq, but they place blinders on the eyes of thousands — perhaps millions — of people who rely on information to make decisions. They foster the very opposite of BI’s mission: to enlighten, leading to better decisions.

You might think it is harsh to call the false claims in Forrester’s report “lies.” Perhaps these claims were made in ignorance. If so, when people claim to be experts, they take on a responsibility for knowing the truth. They can’t be all-knowing, but they have no excuse for ignorance of such fundamental facts.

By Richard Jarvis. July 20th, 2012 at 11:48 am

An excellent article Stephen; I look forward to the day when the business model used by Forrester & Gartner is fully exposed, as I give it as much credibility as American Idol.

The only part of your article I don’t really agree with is at the end where you dismiss the relevance of questions about the platform and infrastructure. It goes without saying the front end capabilities of the toolset are the primary concern, however if the toolset doesn’t scale effectively (e.g. any more than a handful of users degrades performance for everyone else) or is insecure (e.g. unencrypted data cached locally), it may not be suitable for the enterprise market or particular industries.

Given the target audience for their report was the enterprise (at that price it would have to be!), and the title of the report specifically mentions platforms, I think it’s appropriate to include the infrastructure and architecture within the report’s scope.

By Stephen Few. July 20th, 2012 at 12:09 pm

Richard,

Asking these particular questions about the platform doesn’t get to the heart of the matter. The fundamental question is “Does the technology deliver what’s needed in a way that really works?” Their approach assumes that particular platform implementations are the best and only solutions. BI vendors love questions like this, because they can easily put a check next to the question indicating, yes, our software does that. The technology platform is important. I would never dismiss it. Their questions are in fact a dismissal of the platform’s importance in that they reduce platform concerns to a series of specific technologies that may or may not actually work. Using an in-memory engine is no guarantee that the system is adequately responsive. The programming language that was used to develop the product says nothing about the quality of the code.

By Daniel Smith. July 20th, 2012 at 12:33 pm

Thank you, Mr. Few. Reading your blog is refreshing and I always look forward to your posts. This is another great one! Thanks for your attention to detail and for calling out the spread of misinformation. Keep up the good work!

By Richard Jarvis. July 20th, 2012 at 12:44 pm

That’s a good point Stephen. Re-reading the questions I understand where you’re coming from now, so thank you.

BTW, another factor in selecting a toolset for us has been international neutrality. For example, one of the unexpected features I immediately noticed in Tableau when I first tried it was how easy it was to realign the calculation and display of financial quarters; it wasn’t until I saw that I realised how irritating it had always been creating my own groupings in other tools.

To your point, what I really appreciate in tools such as this is how effective they’ve been in removing technical pre-conceptions which might negatively impact the user experience.

By Andrew. July 20th, 2012 at 2:07 pm

@Stephen: “People who spread such lies should be ashamed of themselves.”

That’s an interesting thing to say. What if they believe the lies they’re telling? How can they be ashamed if they don’t even know they’re lying?

Here’s a couple things to consider when attempting to shame people for unethical behavior, such as lying to potential buyers about what features they need in a product:
1. Hanlon’s Razor (http://en.wikipedia.org/wiki/Hanlon%27s_razor), which, for many of the above-mentioned megavendors, I strongly suspect.
2. Psychology. I recently read a post over at Schneier’s blog about the psychology of unethical behavior (http://www.schneier.com/blog/archives/2012/05/the_psychology_5.html); the links in the article are fascinating as well. It’s interesting to me to consider that Gartner and Forrester simply may not see the damage that poor data visualization can cause, especially when compared to the overwhelming interest in the subject of visualization and widespread recognition of its merits.

Of course, none of that means liars shouldn’t be held accountable for lies, and I certainly appreciate anyone who calls them out. I’m just pointing out that it’s not really useful to shame a blind man for being blind.

“You might think it is harsh… Perhaps these claims were made in ignorance. If so, when people claim to be experts, they take on a responsibility for knowing the truth.”

If that’s the case, Hanlon’s Razor might not apply. However, this link might still be relevant: http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Hmm. Seems I have a few links today.

By Mad Hemingway. July 20th, 2012 at 5:56 pm

Evelson and Yuhanna are the perfect Forrester guys. They’re selling. What you see is the big vendors paying someone to hype their next big tool push. Customer reads Forrester report, decides they they’ll give it a try, and voila, demand is created.

Regards,
A Former IBM Guy

By Stephen Few. July 20th, 2012 at 6:30 pm

Andrew,

People who believe lies and spread them when the truth is clear and readily available are behaving shamefully. This is especially true of people who claim to be experts. Yes, they should be ashamed. Those who live in darkness because they refuse to open their eyes get no sympathy from me for being blind.

The authors of this report, in fact, have no expertise in data visualization. They are BI generalists who haven’t taken the time to learn about data visualization. So, the question is, are they so ignorant that they really believe their claims. It’s hard to imagine that ignorance is the basis for these lies. While they are not experts in data visualization, they have had enough exposure to these products to recognize the chasm that separates the leaders from the wannabes. To believe these lies, they would have to not only be ignorant, but be stupid as well, and that’s unlikely. It is much easier to believe that they reward the big vendors that pay them dearly for consideration than that they’re ignorant of the falseness of their claims.

By Stephen Few. July 21st, 2012 at 5:10 pm

UPDATE!

Thanks to a colleague who sent me a copy of Forrester’s report, I can now say that it is actually worse than I imagined. This report contains a mere 17 pages of content. It is roughly equal in size to many of my articles in the Visual Business Intelligence Newsletter, which are more informative, based on a deep understanding of data visualization, and available to anyone without charge.

What an extraordinary service Forrester provides. They have their in-house analysts do so-called research into areas that they don’t understand, based in part on contributions from others who are equally clueless. They then publish a few pages of shallow content. Unsuspecting buyers, desperate for help, assume that the report is reputable research by experts, so they pay $2,500 for information that is more misleading than it is helpful. In other words, Forrester thrives on ignorance: their own to generate the work and that of their customers to pay for it. What a wonderful business model. What drives an organization and its employees to produce such impoverished work? Actually, we know what drives it, don’t we? If you can make a profit building houses of cheap plywood, masking tape, and spackle using unskilled labor, why do more? Why indeed, unless you have integrity?

The report consists of little more than the preview that Evelson provided for free in his blog. It appends a paragraph or two of flesh to each of the bare bones that were exposed the blog and adds a table of scores that were assigned to the 14 vendors based on 20 criteria. Where the content of the report isn’t misguided or otherwise flawed it is ho-hum obvious. The criteria that were used to rate the products fell into three major categories: 1) current offering – a mix of 12 functional and technological features, 2) strategy – commitment, pricing and licensing, transparency, and product direction, and 3) market presence – the vendor’s financials, size of its installed base, “partnership ecosystem,” and “functional BI systems.” As you can see, many of the criteria have nothing to do with data visualization. For the overall ratings that were assigned to the vendors, the criteria were weighted, with 50% of the score coming from current offering criteria, 50% from strategy criteria, and apparently a puzzling 0% from marketing presence criteria. These criteria were not properly designed to produce a reliable comparison of the “advanced data visualization” capabilities of these vendors. Also, the way that scores were assigned was clearly flawed. For example, of the 14 vendors that were considered, those that supposed have the greatest “commitment” to advanced data visualization are Microsoft, IBM, Information Builders, SAP, and SAS. Apart from SAS, what evidence exists of this commitment? Did they derive this information from marketing brochures? In the category of product direction, 7 of the 14 vendors scored better than Tableau. Seriously?

The report includes a bubble plot to visualize the overall results of the ratings.
According to this chart, every one of these 14 vendors is either an advanced data visualization leader or strong performer, with two vendors on the border between strong performers and contenders. None even come close to being “risky bets.” Is this a realistic assessment? Hardly. In the realm of data visualization, at least half of these vendors are risky bets, with nothing yet to show.

If this is any indication of Forrester’s overall value, customers should pull out of the service and demand a refund.

By Kyle. July 24th, 2012 at 12:29 pm

Interesting how little difference there is between the ADV report by Forrester and the The Forrester Wave™: Self-Service Business Intelligence Platforms, Q2 2012 released a mere month prior.

It would be unfortunate if customers paid for both, or either. Link can be found on SAS’s website:

http://www.sas.com/news/analysts/Forrester_Wave_Advanced_Data_Visualization_Platforms_Q3_2012.pdf

http://www.sas.com/news/analysts/Forrester_Wave_Self_Service_105855_0612.pdf

By JD. July 25th, 2012 at 11:27 am

@Kyle - Thanks for saving me thousands of $ on a sub-par report!

By Victor Blaer. July 25th, 2012 at 1:03 pm

They´d rather make a buck than be right.

By Stephen Few. July 25th, 2012 at 2:27 pm

Victor,

Apparently, that’s the case. Sad, isn’t it? Especially when it you can make and buck and be right, which is much more rewarding.

By Stephen Few. July 25th, 2012 at 2:49 pm

I have just learned that Boris Evelson of Forrester has been challenged via a tweet to respond to this blog post. The first time that I exposed his impoverished understanding of data visualization in this blog, he did respond. The interaction did not go well for him. Since then, he solicited my services as an adviser. You may be surprised to learn that Forrester, according to Evelson, however, never pays for the services of expert advisers. They form partnerships with them in the form of “You scratch my back and I’ll scratch yours.” If you’re willing to read Forrester’s reports in advance of publication and suggest corrections and improvements, they return the favor by citing your research and sending clients your way that need expert consulting support. Those who forge such relationships with Forrester are loathe to speak out against them, so this relationship works to Forrester’s advantage, but definitely not to the advantage of their clients. I told Evelson that I would be happy to assist him in the same way that I would assist any organization that requests my help beyond what I freely offer on this web site, for which I charge a daily consulting fee.

I would appreciate a response from Evelson regarding my assessment of his report, but I realize this isn’t likely. Unlike politician’s, most business professionals are loathe to engage in public debate about issues that they don’t understand.

By Kyle McNabb. July 26th, 2012 at 4:22 pm

Stephen,

Our objectivity and independence is at the core of who we are at Forrester Research. Our business success has been built on it. I want to clear up a couple of fallacies that I’ve come across in the post and the comments section:

1. We carefully select the vendors and solutions we evaluate in our Wave evaluation, and then defend them to a group of Research executives (myself included). This selection is driven by our research into the market’s needs and trends.

No vendor has ever paid to be included in a Forrester Wave. In the spirit of transparency I’ll gladly share that many vendors do try, and we politely tell them that’s not how we work. Our Wave process is laborious, taking us considerable time, from inception, to lab reviews, to publication. And we commit to our clients that we will do the same thing again to help them stay on top of the market.

2. Our research is exhaustive. You can disagree with it - we welcome constructive discussions - but our research is not built on “lies.” It is instead built on objectivity and transparency. We interview and speak with thousands of end user organizations and experts - like yourself - to determine what business outcomes they’re trying to accomplish, and understand their needs - not just with a particular feature - but from an overall portfolio standpoint as technology decisions cannot be made in isolation or without regards to an organization’s skills, budget, and risk tolerance. In fact, in the Wave research we do we publish our findings not in a static way, but in a means allowing clients to change the results if they disagree with how we’ve prioritized various features and capabilities.

If any of you are interested in engaging in a respectful discussion about our Wave methodology, please let me know.

Kind regards,

- Kyle McNabb
Vice President & Practice Leader, Forrester Research

By Stephen Few. July 26th, 2012 at 7:27 pm

Kyle,

Thanks so much for responding on behalf of Forrester. As you might imagine, now that you’ve opened the door, I have a few questions for you.

Where do you fit in the organization in relation to the authors of this report about advanced data visualization? Do you have any expertise in data visualization?

You’ve challenged my arguments, so let’s begin with that. I have no doubt that you “carefully select the vendors and solutions” that you evaluate. I’ve suggested that you did so in this case, however, not based on an informed awareness of the best data visualization products, but on your relationship with these vendors. I did not say that these vendors directly paid you to be included in this report, but they paid you nonetheless. I, my readers, and the people who have paid for this report would like to know which of the vendors that were considered in your report are not your clients from whom you’ve received payment for services.

You’ve claimed that your research is “exhaustive.” It was perhaps exhausting for your analysts to compile this report due to their ignorance of data visualization, but this report was far from exhaustive. No credible organization of your type would allow a report of this low quality to be produced. Did anyone with expertise in data visualization vet this report? If so, who? You claim that you “interview and speak with thousands of end user organizations and experts” like me. You did not actually speak to thousands of organizations to produce the content of this report, did you? Even if you did, end users do not possess the expertise that is required to rate the merits of advanced data visualization vendors. Regarding experts, you did not speak with me, so I’m interested in knowing which experts you did speak with. No real expert in the field would endorse the contents of this report. I invite you to prove me wrong by naming any real expert who finds this report worthwhile.

Now, on to other issues.

Do you agree that reports produced by Forrester should be written by experts in the field? Do you agree that those who pay for the reports that you produce expect that they were written by experts? If so, the very fact that Forrester has produced a report of such low quality by people who don’t understand data visualization as if it were the product of expertise is an example of your willingness to lie.

Boris Evelson, one of the authors of this report, has admitted to me privately that he lacks expertise in data visualization. He also explained that it is Forrester’s policy to never pay for the services of outside experts. This is an arrogant and self-defeating policy. You charge your clients a great deal for your “expert” services, but when you lack expertise yourself, you refuse to pay for the expertise of others that’s required to produce useful reports. You should either refrain from producing reports regarding matters that you don’t understand or you should acquire the services of those who possess the expertise. To do otherwise is a lie and a disservice to your customers.

You acknowledge that I am an expert in the field of data visualization, especially as it relates to business intelligence. As such, I’m telling you that this report is worse than worthless. It is actually harmful in that it will lead people to select products that don’t work, resulting in great loss. Either I don’t know what I’m talking about or I’m right and you’re guilty of harm. Which is it?

You refer to this report as the result of research, but it does not qualify as actual research. You solicited opinions from people who know relatively little about data visualization. At best, you have reporrted opinions. This report was not presented as the findings of an opinion poll, however. It was presented as expert guidance regarding the characteristics of advanced data visualization, including an expert evaluation of vendors. Of the facts that you’ve reported, those that are accurate are common knowledge in the field of data visualization. The rest is either outright error or fabrications to falsely elevate the merits of vendors that are your clients.

Useful evaluations are based on appropriate criteria. Your report was not. Please explain why your evaluations were based 50% on criteria that had nothing to do with the data visualization capabilities of the products. I believe that I know why. You were looking for ways to improve the scores of vendors that are currently doing little that is worthwhile in the field of data visualization. Please explain how 5 out of the 14 vendors that were considered scored higher than Tableau in their “commitment” to advanced data visualization. In my opinion, this is an example of a lie. The authors of this report, however ignorant of data visualization, cannot really believe that Microsoft, IBM, Information Builders, SAP, and SAS are more committed to advanced data visualization than Tableau. Of these, only SAS has demonstrated a substantial commitment to advanced data visualization and their commitment, however substantial, is not greater than Tableau’s. Also, please explain why you believe that 7 or the 14 vendors exhibit superior product direction in the field of data visualization than Tableau. Do you expect us to believe that you were not showing favor to your largest clients?

Forrester might produce useful reports about other technologies, but shame on you for producing such drivel about data visualization and having the nerve to charge $2,500 for it. If Forrester as an organization has integrity and the work that you’ve produced about data visualization is an embarrassing exception, then you should own up to your mistake and return the money that anyone has spent on this report. Do that, and you will begin to earn my respect.

By Dan Murray. July 28th, 2012 at 12:49 am

Bravo!

By Gregory Lewandowski. July 30th, 2012 at 7:19 am

Well done Stephen!!
Great questions, well thought out, and I am looking forward to the response…

By Ellie Kesselman. July 30th, 2012 at 8:06 am

This matter, of the price of Forrester Research reports, has piqued my curiosity for years. Five+ years ago, I had access on a two or three occasions to quantitatively-themed Forrester Research reports. (I am a statistician, data quality and regulatory analyst). They were decent quality. Yet I thought to myself,
“Wow! I would not have wanted to pay $400 for that!”.
Since then, I have seen many Forrester Research reports in the range of $2500. Some older reports, described as “outdated, for historical interest only”, cost $1500 to $3000, just like new reports!

I am surprised that Forrester would not pay external subject matter experts to review or validate research prior to publication. That might be acceptable for more “commodity” like subjects. Yet for a field where the Forrester analysts, or Forrester in general, has limited expertise, I am PARTICULARLY surprised that vetting by a known expert in the field (paid at hourly consulting rates) is not standard procedure.

Thank you so much for writing this, Stephen Few. I wanted to ask someone with exposure to Forrester Research. Now, finally, I have finally found it. Highlight: A response by VP Kyle McNabb of Forrester Research, who was so kind as to respond (pleasantly, and at length) in an earlier comment.

By Stephen Few. July 31st, 2012 at 9:14 am

Kyle,

Will we be hearing from you?

If you have anything to say in response to the facts that I’ve presented and the questions that I’ve asked, this is your opportunity to either justify the merits of Forrester’s advanced data visualization report or to acknowledge its deficiency. Admitting the failure of this one report is certainly not an indictment against Forrester in general. I’m inclined to believe that Forrester does excellent work in some areas but that data visualization is unknown territory that you should ignore until you’ve either developed or hired the required expertise. By missing this opportunity to acknowledge the poor quality of this one report, you will cast doubt on the integrity of Forrester’s work in general. Why should we believe that Forrester is an objective and qualified reviewer of other technologies when you defend the merits of this particular report, which is so clearly flawed? When an expert in the field shows your work to be juvenile, ill-conceived, error-filled, misleading, and biased in a self-serving manner that contradicts the objectivity that you claim, a response is in order.

By Kyle McNabb. July 31st, 2012 at 9:17 am

Thanks Stephen.

I lead our research teams focused on helping leaders of application development & delivery, and sourcing & vendor management make better decisions. Boris Evelson is a key member in my organization.

We recognize many, including experts such as yourself, will have different views and assessments of the market than our own. We don’t claim to own a monopoly on expertise and when we develop our research plans we specifically identify experts - from a variety of sources - to interview to help inform our research and complement interviews and data we collect from end users of technology. Our objective is to provide the most objective, fact based research we can to help folks make informed decisions. Those decisions are not always centered on who has the best “widget”, but what “widget” best fits their broader needs that have to include skills, budget, and a variety of their factors and criteria.

We measure our success, partially, on how well we empower decision makers to own their decisions, and reduce their reliance on consultants to both make and implement decisions on their behalf. We stand behind our research, including offering a money back guarantee of our service.

Regards,

Kyle

By Stephen Few. July 31st, 2012 at 10:13 am

Kyle,

You do know, don’t you, that standing behind this report when you don’t personally have the means to evaluate it, is meaningless. Also, standing behind your employees when they produce poor work is not a position that your customers will appreciate.

It’s sad that you have nothing to say in response to my specific observations and questions. It is clear that you have chosen this path because you cannot defend this report. If you measure your success by how well you empower decision makers to own their decisions and reduce their reliance on consultants you’ve failed miserably in this instance. You have given decision makers erroneous, misleading, and harmful advice. This is not a matter of a difference of opinions among experts. This is a matter of Forrester having no expertise whatsoever in this particular field and no substance whatsoever to back its claims.

It is Forrester that has taken the “who has the best widget” approach to evaluating data visualization products. You used a checklist of features in some cases and subjective criteria in others (e.g., best product direction), and reduced your evaluation to counting checks in boxes, wild guesses based on naiveté, and partiality in favor of vendors that pay you. You have provided no evidence that Forrester has any expertise whatsoever in data visualization. The reason is clear: you have none. You have provided no evidence to counter my observation that you designed and conducted this so-called research to serve the interests of those vendors that are your customers. The reason is clear: my observations are true. You have failed to name a single expert in data visualization that you consulted when preparing this report. The reason is clear: none were consulted.

What I find particularly interesting in your response is the fact that you don’t recognize the logical error in the basic premise of your work: “We empower decision makers to own their decisions, and reduce their reliance on consultants.” You are a consulting organization. Your entire business model is based on a reliance of customers on your consultants to inform their decisions. You fundamentally fail your customers by providing unreliable advice that will cost them dearly. Your business model — rely on Forrester’s consultants so you don’t need the services of other consultants — is only viable if you provide useful and comprehensive advice. This little interaction of ours demonstrates how little you actually care about your customers. You can ignore people like me who reveal errors in your work, but you do so to your own eventual peril. People aren’t as stupid as you seem to believe.

You have a choice: either stick to what you know and work hard to provide good advice, or stick your fingers into every pocket that is bulging with opportunity (data visualization, in this case) without preparation or skill and hope that no one notices your sleight of hand.

By Scott. August 1st, 2012 at 8:33 am

I say well done as well.

Too many companies out there think they can call themselves experts, when they really know nothing about the subject they are reporting on, just to make a buck. Based on what I have read, they know they made a mistake and are just trying to do public relations cleanup. I look forward to a response.

Keep up the good work and continue to call out people and companies whose so-called research is fundamentally flawed.

By Stephen Few. August 3rd, 2012 at 1:30 pm

Several days have now passed without further response from Forrester. They must have nothing more to say, which is disappointing, because they’ve said what amounts to nothing. They chose the approach of a politician whose hand has just been caught in the till. Hollow words, bereft of meaning.

What Forrester does (and perhaps other organizations of their type as well) amounts to extortion. Technology vendors pay them for protection. Like the Mob, you either pay up or you suffer. By paying up, you get to be included favorably in Forrester’s so-called research reports. The more you pay, the more slack you’re given for your shortcomings. By failing to pay, you get left out as if you don’t exist, even if you’re better than the vendors that paid. This is a racket.

The only way to put an end to this is for the vendors – lots of them – to refuse to pay. In my opinion, this is precisely what good vendors should do to act responsibly. By paying Forrester, they’re endorsing Forrester’s work, which in the case of this report about advanced data visualization is doing harm by giving bad advice to customers who deserve better and are certainly paying for better.

If you’re a customer that pays Forrester for advice, stop assuming that their advice is good just because you paid dearly for it. Demand that they either produce work that is grounded in expertise or stop paying it. Take the questions that I’ve asked them about the advanced data visualization report, which they’ve refused to answer, and ask Forrester these questions yourselves directly. If they deign to respond and you want to pass their answers through a shit-detector, send them to me and I’ll happily oblige.

BI software companies that have no right to be called leaders in advanced data visualization are now proudly displaying their award from Forrester in the media and to their customers. I saw another press release today that made me sick. This galls me. Why? Because it’s wrong and it’s harmful. I work directly with people who struggle every day to squeeze something useful from bad data visualization products. It hurts to see this. How many of them are stuck with dysfunctional tools because their CIO trusted Forrester or some other misinformed technology pundit? We have a voice; let’s use it. We have a choice; let’s make it count.

By Nick G. August 10th, 2012 at 5:14 am

Stephen - I think you are right in your challenge to Forrester. Same type of things go on with “analytics” packages. Lots of reports and articles about toolsets but a noticeable lack of understanding on how the work actually gets done.

Thanks for pressing on with this. I’m hopeful that it helps to bring about a higher quality of reporting on this topic.

By Brian Hackett. August 11th, 2012 at 8:17 am

As a researcher who has worked for some of these “best practice” and “benchmarking” firms, all I can say is great job Stephen. More light should be shed on all of these firms that claim to be experts and then push shallow and often harmful “objective research”.

By Doug Magnuson. August 24th, 2012 at 6:31 pm

I work in a professional school, in academia, not in business intelligence. Your critique here and the suggestions in your newsletter applies as well to the use of data and the methods of generating it in the human services. Nicely done.

By Matt Samelson. August 27th, 2012 at 7:55 am

You have hit a very important point on the head - many so called “analysts” or “experts” have slim to no experience in their field of expertise. Furthermore - and as a result - the “research” they produce has virtually no value.

I work for a consulting and research firm where all professionals have at least 10 years of direct, applicable, experience in their fields. As a result, I believe that our research (which, by the ways, IS truly objective), is a far superior product.

It is important not to throw the proverbial baby out with the bath water as not all “research” falls into this this category. Unfortunately, a lot does.

By Ryan Robitaille. August 28th, 2012 at 12:48 pm

Excellent article (and comment thread), Stephen.

Reminds me of a common fallacy that people fall into when USING “advanced” data visualization tools - as well as when purchasing them (i.e. Forrester’s checkboxes and “eval cirteria” above).

NOT Focusing on what actually MATTERS.

If it doesn’t help you accomplish your goals at the end of the day - it’s just worthless BS. Regardless of how fancy the “intermediate storage platform” and “in-memory data model handling” is…

By Stephen Few. September 7th, 2012 at 8:30 am

Poor research like this from Forrester has severe and ongoing consequences. CIO magazine has now repeated this list of advanced data visualization vendors as if it were gospel, and they reordered the 10 vendors in a way that placed the best at the end of the list, suggesting that they’re low on the totem pole. This is how lies are propagated.

By Andrew McDaniel. September 12th, 2012 at 7:17 pm

Mr.Few,

I don’t know if you saw this, but ZDNET had an article that cites another Forrester employee on the subject of “Data visualization tools need to be intutive” Duh. Both the author of the blog and the Forrester employee provided only drivel, no new ideas. Here’s the link http://www.zdnet.com/data-visualization-tools-need-to-be-intuitive-7000004034/.