‘“business-actionable alerts” (don’t you just love the jargon?)’

Just as with their understanding of data visualization, appearance is everything. It doesn’t matter if the data make sense if they look good, and it doesn’t matter if the words mean anything if they sound professional.

Also it’s redundant; shouldn’t *all* alerts be actionable? If I get an alert and it doesn’t require that I take action, then what’s the point? Such an alert is only wasting my time and training me to ignore alerts.

An excellent article Stephen; I look forward to the day when the business model used by Forrester & Gartner is fully exposed, as I give it as much credibility as American Idol.

The only part of your article I don’t really agree with is at the end where you dismiss the relevance of questions about the platform and infrastructure. It goes without saying the front end capabilities of the toolset are the primary concern, however if the toolset doesn’t scale effectively (e.g. any more than a handful of users degrades performance for everyone else) or is insecure (e.g. unencrypted data cached locally), it may not be suitable for the enterprise market or particular industries.

Given the target audience for their report was the enterprise (at that price it would have to be!), and the title of the report specifically mentions platforms, I think it’s appropriate to include the infrastructure and architecture within the report’s scope.

Thank you, Mr. Few. Reading your blog is refreshing and I always look forward to your posts. This is another great one! Thanks for your attention to detail and for calling out the spread of misinformation. Keep up the good work!

@Stephen: “People who spread such lies should be ashamed of themselves.”

That’s an interesting thing to say. What if they believe the lies they’re telling? How can they be ashamed if they don’t even know they’re lying?

Here’s a couple things to consider when attempting to shame people for unethical behavior, such as lying to potential buyers about what features they need in a product:
1. Hanlon’s Razor (http://en.wikipedia.org/wiki/Hanlon%27s_razor), which, for many of the above-mentioned megavendors, I strongly suspect.
2. Psychology. I recently read a post over at Schneier’s blog about the psychology of unethical behavior (http://www.schneier.com/blog/archives/2012/05/the_psychology_5.html); the links in the article are fascinating as well. It’s interesting to me to consider that Gartner and Forrester simply may not see the damage that poor data visualization can cause, especially when compared to the overwhelming interest in the subject of visualization and widespread recognition of its merits.

Of course, none of that means liars shouldn’t be held accountable for lies, and I certainly appreciate anyone who calls them out. I’m just pointing out that it’s not really useful to shame a blind man for being blind.

“You might think it is harsh… Perhaps these claims were made in ignorance. If so, when people claim to be experts, they take on a responsibility for knowing the truth.”

If that’s the case, Hanlon’s Razor might not apply. However, this link might still be relevant: http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Hmm. Seems I have a few links today.

Andrew,

People who believe lies and spread them when the truth is clear and readily available are behaving shamefully. This is especially true of people who claim to be experts. Yes, they should be ashamed. Those who live in darkness because they refuse to open their eyes get no sympathy from me for being blind.

The authors of this report, in fact, have no expertise in data visualization. They are BI generalists who haven’t taken the time to learn about data visualization. So, the question is, are they so ignorant that they really believe their claims. It’s hard to imagine that ignorance is the basis for these lies. While they are not experts in data visualization, they have had enough exposure to these products to recognize the chasm that separates the leaders from the wannabes. To believe these lies, they would have to not only be ignorant, but be stupid as well, and that’s unlikely. It is much easier to believe that they reward the big vendors that pay them dearly for consideration than that they’re ignorant of the falseness of their claims.

Interesting how little difference there is between the ADV report by Forrester and the The Forrester Waveâ„¢: Self-Service Business Intelligence Platforms, Q2 2012 released a mere month prior.

It would be unfortunate if customers paid for both, or either. Link can be found on SAS’s website:

http://www.sas.com/news/analysts/Forrester_Wave_Advanced_Data_Visualization_Platforms_Q3_2012.pdf

http://www.sas.com/news/analysts/Forrester_Wave_Self_Service_105855_0612.pdf

They´d rather make a buck than be right.

I have just learned that Boris Evelson of Forrester has been challenged via a tweet to respond to this blog post. The first time that I exposed his impoverished understanding of data visualization in this blog, he did respond. The interaction did not go well for him. Since then, he solicited my services as an adviser. You may be surprised to learn that Forrester, according to Evelson, however, never pays for the services of expert advisers. They form partnerships with them in the form of “You scratch my back and I’ll scratch yours.” If you’re willing to read Forrester’s reports in advance of publication and suggest corrections and improvements, they return the favor by citing your research and sending clients your way that need expert consulting support. Those who forge such relationships with Forrester are loathe to speak out against them, so this relationship works to Forrester’s advantage, but definitely not to the advantage of their clients. I told Evelson that I would be happy to assist him in the same way that I would assist any organization that requests my help beyond what I freely offer on this web site, for which I charge a daily consulting fee.

I would appreciate a response from Evelson regarding my assessment of his report, but I realize this isn’t likely. Unlike politician’s, most business professionals are loathe to engage in public debate about issues that they don’t understand.

Kyle,

Thanks so much for responding on behalf of Forrester. As you might imagine, now that you’ve opened the door, I have a few questions for you.

Where do you fit in the organization in relation to the authors of this report about advanced data visualization? Do you have any expertise in data visualization?

You’ve challenged my arguments, so let’s begin with that. I have no doubt that you “carefully select the vendors and solutions” that you evaluate. I’ve suggested that you did so in this case, however, not based on an informed awareness of the best data visualization products, but on your relationship with these vendors. I did not say that these vendors directly paid you to be included in this report, but they paid you nonetheless. I, my readers, and the people who have paid for this report would like to know which of the vendors that were considered in your report are not your clients from whom you’ve received payment for services.

You’ve claimed that your research is “exhaustive.” It was perhaps exhausting for your analysts to compile this report due to their ignorance of data visualization, but this report was far from exhaustive. No credible organization of your type would allow a report of this low quality to be produced. Did anyone with expertise in data visualization vet this report? If so, who? You claim that you “interview and speak with thousands of end user organizations and experts” like me. You did not actually speak to thousands of organizations to produce the content of this report, did you? Even if you did, end users do not possess the expertise that is required to rate the merits of advanced data visualization vendors. Regarding experts, you did not speak with me, so I’m interested in knowing which experts you did speak with. No real expert in the field would endorse the contents of this report. I invite you to prove me wrong by naming any real expert who finds this report worthwhile.

Now, on to other issues.

Do you agree that reports produced by Forrester should be written by experts in the field? Do you agree that those who pay for the reports that you produce expect that they were written by experts? If so, the very fact that Forrester has produced a report of such low quality by people who don’t understand data visualization as if it were the product of expertise is an example of your willingness to lie.

Boris Evelson, one of the authors of this report, has admitted to me privately that he lacks expertise in data visualization. He also explained that it is Forrester’s policy to never pay for the services of outside experts. This is an arrogant and self-defeating policy. You charge your clients a great deal for your “expert” services, but when you lack expertise yourself, you refuse to pay for the expertise of others that’s required to produce useful reports. You should either refrain from producing reports regarding matters that you don’t understand or you should acquire the services of those who possess the expertise. To do otherwise is a lie and a disservice to your customers.

You acknowledge that I am an expert in the field of data visualization, especially as it relates to business intelligence. As such, I’m telling you that this report is worse than worthless. It is actually harmful in that it will lead people to select products that don’t work, resulting in great loss. Either I don’t know what I’m talking about or I’m right and you’re guilty of harm. Which is it?

You refer to this report as the result of research, but it does not qualify as actual research. You solicited opinions from people who know relatively little about data visualization. At best, you have reporrted opinions. This report was not presented as the findings of an opinion poll, however. It was presented as expert guidance regarding the characteristics of advanced data visualization, including an expert evaluation of vendors. Of the facts that you’ve reported, those that are accurate are common knowledge in the field of data visualization. The rest is either outright error or fabrications to falsely elevate the merits of vendors that are your clients.

Useful evaluations are based on appropriate criteria. Your report was not. Please explain why your evaluations were based 50% on criteria that had nothing to do with the data visualization capabilities of the products. I believe that I know why. You were looking for ways to improve the scores of vendors that are currently doing little that is worthwhile in the field of data visualization. Please explain how 5 out of the 14 vendors that were considered scored higher than Tableau in their “commitment” to advanced data visualization. In my opinion, this is an example of a lie. The authors of this report, however ignorant of data visualization, cannot really believe that Microsoft, IBM, Information Builders, SAP, and SAS are more committed to advanced data visualization than Tableau. Of these, only SAS has demonstrated a substantial commitment to advanced data visualization and their commitment, however substantial, is not greater than Tableau’s. Also, please explain why you believe that 7 or the 14 vendors exhibit superior product direction in the field of data visualization than Tableau. Do you expect us to believe that you were not showing favor to your largest clients?

Forrester might produce useful reports about other technologies, but shame on you for producing such drivel about data visualization and having the nerve to charge $2,500 for it. If Forrester as an organization has integrity and the work that you’ve produced about data visualization is an embarrassing exception, then you should own up to your mistake and return the money that anyone has spent on this report. Do that, and you will begin to earn my respect.

Well done Stephen!!
Great questions, well thought out, and I am looking forward to the response…

Kyle,

Will we be hearing from you?

If you have anything to say in response to the facts that I’ve presented and the questions that I’ve asked, this is your opportunity to either justify the merits of Forrester’s advanced data visualization report or to acknowledge its deficiency. Admitting the failure of this one report is certainly not an indictment against Forrester in general. I’m inclined to believe that Forrester does excellent work in some areas but that data visualization is unknown territory that you should ignore until you’ve either developed or hired the required expertise. By missing this opportunity to acknowledge the poor quality of this one report, you will cast doubt on the integrity of Forrester’s work in general. Why should we believe that Forrester is an objective and qualified reviewer of other technologies when you defend the merits of this particular report, which is so clearly flawed? When an expert in the field shows your work to be juvenile, ill-conceived, error-filled, misleading, and biased in a self-serving manner that contradicts the objectivity that you claim, a response is in order.

Kyle,

You do know, don’t you, that standing behind this report when you don’t personally have the means to evaluate it, is meaningless. Also, standing behind your employees when they produce poor work is not a position that your customers will appreciate.

It’s sad that you have nothing to say in response to my specific observations and questions. It is clear that you have chosen this path because you cannot defend this report. If you measure your success by how well you empower decision makers to own their decisions and reduce their reliance on consultants you’ve failed miserably in this instance. You have given decision makers erroneous, misleading, and harmful advice. This is not a matter of a difference of opinions among experts. This is a matter of Forrester having no expertise whatsoever in this particular field and no substance whatsoever to back its claims.

It is Forrester that has taken the “who has the best widget” approach to evaluating data visualization products. You used a checklist of features in some cases and subjective criteria in others (e.g., best product direction), and reduced your evaluation to counting checks in boxes, wild guesses based on naiveté, and partiality in favor of vendors that pay you. You have provided no evidence that Forrester has any expertise whatsoever in data visualization. The reason is clear: you have none. You have provided no evidence to counter my observation that you designed and conducted this so-called research to serve the interests of those vendors that are your customers. The reason is clear: my observations are true. You have failed to name a single expert in data visualization that you consulted when preparing this report. The reason is clear: none were consulted.

What I find particularly interesting in your response is the fact that you don’t recognize the logical error in the basic premise of your work: “We empower decision makers to own their decisions, and reduce their reliance on consultants.” You are a consulting organization. Your entire business model is based on a reliance of customers on your consultants to inform their decisions. You fundamentally fail your customers by providing unreliable advice that will cost them dearly. Your business model — rely on Forrester’s consultants so you don’t need the services of other consultants — is only viable if you provide useful and comprehensive advice. This little interaction of ours demonstrates how little you actually care about your customers. You can ignore people like me who reveal errors in your work, but you do so to your own eventual peril. People aren’t as stupid as you seem to believe.

You have a choice: either stick to what you know and work hard to provide good advice, or stick your fingers into every pocket that is bulging with opportunity (data visualization, in this case) without preparation or skill and hope that no one notices your sleight of hand.

Several days have now passed without further response from Forrester. They must have nothing more to say, which is disappointing, because they’ve said what amounts to nothing. They chose the approach of a politician whose hand has just been caught in the till. Hollow words, bereft of meaning.

What Forrester does (and perhaps other organizations of their type as well) amounts to extortion. Technology vendors pay them for protection. Like the Mob, you either pay up or you suffer. By paying up, you get to be included favorably in Forrester’s so-called research reports. The more you pay, the more slack you’re given for your shortcomings. By failing to pay, you get left out as if you don’t exist, even if you’re better than the vendors that paid. This is a racket.

The only way to put an end to this is for the vendors – lots of them – to refuse to pay. In my opinion, this is precisely what good vendors should do to act responsibly. By paying Forrester, they’re endorsing Forrester’s work, which in the case of this report about advanced data visualization is doing harm by giving bad advice to customers who deserve better and are certainly paying for better.

If you’re a customer that pays Forrester for advice, stop assuming that their advice is good just because you paid dearly for it. Demand that they either produce work that is grounded in expertise or stop paying it. Take the questions that I’ve asked them about the advanced data visualization report, which they’ve refused to answer, and ask Forrester these questions yourselves directly. If they deign to respond and you want to pass their answers through a shit-detector, send them to me and I’ll happily oblige.

BI software companies that have no right to be called leaders in advanced data visualization are now proudly displaying their award from Forrester in the media and to their customers. I saw another press release today that made me sick. This galls me. Why? Because it’s wrong and it’s harmful. I work directly with people who struggle every day to squeeze something useful from bad data visualization products. It hurts to see this. How many of them are stuck with dysfunctional tools because their CIO trusted Forrester or some other misinformed technology pundit? We have a voice; let’s use it. We have a choice; let’s make it count.

As a researcher who has worked for some of these “best practice” and “benchmarking” firms, all I can say is great job Stephen. More light should be shed on all of these firms that claim to be experts and then push shallow and often harmful “objective research”.

You have hit a very important point on the head – many so called “analysts” or “experts” have slim to no experience in their field of expertise. Furthermore – and as a result – the “research” they produce has virtually no value.

I work for a consulting and research firm where all professionals have at least 10 years of direct, applicable, experience in their fields. As a result, I believe that our research (which, by the ways, IS truly objective), is a far superior product.

It is important not to throw the proverbial baby out with the bath water as not all “research” falls into this this category. Unfortunately, a lot does.

Poor research like this from Forrester has severe and ongoing consequences. CIO magazine has now repeated this list of advanced data visualization vendors as if it were gospel, and they reordered the 10 vendors in a way that placed the best at the end of the list, suggesting that they’re low on the totem pole. This is how lies are propagated.