We Must Vet Our Data Visualization Advisers with Care

When we need advice in our personal lives, to whom do we turn? To someone we trust, who has our interests at heart and is wise. So why then do we often rely on advisers in our professional lives whose interests are in conflict with our own? If your work involves business intelligence, analytics, data visualization, or the like, from whom do you seek advice about products and services? If you’re like most professionals, you unwittingly seek advice from people and organizations with incentives to sell you something. You either get advice from the vendors themselves, from technology analysts with close ties to those vendors, or from journalists who are secretly compensated by those vendors. That’s not rational, so why do we do it? Usually because it’s convenient and sometimes because we don’t really care if the advice is good or not, for it is our employers, not us, who will suffer the consequences. If we actually care, however, we should do a better job of vetting our advisers.

It should be obvious that we cannot expect objectivity from the vendors themselves. Even when a vendor’s employees post advice from independent websites and claim that their opinions are their own, they remain loyal to their employers. In fact, it’s a great marketing ploy for vendors to have their employees post advice from independent sites rather than from their own. It suggests a level of objectivity that serves the vendor’s interests and multiples their presence on the web. We must also question with similar suspicion the objectivity of consultants and teachers who have built their work around a single product.

What about technology analyst groups, such as Gartner, Forrester, and TDWI, to name a few of the big guys? These organizations fail in many ways to maintain a healthy distance from the very technology vendors that are the subject of their advice. In fact, they are downright cozy with the vendors.

Trustworthy technology advisers go to great pains to maintain objectivity. They are few and far between. To be objective, I believe that advisers should do the following:

  • Disclose all of their relationships with vendors. This is especially true of relationships that involve the exchange of money. If they accept money from vendors, they should willingly disclose the figures upon request.
  • Do not allow vendors to advertise on their websites, in their publications, or at their events.
  • Only accept payments from vendors for professional services specifically rendered to improve the vendor’s products or services. Payments for marketing advice does not qualify.
  • Do not publish content prepared by vendors.

Try to find technology analysts and journalists who follow these guidelines. Even with diligent effort, you won’t find many, because there aren’t many to find.

Try an experiment. If your company subscribes to one of the big technology analyst services (Gartner, etc.), next time they produce a report that scores BI, analytics, or data visualization products, ask them for a copy of the data on which they based those scores, along with the algorithms that processed the data. This is likely done in an Excel spreadsheet, so just ask them for a copy of the file. After making the request, watch them squirm and expect creative excuses. Most likely they’ll say something along these lines: “Our scoring system is based on a sophisticated and proprietary algorithm that we cannot make public because it gives us an edge over the competition.” Bullshit. There is definitely a secret in that spreadsheet that they don’t want to share, but it is not a sophisticated algorithm.

After they refuse to show their work, move on to the following request: “Please give me a list of the vendors that you evaluated along with the amount of money that you have received from each for the last few years.” They won’t give it to you, of course, and they’ll explain that they cannot for reasons of confidentiality. Think about that for a moment. It is no doubt true that they promised to never reveal the money that changed hands between them and the vendors, but shouldn’t this clear conflict of interest be subject to scrutiny? Technology analysts and the vendors that they support are not fans of transparency.

There are a few technology advisers who do good work and do it with integrity. If you want objective and expert advice from someone who is looking out for your interests, be sure to vet your advisers with diligence and care. Question their motives. If it looks like they’re acting as an extension of vendor marketing efforts, they probably are. If, on the other hand, you’re just looking for easy answers, abandon all skepticism and do a quick Google search and then read the advice that receives top ranking. Or, better yet, schedule a call with the analyst group for whose advice you pay dearly in the form of an annual subscription.

Take care,


(Postscript: Yes, I consider myself one of the few data visualization advisers in whom you can trust.)

14 Comments on “We Must Vet Our Data Visualization Advisers with Care”

By Jonathon Carrell. June 24th, 2016 at 1:27 pm

Stephen, your concerns here really aren’t unique to the data visualization field. It is pretty much a universal reality in every aspect of business.

We, as professionals, must be mindful of these sorts of bias and take information provided by vendors, groups, and individuals with a grain (or cup) of salt. That isn’t to say that there is nothing of value to found in articles written or sponsored by vendors, but a certain amount of bias is expected in these cases and must be taken into account when weighing the value (accuracy/objectivity) of the information provided.

What it is most concerning to me is when bias is intentionally masked to mislead the audience into thinking the information provided is vendor neutral or otherwise without bias. I find this type of activity to be deplorable and a dishonest business practice. Vendors/individuals that engage in these practices are unworthy of your time or business.

I don’t expect vendors (or their representatives) to be unbiased … it is natural that they are so, but I do hold them accountable to accept and acknowledge such bias.

By Chris Love. June 24th, 2016 at 2:37 pm

I look forward to the day we see a Few Magic Quadrant or similar rankings.

By Stephen Few. June 24th, 2016 at 3:20 pm

Jonathon — While this problem is not unique to data visualization, its existence in and impact on this field concerns me uniquely, for obvious reasons.

Chris — If I were to measure the effectiveness of data visualization products and display the results in a scatter plot, divided into quadrants, it wouldn’t be magical (no smoke and mirrors would be involved) and the upper right quadrant would be empty, waiting for a courageous vendor to do what’s needed.

By kris erickson. June 27th, 2016 at 11:24 am


What are your thoughts on an ecosystem like R or JavaScript? If various people or groups could submit libraries that produce visualizations, would a single source of bias would be far more diffuse and make the disclosure problem less of an issue?

Secondly what if the visualization tool, the visualization server, and the visualization codebase were all divorced? R (or maybe it’s the company Plot.ly) has a module that allows for a user to post their interactive visualization on Plot.ly. Plot.ly as a company just serves up the visual and isn’t necessarily vested in the tool that creates the visual as long as it can be rendered in Javascript (I assume it’s Javascript based).

By Stephen Few. June 27th, 2016 at 11:40 am


If a company such as Tableau exclusively used R and Javascript to produce its product, its bias in favor of its own product would not be diminished. Do you have reason to believe that a company such as Plot.ly would be less biased towards its own platform and invested in getting people to use it?

By kris erickson. June 27th, 2016 at 12:06 pm

No I do not. Already there are multiple companies offering products that play well with Tableau and thus effectively aligning the different biases. I am only thinking there is a possibility using different vendors for different parts of the process *may* negate some bias.

By Stephen Few. June 27th, 2016 at 12:33 pm


I’m not talking about bias that is built into the tools themselves, which affect their usefulness. Instead, I’m talking about the need for advice about data visualization, analytics, etc., to be as objective as possible. For example, I’m arguing that advice columns (e.g., blogs and articles) that are written by BI software vendors, including any of their employees, should be viewed with suspicion as biased. Even many articles that are written by journalists, whom we would assume were objective, are based almost entirely on a vendor’s marketing materials.

It is indeed helpful at times to work with various tools rather than one only, but this is primarily because doing so will provide a broader scope of functionality, which is beneficial even though it comes with the added complexity of learning multiple tools, which must be taken into account.

By Hicham Bou Habib. June 27th, 2016 at 10:55 pm

Dear Stephen:

You are an honorable BS detector. To me you are in the same league as my two other gurus Nassim Nicholas Taleb and Edward Tufte.

If just each industry has a dozen replica of You, the world would be a better place.

The healthcare industry is a case in point. Study after study shows the links between the money the pharma industry spends on entertaining doctors, and how often those doctors prescribe the drugs of the companies that entertain them.

I know you’re not into social media, but many of the so called dataviz experts–with thousands of Twitter followers–have their podcasts and websites sponsored by the vendors and dataviz tools.


By Jonathon Carrell. June 28th, 2016 at 8:22 am

–I know you’re not into social media, but many of the so called dataviz experts–with thousands of Twitter followers–have their podcasts and websites sponsored by the vendors and dataviz tools.–

While this isn’t surprising, it is disturbing … especially if such sponsorship isn’t made public. Corporate sponsorship voids the validity of any claims of impartiality concerning the sponsor or their competitors.


By Stephen Few. June 28th, 2016 at 10:50 am

Hicham and Jonathon,

For several years I accepted invitations from BI software vendors to speak at their events or in webcasts and to write white papers that they sponsored. I only accepted those invitations, however, with the clear understanding that in the content that I delivered I would not speak about their products and that I certainly would not endorse them. I also made it clear that my content would be educational in nature, never promotional, and that I would not constrain it in any way to avoid best practices that their products did not or could not support. The BI vendors that I did work of this type for are all listed on as clients on my website. Even though I never compromised my content in any way to accommodate the interests of these vendors, I eventually decided to end this practice because I wanted to avoid even the appearance of potential bias in their favor. The only work that I do now for BI software vendors is to provide consulting services to help them improve their products if they invite me to do so and give me a good reason to believe that they genuinely wish to improve. My current policy seems to be the best policy for someone in my position. I recommend it to all data visualizatoin consultants and teachers who request my opinion.

By Jonathon Carrell. June 28th, 2016 at 11:29 am


I’m familiar with many of the white papers you’ve penned for various vendors, and if you had been championing their products or contrasting them to their competitors your credibility on the topic would have clearly taken a hit. However, as you have pointed out, I’ve never seen an article or book in which you actively promote any product. At most, you’ve occasionally acknowledged what tool you used to create specific examples (perfectly acceptable). I agree that avoiding corporate sponsorship altogether is a prudent course, but I can also see why so many people are so eager to hop on the wagon. If you’re relatively new to the industry, writing an article for a vendor gives them a platform and microphone … it may also give them a false sense of legitimacy.

You sir, do not fall into this category.

Sidebar: While there are a few products that I feel have furthered the cause of data visualization, I have yet to find one that can be promoted as a complete solution. A thorough testing and review of any of them would reveal numerous shortcomings in each. Sometimes I feel that vendors are more interested in kowtowing to “market pressure” and suffer from an acute case of featuritis while neglecting or outright ignoring existing issues that need to be addressed.

By Stephen Few. June 28th, 2016 at 11:45 am


Your concern with the motives of vendors is legitimate. The only data visualization product I’m aware of that hasn’t bowed to “market pressure” to add useless features and harmful effects is JMP from SAS. The product’s creator, John Sall, co-founder of SAS, still guides its development with a firm hand that is not tempted by this kind of nonsense. Unfortunately, as a sophisticated statistician, John has insufficient appreciation for effective interface design, which limits JMP’s reach to other statisticians who, like him, are focused on statistical functionality and little else.

By Dale Lehman. July 6th, 2016 at 6:26 am

Interesting that you should mention JMP. I have been using and teaching it in all of my courses for 20 years now and agree with what you have said. What continues to puzzle me is why it is so less known than SPSS, STATA, or R. They certainly advertise and continue to invest in the product. It is known within the statistics community, but I can honestly say that every school I have taught at in the past 2 decades (at least 6 that I can count) have been completely unaware that JMP exists. I am curious as to why you think this might be the case.

By Stephen Few. July 6th, 2016 at 7:55 am


My guess is that those schools are familiar with SAS, but not with JMP in particular. If this is the case, it is probably because JMP has always functioned autonomously from the rest of SAS. As a result, JMP has never been a product that the SAS salesforce understood or promoted. This autonomy has made it possible for John Sall and his team to tightly control the quality of the product, but it has limited their marketing and sales reach, which they’ve had to handle primarily on their own.

Leave a Reply