The Incentives Are All Wrong

Perhaps you’ve heard that a large proportion of published scientific research findings are false. If anyone qualified took the time to validate them, up to half or more could be exposed as erroneous at the time of publication. Most of the errors are due to bad research methodology. Most of the bad research methodology is either due to insufficient training, incentives that are in conflict with good science, or both. The primary area of training that’s lacking is in statistics. Researchers are incentivized to be productive and innovative above all else. It’s all about getting published and then getting cited. It isn’t about getting it right.

A recent commentary in the Lancet, medicine’s foremost journal, titled “Offline: What is medicine’s 5 sigma?” (Volume 385, April 11, 2015) by Editor-in-Chief Richard Horton, describes his concern that “something has gone fundamentally wrong with one of our greatest human creations.”

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.

If poorly conducted research exists to this extent in mature fields of study such as medicine and physics, it isn’t surprising that it’s even more prevalent in the fledgling field of information visualization. Horton ended his commentary with some good news and some bad news:

The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.

So far in the realm information visualization research, only the bad news applies.

Take care,


7 Comments on “The Incentives Are All Wrong”

By Dale Lehman. July 8th, 2015 at 6:06 am

This hits most of my nerves. I have spent much of my career trying to replicate other people’s work – almost never being able to do so. In the rare case where I am able to replicate, I discover errors, many of which are major. In my area particularly, economics, the incentives are indeed perverse. Rarely is anybody penalized for making mistakes (intentional or not) and rare is the case where replication is even possible – the data is usually not provided. Even journals with policies that data be made publicly available have exceptions – and from my experience, half or more of the published manuscripts will claim that their data is proprietary and so it is not available.

So, yes, it is clear that the incentives are poor. But it will not change unless people with influence (not me) want it to change. If academic departments started caring about the quality of people’s work rather than the quantity, it could change. If journal editors insisted that data be made available, it would change. If journal editors sought out replication studies, it would change. But editors get to their positions by playing the right academic games, not by bucking the system, so it is unlikely that they will take the lead on changing that system. So, too, for people with established reputations. What do they have to gain by changing the system that has worked well for themselves?

There is progress – much more so in medicine (where it is intensely controversial) and science than in social science. In the latter, standards are particularly poor. Yet things could change. If policy makers (regulators, legislators, etc.) simply said they will ignore research that does not make its data publicly available, then much of the incentive for that research would change. It has always puzzled me that regulators do not seem to care – in fact, they appear to prefer that studies are not replicated. They simply choose the think tank that says what they want to hear – as do their opponents. And, this results in “science” that is neither objective or of high quality.

I have a substantial background in expert testimony and that setting is far superior to academic research (although it is far far from perfect). At least, in the adversarial setting of the courtroom, your data can be obtained and cross examination improves the incentives. In academia, conferences with discussants do not come close to that level of scrutiny. I am far along in my career and have done well in both circles, but it upsets me greatly to see how slowly things change (if they change at all – arguably things are getting worse). As data sets become larger and techniques become more complex, it is easy to speculate that the problems will only get worse.

Visualization is but the tip of this iceberg – but the most visible part, and perhaps the most influential. The visualization techniques are becoming more robust, and examples of good visualizations and analysis (such as what you, Stephen, have been providing for years) are increasingly available. But visualization lives within the world of research – both academic and commercial. The exchange over the previous blog entry is a perfect example of how poor the incentives are and how difficult they are to change. The payoff to being dramatic and overstating results is far greater than the payoff to careful and humble work. As everything in life becomes faster and more condensed (just look at the typical movie pace from 50 years ago relative to today), patience for incremental improvements in knowledge is waning, not growing. I fear that I see the world as a glass less than half full.

By Stephen Few. July 8th, 2015 at 8:25 am


Thanks for a thoughtful and articulate extension of my comments. Clearly, you live in the midst of this problem and care a great deal about solving it. Keep speaking out, even when it seems like no one is listening. This concern is growing.

By David Leppik. July 9th, 2015 at 8:10 am

I heard a proposal once to replace calculus with statistics as the default advanced math option for US high school seniors. The argument is that while both are required for many sciences, statistics is more important for public policy, and therefore more important for being an informed citizen.

I have to say, this idea makes a lot of sense to me. Understanding statistics has always been useful in public policy, but today we are awash in hard numbers like never before.

Not that knowledge of statistics will fix the problems both of you have described, but it would elevate the discussion if more voters and politicians understood ideas like systematic bias and “correlation is not causation.”

By Andrew. July 14th, 2015 at 9:23 am

Even with published data there are issues that concern me. Like p-hacking ( and publication bias.

I was actually reading an article about this stuff just the other day. The article describes an authentic scientific study with an actual clinical trial, the goal of which was to practice bad science and see how easy it would be to get it published. Long story short, it was too easy:

By Dale Lehman. July 14th, 2015 at 10:08 am

This authentic “fake” study has now been reported in a number of places. I think it is important to point out that the actual data was not released – the study was “published” (though not in a reputable place) and “publicized” (though via intentional deceptions to the media). Requiring that data be made publicly available will not prevent all such problems. However, if there is a requirement that it be made available, and if people and journal editors (and referees, etc.) are held responsible for their work, then the worst abuses may be avoided. In the absence of these measures, it is all to easy to propagate errors and publish misleading, incorrect, and deceptive work.

By Andrew. July 15th, 2015 at 9:40 am


Of course it was intentional deception – that was the idea! ;)

But yes, I agree 100% that there should be requirements that data be made publicly available.

By WalterT. July 22nd, 2015 at 6:17 am

Hi there. Many years ago, I spent my company’s money and time as a member and eventually chairman of a national standards technical committee. A thankless but necessary task because the participants were either “living democracy” geeks or corporate lobbyists. The former toiling in the fields of simple physics to establish rules to measure this or that phenomena necessary in commerce; the latter urging the laws of physics be tweaked to allow inclusion of their company’s special tool for subject measurements in the proposed standard.

What I took away from this experience was that the two concepts of repeatability and reproduceablity remained very strong barriers to cheating, especially when inconsistent results were subjected to withering scrutiny in the civil courts.

Such comprehensive “glasnost’ costs time & money but that should not prevent a presumption that all data asserting a controversy be made available for all and any to peruse. Some will draw wrong conclusions through errors in principal as well as arithmetical, but legitimate experts ought to be able to bring clarity for the public good. I’m thinking of AGW data specifically. I don’t refuse to believe that mankind has had an effect, I just believe that the Sun has a rather stronger, if less obvious or directly measureable, influence on space and terra weather.
Nice site, thanks.

Leave a Reply