This hits most of my nerves. I have spent much of my career trying to replicate other people’s work – almost never being able to do so. In the rare case where I am able to replicate, I discover errors, many of which are major. In my area particularly, economics, the incentives are indeed perverse. Rarely is anybody penalized for making mistakes (intentional or not) and rare is the case where replication is even possible – the data is usually not provided. Even journals with policies that data be made publicly available have exceptions – and from my experience, half or more of the published manuscripts will claim that their data is proprietary and so it is not available.

So, yes, it is clear that the incentives are poor. But it will not change unless people with influence (not me) want it to change. If academic departments started caring about the quality of people’s work rather than the quantity, it could change. If journal editors insisted that data be made available, it would change. If journal editors sought out replication studies, it would change. But editors get to their positions by playing the right academic games, not by bucking the system, so it is unlikely that they will take the lead on changing that system. So, too, for people with established reputations. What do they have to gain by changing the system that has worked well for themselves?

There is progress – much more so in medicine (where it is intensely controversial) and science than in social science. In the latter, standards are particularly poor. Yet things could change. If policy makers (regulators, legislators, etc.) simply said they will ignore research that does not make its data publicly available, then much of the incentive for that research would change. It has always puzzled me that regulators do not seem to care – in fact, they appear to prefer that studies are not replicated. They simply choose the think tank that says what they want to hear – as do their opponents. And, this results in “science” that is neither objective or of high quality.

I have a substantial background in expert testimony and that setting is far superior to academic research (although it is far far from perfect). At least, in the adversarial setting of the courtroom, your data can be obtained and cross examination improves the incentives. In academia, conferences with discussants do not come close to that level of scrutiny. I am far along in my career and have done well in both circles, but it upsets me greatly to see how slowly things change (if they change at all – arguably things are getting worse). As data sets become larger and techniques become more complex, it is easy to speculate that the problems will only get worse.

Visualization is but the tip of this iceberg – but the most visible part, and perhaps the most influential. The visualization techniques are becoming more robust, and examples of good visualizations and analysis (such as what you, Stephen, have been providing for years) are increasingly available. But visualization lives within the world of research – both academic and commercial. The exchange over the previous blog entry is a perfect example of how poor the incentives are and how difficult they are to change. The payoff to being dramatic and overstating results is far greater than the payoff to careful and humble work. As everything in life becomes faster and more condensed (just look at the typical movie pace from 50 years ago relative to today), patience for incremental improvements in knowledge is waning, not growing. I fear that I see the world as a glass less than half full.

I heard a proposal once to replace calculus with statistics as the default advanced math option for US high school seniors. The argument is that while both are required for many sciences, statistics is more important for public policy, and therefore more important for being an informed citizen.

I have to say, this idea makes a lot of sense to me. Understanding statistics has always been useful in public policy, but today we are awash in hard numbers like never before.

Not that knowledge of statistics will fix the problems both of you have described, but it would elevate the discussion if more voters and politicians understood ideas like systematic bias and “correlation is not causation.”

This authentic “fake” study has now been reported in a number of places. I think it is important to point out that the actual data was not released – the study was “published” (though not in a reputable place) and “publicized” (though via intentional deceptions to the media). Requiring that data be made publicly available will not prevent all such problems. However, if there is a requirement that it be made available, and if people and journal editors (and referees, etc.) are held responsible for their work, then the worst abuses may be avoided. In the absence of these measures, it is all to easy to propagate errors and publish misleading, incorrect, and deceptive work.

Hi there. Many years ago, I spent my company’s money and time as a member and eventually chairman of a national standards technical committee. A thankless but necessary task because the participants were either “living democracy” geeks or corporate lobbyists. The former toiling in the fields of simple physics to establish rules to measure this or that phenomena necessary in commerce; the latter urging the laws of physics be tweaked to allow inclusion of their company’s special tool for subject measurements in the proposed standard.

What I took away from this experience was that the two concepts of repeatability and reproduceablity remained very strong barriers to cheating, especially when inconsistent results were subjected to withering scrutiny in the civil courts.

Such comprehensive “glasnost’ costs time & money but that should not prevent a presumption that all data asserting a controversy be made available for all and any to peruse. Some will draw wrong conclusions through errors in principal as well as arithmetical, but legitimate experts ought to be able to bring clarity for the public good. I’m thinking of AGW data specifically. I don’t refuse to believe that mankind has had an effect, I just believe that the Sun has a rather stronger, if less obvious or directly measureable, influence on space and terra weather.
Nice site, thanks.