Bad Science and the Fear of “Methodological Terrorism”

A few days ago a data visualization developer friend of mine, Robert Monfera, sent me a link to a blog post titled “On methodological terrorism” by a thoughtful statistician named Robert Grant. Grant lays out an intelligent and entertaining case for speaking out against methodological flaws in scientific research—a practice that some on the receiving end characterize as “methodological terrorism.” He, I, and a growing number of others are speaking out to expose bad methodological practices in scientific research, not because we enjoy conflict, and certainly not because we’re assholes, but because bad science is always a waste of time and resources and it sometimes causes harm.

Grant wrote his blog post, in part, as a response to an article in a magazine of the Association for Psychological Science written by Susan Fiske, who decried the venomous nature of critiques and coined the term “methodological terrorism,” along with a few other bombastic terms, including “destructo-critics, “data police,” and “vigilante critique.” Fiske seems to be describing people such as Andrew Gelman, Ben Goldacre, Gerd Gigerenzer, and John Ioannides, whose work and integrity I greatly admire.

Here are a few of my favorite excerpts from Grant’s blog post:

If we view it as our civic duty to promote good research, it is also our civic duty not to tolerate bad research.

There is a corrupt system which you are obliged to end, and you will have to act outside the system to do so. Not by blowing up their offices…but by confronting their work when it is wrong, in the best scientific tradition, and refusing to go away until it is fixed.

Fiske [the scientist who coined the term “methodological terrorism”] said “it’s careers that are getting broken”; yes, that is precisely the objective. Acting out of ignorance, then seeing the light and fixing the problem is one thing, fighting not to change is another, and someone who refuses to learn and improve is not a scientist…

We should scare them all right, but in a thoroughly scientific way. It needs to be clear that nobody’s blunders are safe from being called out. We need to go after anyone and everyone, not just the big names.

The following excerpt from Grant’s blog post describes the circle-the-wagons resistance that the information visualization research community has exhibited in responses to my critiques:

The current system of a small number of the same people approving funding for studies, doing them, and editing the journals where they are published is arguably corrupt. The subject experts who run it benefit so much from it that they certainly don’t allow dissenting voices on their patch, and, unable to control self-publication on blogs and social media, react forcefully. Journals and conferences are used as an organ of repression, and we should focus on influencing them and not allowing them to be a refuge for irresponsible conduct.

Grant points out that researchers who cry foul in response to critiques of their work or the work of their communities tend to characterize those critiques as crossing the line into meanness. It is ironic that they often oppose these critiques through truly mean attempts at character assassination (“kill the messenger”) rather than rational discourse, which demonstrates the weakness of their position. Borrowing an analogy that was used during a keynote presentation last year by the designer Mike Monteiro, Grant likens the work of critics to that of dentists:

Now, consider the dentist. You pay them to tell it like it is. If your molar is rotten and has to come out, you want to hear it and have some straight-talking advice on what to do about it. You don’t enjoy hearing the news, but better now than later in agony. That is the service they provide — to tell you the facts, not to be your friend. We need to stop being friends of subject experts and start being their dentists instead.

Take a few minutes to read Grant’s blog post in full. If you care about science, you’ll find it worthwhile.

Take care,

Signature

2 Comments on “Bad Science and the Fear of “Methodological Terrorism””


By Brad. October 25th, 2016 at 4:00 pm

Steven –
I often find myself playing the role of “bad cop” – and have to admit it can get exhausting. It is especially exhausting when dealing with people who tend to nod in agreement on anything presented that looks ‘pretty’ and ‘glitzy’, regardless if it is accurate or provide value or insight.
Nice to know there are other people out there who might even call me a ‘good cop’.

By Robert Monfera. October 28th, 2016 at 2:24 am

The problems described in the article, relating to research and analytics methods that purport to support studies, generalize to decision support. Poll graphs, is a form of decision support – it gives information to voters, often with unveiled bias, including picking surveys and timing them to show something favorable, and even then, distorting it: https://www.washingtonpost.com/graphics/politics/2016-election/trump-charts/?utm_campaign=buffer&utm_content=buffer849f9&utm_medium=social&utm_source=twitter.com

In some areas – games, entertainment, sports fandomship – imprecise visualization doesn’t do much harm, and often, techniques such as projections, visual primitives and grammar from data visualization can be used for artistic purposes, or as illustrations, growing interest in STEM, or whatever topic.

In business and finance, people, companies or countries stand to lose if good practices aren’t followed, and it has impact in personal finance, insurance, financial stability, pensions, education and public finance.

And in medicine, pharmacological research, public policy or the justice system, misleading decision support – including visualization – is a matter of life and death, or biased justice, or bad military choices, or impaired education, opportunity and gender pay fairness.

When some people are committed to a specific way of doing things, while others say “well there’s many ways of doing it, they’re fine depending on the use, we could go one way or another”, then the former group of people may hold values and context in mind that the latter group doesn’t focus on. Discussing topics would be easier if there were clarity on what range we cover on the continuum of artistic/playful/entertaining/experimental on one end, and life, death and societal decisions on the other.

Leave a Reply