2014: A Year to Surpass
Perhaps you’ve noticed that I didn’t write a year-in-review blog post about 2014, extolling the wonderful progress that we made and predicting the even-more-wonderful breakthroughs that we’ll make in 2015. That’s because, in the field of data sensemaking and presentation in general and data visualization in particular, we didn’t make any noticeable progress last year, despite grand claims by vendors and so-called thought leaders in the field. Since the advent of the computer (and before that the printing press, and before that writing, and before that language), data has always been BIG, and Data Science has existed at least since the time of Kepler. Something did happen last year that is noteworthy, however, but it isn’t praiseworthy: many organizations around the world invested heavily in information technologies that they either don’t need or don’t have the skills to use.
I know that during the last year many skilled data sensemakers used their talents to find important signals in data that made a difference to their organizations. Smart, dedicated, and properly skilled people will always manage to do good work, despite the limitations of their tools and the naiveté of their organizations. I don’t mean to diminish these small pockets of progress in the least. I just want data sensemaking progress to become more widespread, less of an exception to the norm.
Data sensemaking is hard work. It involves intelligence, discipline, and skill. What organizations must do to use data more effectively doesn’t come in a magical product and cannot be expressed as a marketing campaign with a catchy name, such as Big Data or Data Science.
Dammit! This is not the answer that people want to hear. We’re lazy. We want the world to be served up as a McDonald’s Happy Meal. We want answers at the click of a button. The problem with these expectations, however, is not only that they’re unrealistic, but also that they describe a world that only idiots could endure. Using and developing our brains is what we evolved to do better than any other animal. Learning can be ecstatic.
Most of you who read this blog already know this. I’m preaching to the choir, I suppose, but I keep hoping that, with enough time and effort, the word will spread. A better world can only be built on better decisions. Better decisions can only be made with better understanding. Better understanding can only be achieved by thoughtfully and skillfully sifting through information about the world. Isn’t it time that we abandoned our magical thinking and got to work?
Take care,
16 Comments on “2014: A Year to Surpass”
I think the biggest step forward in terms of data analysis for 2014 was the maturing of the Apache Spark project. I look forward to using it in 2015 and getting to work :)
Jack,
Why does the maturing of the Apache Spark project strike you as a big step forward? What has it enabled you to do that you couldn’t do before? In what manner does it do this better than other technologies? And, for the sake of full disclosure, for whom do you work?
Steve, maybe we don’t need “wonderful breakthroughs” or even “noticeable progress”. Maybe we just need some time to breathe and let the lay user become more aware of what data sensemaking is all about. There are more conferences, more training, more experts not complaining about lack of work, more blogs, more tools, more papers, more social media accounts discussing data visualization. Maybe we need to reach a critical mass before the next step, and organizations making stupid buying decisions is part of the learning process. Sometimes the future comes quietly.
Yeah, I’m reading Voltaire’s Candide.
“We want the world to be served up as a McDonald’s Happy Meal. We want answers at the click of a button.”
Hear, hear – the “Fix my business” button that too many “lazy” executives imagine analytics, of any persuasion, will deliver by magic.
It really *is* time to abandon magical thinking and get to work – thanks, Stephen, for calling it like it is.
Your perspectives have always been insightful, delightful and inspiration Stephen! I truly understand your frustration with the promise of one-click answers with shining interfaces while we still do not have the means to truly grasp what our data is about and what it may communicate with us. I’m also tired with more and more data visualization and analytics “playgrounds” which create more means to follow bad practices than follow the fewer, effectively working solutions. Do we need more pie charts, tag clouds, or *novel* yet weird visual mappings that demonstrate no purpose at all? And, when will people stop creating stacked charts with 20 categories, treemaps for non-categorical data, overlapping spaghetti designs, and countless more? Don’t we already have a better understanding of what works good when, and why do we keep re-discovering and forgetting the same core issues? It can be discouraging to see the organizational and individual leaders in the field push forward so many ineffective designs, and an increase of interest in beauty rather than utility when it comes to understanding data. I think part of the blame is on those who push forward this fancier bigger visual designs as a marketing strategy, and tools that really do not understand what the people really need to achieve and what the data is actually about. Creativity in designing new visualizations is not that important (or uncommon), yet creativity in understanding data actually is. I believe that’s where we need to focus on in 2015 and beyond.
Adil
Jorge,
I’m definitely not looking for “wonderful breakthroughs,” but “noticeable progress” would be nice. The fact that there are “more conferences,” etc., would only be helpful if those conferences, courses, experts, blogs, tools, and papers were promoting and teaching effective principles and practices, which is sadly not the case. Most are just creating confusion. I have probably turned down 20 invitations to speak at such events in the last three years, mostly out of disgust for their inability of unwillingness to distinguish good content and speakers from bad.
In response to your “Candide” reference,” let me share an appropriate quotation: “Optimism,” said Cacambo, “What is that?” “Alas!” replied Candide, “It is the obstinacy of maintaining that everything is best when it is worst.”
Whether quietly or with a bang, the future comes without fail. The question is, “What future?” We have a chance to shape it. We have a responsibility to shape it. If we don’t, the future that comes won’t be the future we desire.
We are just transitioning to a new BI platform. Works fine for delivering reports and so on, but visualization options are terrible – I’m trying to make the best of it…?
Anyway, I was in a training session, and my instructor mentioned your name, saying “yeah, I don’t really agree with any of the crazy stuff that guy Stephen Few says”. I asked her what was so crazy about it. “He says dashboards should be all black and white, and everything has to be bar charts.” I don’t usually like to speak for other people, but in this case I had to explain to her that wasn’t your opinion at all (right?).
So I guess one of the difficulties in getting the world to make better decisions is getting them to just listen to what the hell you’re actually saying in the first place.
Andrew,
As you can imagine, the situation that you’ve described makes me nuts. The “straw man” argument is used against me frequently. When people cannot attack my position legitimately, they erect a straw man (an inaccurate and obviously flawed version of my position) and then proceed to tear it down. The straw man argument is common practice among politicians and anyone else who cannot make their cases rationally.
It’s shocking that so many of the people who claim expertise in business intelligence and analytics cannot make a rational argument. Basic critical thinking skills are required for work in these fields, yet even some of the so-called thought leaders lack these basic skills. That’s sad.
Stephen,
it’s really strange to me knowing that someone used the “straw man” argument against your excellent work. Probably they were trying to sell some product that was supposed to produce Happy Meals.
Every single time (literally) I shown examples with real data based on Stephen Few work, everybody understood why the “fancy” design wasn’t so useful.
Thanks for your wonderful contribution to a future like we desire.
@Stephen
I’m not so sure it was a deliberate straw-man – more likely she just had no idea what she was talking about, in this case. In fact, she’s probably never even examined your position herself, rather her shortsighted opinion was informed by what others (who perhaps WERE deliberately misquoting you) might have said.
That… probably doesn’t reassure you. Sorry.
But know that there are some of us out there who care more about the decisions being made from data, than we do about the fancy whizbang tools and fashions that dazzle our thoughtless PHBs and pad our resumes with Big Words and initialisms. Some of us aren’t fooled by frivolous things like “new” industry trends, nonsensical visualizations, or triumphs over strawmen. Some of us just want to make sense of data, and find clarity in noise.
And we appreciate your work.
Andrew,
Am I correct in guessing that the instructor wrote a book related to this topic in the “for dummies” series? If so, she is definitely familiar with my work and constructed the straw man intentionally. If not, if you send me her name privately, I’ll send her a copy of Information Dashboard Design to make sure that she is informed.
Jack:
I agree about Apache Spark. It means that if I ever move into the realm of “Big Data” I’ll be able to reuse the exact same code I use in my world of “Small But Statistically Significant Data.” (SBSS data?)
Stephen:
The trend, in terms of data analysis and interpretation, seems to not be towards greater or less idiocy, but towards more accessibility. Which means a lot more idiots along with a few more geniuses. There’s always been snake oil (it used to really be snake oil.) There’s always been wishful thinking. (GIGO and DWIM date back to the Analytical Engine.) I appreciate your crusade against both of them. It’s an important fight, and one where there can be progress, but never a conclusive end.
Corporations, in my experience, have always been swayed by silver bullets. There’s a lot of dysfunction, and one reason they don’t invest in learning the truth may be that the folks at the top are afraid of what they might learn. Or they might have to explicitly choose between long-term growth and quarterly results.
In my own little world of customer satisfaction I’ve seen improvements. Companies used to pay lip service to caring about customers. Now more of them are actually spending money to measure and improve it. The next step is for them to actually get good at improving it.
But I suspect that the reason for the improvement is because they (and their vendors) aren’t the only ones measuring it. You can get a rough measurement of any company’s customer satisfaction just by googling “X rocks” vs. “X sucks.” It’s the power of the uninvited second opinion.
Which brings me back to Apache Spark. Tools which lower the bar for good analytics are awesome, even if they lower the bar for sloppy (but better informed) analysis.
David,
I don’t think I’ve ever claimed that idiocy is on the rise as a proportion of total human behavior. That would be hard to measure. I have claimed, however, that idiocy is in full bloom and we should oppose it. I find it interesting that you believe corporations have improved in customer service. I’ve seen no data to back this claim and my own experience suggests the opposite is true: more lip service but less actual customer service. I’m afraid that Googling “X rocks” or “X sucks” is not a valid way to determine improvements in customer service, even though many advocates of Big Data would have you believe that it is.
Regarding Apache Spark, which I have no opinion of one way or the other, perhaps you could provide me and my readers some answers that Jack Golding (the fellow who mentioned it initially) has failed to provide. What has Apache Spark enabled you to do that you couldn’t do before? In what manner does it do this better than other technologies?
Thanks
I think we are talking past each other a bit here. I am not well conversant in big data technologies (such as Apache Spark) and have the same initial reaction as you had, Stephen. However, a little searching reveals that Apache Spark may provide much enhanced capabilities to quickly and efficiently analyze (for example, running a logistic regression) on massive datasets. For people that unquestioningly find such analysis valuable, Spark may indeed be a huge step forward.
However, I think your point, Stephen, is that the need and value of conducting the massive data analysis that this permits has not been demonstrated. Certainly we can all agree that there are valuable uses of being able to analyze massive amounts of data – if they are available and if they are meaningful. But it is the very presumption that massive data is inherently meaningful that is really at issue here (in terms of whether we might characterize Spark as a “big step forward.”
To put this more concretely: suppose Apache Spark allows us to build predictive models of retail sales data based on extensive use of twitter feeds, blog comments, etc. These are things we can do today, but let’s suppose that Spark allows us to use order of magnitudes more of this data than we could incorporate before. Is that a “big step forward?” I am skeptical, without more context concerning the “data sensemaking” that Stephen is referring to. Jack Golding’s initial comment sounds like an automatic response that “of course it is a big step forward” because I can do even more of what I could do before. But it begs the question of whether what you could do before was really worth doing.
For me, that is the disconnect. The technologies to do more and bigger data analysis keep “improving” but the rate of improvement seems to vastly exceed our ability to make sense out of the data. It is hard to see how an increase in these technological capabilities, even a BIG increase, constitute a real contribution to data sensemaking without some corresponding improvement in our ability to make sense of those technologies. It is hard for me to see how Apache Spark does anything in that regard.
@Stephen: “Am I correct in guessing that the instructor wrote a book related to this topic in the ‘for dummies’ series?”
No, but she fits the target audience. (ba dum tsh)
But seriously, much of the class was adequately disappointing.
Actually, black and white barcharts would probably make for more effective visualizations than most of the crap I see prople produce.