Thanks for taking the time to read my thoughts about Visual Business
Intelligence. This blog provides me (and others on occasion) with a venue for ideas and opinions
that are either too urgent to wait for a full-blown article or too
limited in length, scope, or development to require the larger venue.
For a selection of articles, white papers, and books, please visit
June 8th, 2015
Data sensemaking requires skill augmented by good technologies. Even though data sensemaking skills can be developed by almost anyone, they can only be acquired through sustained effort to learn the relevant concepts, principles, and practices, which takes time. This hard fact isn’t as appealing as the fantasy that technologies can turn us into data analysts over night. It is largely because technology vendors have been selling us this myth that we have been trapped in the “data age” but have not yet managed to enter the “information age.” For 35 years I’ve been involved in decision support, data warehousing, business intelligence, analytics, data science—call it what you will—and I find it appalling that, despite all the hype about the information age, we’ve made little progress in deriving value from data. Until we accept the fact that data sensemaking and the decisions that it informs cannot succeed without skill, we’ll remain stuck. Besides skill augmented by good technologies, there are two other prerequisites for successful data sensemaking that are usually ignored but deserve mention: time and attention.
Only those data analysts who are given time to explore and analyze data thoughtfully and thoroughly are consistently successful. Most of the great data sensemaking discoveries that we hear about are the result of work by those rare data analysts whose organizations have given them time to do their jobs well. In contrast, most data analysts are either churning out responses—usually in the form of reports—to a long list of “urgent” requests at a breakneck pace or have other jobs that take most of their time, so they squeeze their data sensemaking activities in here and there whenever they can. This is the norm because few organizations have realized that getting real value from data doesn’t just happen “techno-magically,” as they’ve been led by vendors to expect. It takes time to learn and develop data sensemaking skills and it continues to take time to apply those skills each day. This is because data sensemaking involves analytical thinking and analytical thinking takes time. Technologies can assist by doing fast calculations and other forms of data processing, but the thinking that’s required is slow.
Effective data sensemaking also involves attention. This is one of the requirements for rich thinking that has become more difficult to achieve in recent years. Our lives have become increasingly disrupted by the constant demands of the “persistently connected” technologies that we’ve adopted. For how many minutes can you become lost in concentration without being pulled out of your intellectual reverie by a beep, ding, ringtone, vibration, or the sudden appearance of an alert on your screen? This is not what’s usually meant by disruptive technologies and it certainly isn’t a desired effect. Whenever our attention is pulled away from a data sensemaking task, it takes time and effort to get it back, and much can be lost in the meantime.
Data analysts need an environment that supports concentration without interruption for extended periods of time. The open floor plans that many organizations have been experimenting with to promote collaboration (and, let’s face it, to also save money) are perhaps appropriate for some jobs, but not for data sensemaking. Most of us need to shut the door, turn off all possible sources of interruption, sit in a comfortable chair, and think attentively for long periods of time. I never had a chance to meet the celebrated Princeton statistician John Tukey before he died, but I’ve heard from friends who did that when he took on a consulting job for a client, he would begin with a meeting to discuss the problem and he would then cloister himself in his hotel room for a couple of days before emerging with a solution. Imagine Tukey trying to navigate the disruptive environment of the modern workplace. If he had, perhaps we would have never heard of him. Perhaps the box plot and his many other analytical inventions would not have graced our world. He knew that data sensemaking required attention and he structured his environment to provide it. We must do the same, which means that our employers must recognize the need and support it.
I recently gave a keynote presentation at the University of Cincinnati’s Analytics Summit, and when I mentioned this rarely addressed need for time and attention, you should have seen the heads nodding appreciatively throughout the ballroom. Their eyes brimmed with gratitude for my recognition that the conditions of their work do not match the professed commitment of their organizations to analytics. Claiming to embrace analytics is a far cry from the investment that must be made to fulfill this claim.
Is your organization analytically savvy? Does it exhibit a culture of analysis? If your answer is “Yes” but your organization doesn’t give you adequate time and a work environment fit for focused attention, you’re setting the bar too low. Imagine how much better a data sensemaker you could be with more time and attention.
June 1st, 2015
My new book Signal is finally available for purchase from booksellers, including Amazon. Let me help you decide if it’s a book that you’ll find helpful.
Two years ago, after writing the second edition of Information Dashboard Design, I began asking “What next?” I spent considerable time trying to figure out what people who have already read my existing books most need as the next stage of their data sensemaking development. I noticed that many people spent their time chasing data that didn’t matter, including random variation that meant nothing and couldn’t be changed no matter how great the effort. They spent their days exploring, dissecting, and publishing noise.
This problem is actually growing. In this day of so-called Big Data, organizations are scrambling to implement new software and hardware to increase the amount of data that they collect and store. In so doing they are unwittingly making it harder to find the needles of useful information in the rapidly growing mounds of hay. If you don’t know how to differentiate signals from noise, adding more noise only makes matters worse.
When we rely on data for decision making, how do we tell what qualifies as a signal and what is merely noise? In and of itself, data is neither. It is merely a collection of facts. When a fact is true, useful, and deserves a response, only then is it a signal. When it isn’t, it’s noise. It’s that simple.
In Signal, I provide straightforward and practical instruction in everyday signal detection. Using data visualization methods, I teach how you can apply statistics to gain a comprehensive understanding of your data, which will serve as the context for signal detection. I then adapt the techniques of Statistical Process Control in new ways to detect not just changes in the measures but also significant changes in the patterns that characterize your data.
Only the signals matter.
April 11th, 2015
Sometimes in life the fates attack from all sides and leave us heaving for breath, overwhelmed. The last few months have been such a period for me. So much that’s been happening in my personal life has been nuts—the product of idiocy, incompetence, and at times pure meanness. A few days ago, in utter frustration I exclaimed, “I want to live in a world that makes sense.” Wouldn’t that be wonderful? To live in a world that works according to thoughtful and compassionate principles. Brainless bureaucracies would be a thing of the past. People who complicate our lives through incompetence or pettiness would suddenly grow up and give a damn. Systems that have developed to promote the interests of some to the detriment of others would be torn asunder. To the degree that natural inequities still exist, we could balance the playing field by identifying the causes and addressing them. The world wouldn’t be perfect, but we could address life’s problems thoughtfully and compassionately.
As I was thinking this, it occurred to me that this wish might be common among people who, like me, work to make sense of data. Perhaps we’re drawn to data sensemaking because we long for a sensible world, and this is our attempt to create a bit more order in the midst of chaos. I meet many fellow data sensemakers in my work and, based on the fine and dedicated people who attend my courses and read my books, I suspect that this correlation is real. If we pair this desire with the right skills and tools to make better sense of the world, we can use that knowledge to make the world a more sensible place. This dream is too precious to fritter away. The signals that live in our data are too precious to miss in the midst of deafening noise. Let’s focus our vision and double our effort. Let’s turn down the noise.
March 17th, 2015
The number of viable visual data exploration and analysis tools can be counted on the fingers of one hand. TIBCO Spotfire is among them. The merits of this product are undermined, however, by the irresponsible ways that TIBCO is currently promoting it. A new marketing campaign by TIBCO illustrates what happens when marketing professionals who either don’t understand analytics or care little for the truth are allowed free rein.
Here are a few lines from TIBCO Spotfire’s new “Finally…Answers Made Easy” campaign (emphasis mine), supposedly written by the company’s CTO, Matt Quinn:
At TIBCO, we believe just visualizing data isn’t enough. Embedded deep in the brains of data scientists lies a knowledge set that can truly benefit any one of us who has ever struggled with the dilemma of which graph to choose for a given data set. How many times have you highlighted a data set in Excel, selected Insert Chart and ended up with nonsense? You try a different chart, play with the axes, change the numerous options – before you know it, you’ve wasted an hour and haven’t made any progress. You certainly haven’t gotten anywhere near insight or understanding. Imagine if your software knew what you needed to see, even if you didn’t?
We have mined the data in the data scientists’ brains and shared what they know about visualizations: all the arcane rules about using measures on density plots, when to use aggregations and how to use time series correctly. Spotfire will automatically examine your data and recommend the best visualizations for it. Allowing our algorithm to choose the correct visualization will let you focus on what you know best – your business.
When you took your driving test, they didn’t ask you to explain the principles of the internal combustion engine – you just trust it works. Whereas your grandparents may have been a dab hand with a spanner and an oil can, life has moved on. So it will be for the future of analytics – it will work smarter, so you don’t have to.
Similar to Tableau, Spotfire attempts to determine an appropriate chart based on the data that you’ve selected. This is a useful time saver when it’s done well, but it can’t peek into your mind to determine what you want to see, so its guesses are frequently wrong. This feature can also serve as a useful guide for data analysis novices, but in this potential also lies the problem: you can’t let software do your thinking for you. The big lie that’s being told here appears in the last few words: “It will work smarter, so you won’t have to.” This is not only a lie—it’s a dangerous lie that keeps organizations trapped in ignorance, wasting their time, unable to tap into the value of their data.
Well-designed software can indeed help you “work smarter,” but not “so you won’t have to” work smart yourself. Data exploration and analysis software, no matter how good it is, cannot provide a workaround for your lack of analytical skill. Software vendors hurt you and ultimately hurt themselves when they claim that their products can be used effectively without the requisite analytical skills. They hurt themselves because, when customers learn that they were sold a lie and can’t actually use the software effectively, they become disgruntled and eventually move on to another product. Sadly, they rarely make a better choice the next time around, and the doomed process begins anew. No one wants to believe that a product that they spent a great deal of money to buy won’t solve their problems.
This marketing lie is in line with the “self-service BI” lie that’s been told for ages. The notion that BI software can auto-magically enable people without analytical skills to make sense of data is ludicrous, yet it’s an appealing lie. We want something for nothing, but the world doesn’t work this way. Analytical tools can’t help us do better and faster what we don’t already know how to do ourselves. It can only augment our intelligence—extend our reach and help us work around limitations—never replace our need for intelligence and skill.
TIBCO is certainly not alone in its willingness to spread misinformation in its attempts to sell its products. Every one of the viable visual data exploration and analysis software vendors have played fast and loose with the truth and mislead potential buyers to varying degrees. Most of the wannabe (i.e., not viable) vendors in the space are even worse.
I suspect that the first vendor in the analytics space that’s willing to tell the truth about its product and what’s required to use it will eventually lead the market, assuming that its product is good, even though they’ll lose many sales in the process. A vendor could differentiate itself from the pack by being truthful. The people who spend their days trying to make sense of data tend to respect truth. They’d find it refreshing to witness honesty coming from a software vendor. This vendor could honestly say, “Here’s the good news. The skills needed to analyze data can be learned by any reasonably intelligent person, given the right resources and enough practice.” This is indeed good news, but it’s not as sexy as the claim that a software product can replace the need for skill.
Damn, damn, damn…getting value from data requires skill and effort. After all of these years of trying and failing to get value from data without paying our dues, why are we still so willing to believe otherwise? There are no shortcuts to enlightenment.
January 6th, 2015
Perhaps you’ve noticed that I didn’t write a year-in-review blog post about 2014, extolling the wonderful progress that we made and predicting the even-more-wonderful breakthroughs that we’ll make in 2015. That’s because, in the field of data sensemaking and presentation in general and data visualization in particular, we didn’t make any noticeable progress last year, despite grand claims by vendors and so-called thought leaders in the field. Since the advent of the computer (and before that the printing press, and before that writing, and before that language), data has always been BIG, and Data Science has existed at least since the time of Kepler. Something did happen last year that is noteworthy, however, but it isn’t praiseworthy: many organizations around the world invested heavily in information technologies that they either don’t need or don’t have the skills to use.
I know that during the last year many skilled data sensemakers used their talents to find important signals in data that made a difference to their organizations. Smart, dedicated, and properly skilled people will always manage to do good work, despite the limitations of their tools and the naiveté of their organizations. I don’t mean to diminish these small pockets of progress in the least. I just want data sensemaking progress to become more widespread, less of an exception to the norm.
Data sensemaking is hard work. It involves intelligence, discipline, and skill. What organizations must do to use data more effectively doesn’t come in a magical product and cannot be expressed as a marketing campaign with a catchy name, such as Big Data or Data Science.
Dammit! This is not the answer that people want to hear. We’re lazy. We want the world to be served up as a McDonald’s Happy Meal. We want answers at the click of a button. The problem with these expectations, however, is not only that they’re unrealistic, but also that they describe a world that only idiots could endure. Using and developing our brains is what we evolved to do better than any other animal. Learning can be ecstatic.
Most of you who read this blog already know this. I’m preaching to the choir, I suppose, but I keep hoping that, with enough time and effort, the word will spread. A better world can only be built on better decisions. Better decisions can only be made with better understanding. Better understanding can only be achieved by thoughtfully and skillfully sifting through information about the world. Isn’t it time that we abandoned our magical thinking and got to work?