Thanks for taking the time to read my thoughts about Visual Business Intelligence. This blog provides me (and others on occasion) with a venue for ideas and opinions that are either too urgent to wait for a full-blown article or too limited in length, scope, or development to require the larger venue. For a selection of articles, white papers, and books, please visit my library.

 

The Ghost Map — Data visualization in the 19th century

April 29th, 2007

I recently finished reading a wonderful book by Steven Johnson entitled The Ghost Map: The Story of London’s Most Terrifying Epidemic – and How It Changed Science, Cities, and the Modern World. In the summer of 1854 cholera swept through a section of London with unprecedented intensity. At the time, the cause of cholera was unknown and rapidly growing modern cities such as London, with dense populations packed into small areas, were rich breeding grounds for this disease. Most of those who concerned themselves with disease and its cure held tightly to the miasma theory that cholera spread through the air and was associated with the bad smells and the unclean urban environments that produced them. In fact, cholera is a bacterium, which was spreading through the water supply. This book tells the story much as a journalist who witnessed it firsthand would do, but a journalist who had the advantage of hindsight informed by knowledge of modern medicine.

Several people of the time play important roles in this story – none more than John Snow, a medical doctor and research scientist. The ghost map refers to a map that he drew by hand during the process of his investigations, which could clearly demonstrate to anyone with open eyes that the source of the outbreak was the Broad Street well. Despite the evidence that this map displayed, however, the miasma theory of cholera transmission prevailed for several years after the epidemic. Eventually, due largely to the tenacious efforts of John Snow and an unlikely supporter, Reverend Henry Whitehead, the evidence won out and steps were taken to eliminate the conditions in which cholera could spread.

This story is important in the history of data visualization, because it is one of the earliest accounts of how a visual representation of important data was able to bring to light evidence that might have otherwise remained obscured for much longer if relegated to a tabular display. In this case, a picture (in the form of a map with quantitative data) was indeed worth a thousand words and helped to save many thousands of lives.

This is more than the story of a great map, however. It tackles larger issues, such as how new ideas and scientific discoveries become adopted, often against great resistance, even from the intellectuals of the day. John Snow and Henry Whitehead are great role models for all of us who care about discovering and communicating the truth, even when it is unpopular. I recommend this book highly.

Take care,

Signature

Ghost Map

WebCharts3D — Dysfunction at its finest

April 24th, 2007

Would you buy a pair of glasses with lenses that were so scratched up you couldn’t see through them, even if the frames looked cool? Not if you want to get from point A to point B without injury. So why would you ever buy charting software that transforms simple information into a completely unreadable display? Yesterday, GreenPoint, a self-proclaimed “leader in enterprise-wide visualization solutions,” issued a press release announcing the latest release of WebCharts3D. Even this product’s name advertises its dysfunction. Adding a third dimension of depth to bars, lines, and pies obscures the data. To this GreenPoint adds more dysfunction by making the objects in charts transparent (for example, see-through bars), resulting in a maze of lines and angles that must be unraveled to make sense of the data.

Try to decipher the patterns and values in the following chart. Come on, give it your best shot. Even if I offered a cash prize to anyone who managed to come close, it wouldn’t be worth your effort to try, because you’d be forced to use the prize money to pay a doctor to fix the damage done to your eyes.

Band Chart

Here are a few more examples:

Column Chart
Band Chart
Pyramid Chart

Disinformation in all shapes and sizes. If this is what you’re after, then don’t hesitate to buy this product. If, however, you want your charts to actually communicate information, look for a product that proudly advertises charts that are easy to read.

Just to be fair, most but not all of this product’s charts are transparent. For example, here’s a radar chart that you could use to compare the performance of three products across eight years of time. Did you know that time is circular and that in the year 2007 we have returned to where we began in 1999? Despite this revelation, I’m finding it hard to relinquish my notion that time is linear and my desire to see this information in a simple line graph.

Radar Chart

WebCharts3D is not alone in its ability to obscure otherwise clear and simple data, but when a product this bad issues a press release, it’s hard to ignore.

Take care,

Signature

P.S. For the benefit of Ryan, who has posted a response to this blog topic, and readers who wish to see a more comprehensive sample of the charts that are available in WebChart3D, here are all six versions of the Step Chart that appear in the sample gallery.

All Step Charts

“Sources of Power” in Data Visualization and Decision Making

April 19th, 2007

I have sometimes been amused during my attendance at high-level business meetings in American industry, amused at the discrepancy between the way we are told that important decisions get made and the truth. (Donald A. Norman, Things That Make Us Smart: Defending Human Attributes in the Age of the Machine, 1993, Basic Books, New York)

The logical-rational decision making models that we are taught in college are worthwhile and necessary, but they are rarely used in the course of everyday business. Although there are times when they ought to be used but aren’t, it is appropriate that they are used seldom. One reason for this is that these processes require a great deal of time — something we rarely have. The other reason is that if you have expertise in a domain, you are able to make decisions that are usually just as good, but require little time.

Gary Klein, Chief Scientist at Klein Associates, Inc., has written an informative book about decision making that approaches the topic quite differently from most other books and articles. In the introductory chapter of Sources of Power: How People Make Decisions (MIT Press, 1999) he sets the stage for his treatment of the topic as follows:

During the past twenty-five years, the field of decision making has concentrated on showing the limitations of decision makers — that is, that they are not very rational or competent. Books have been written documenting human limitations and suggesting remedies: training methods to help us think clearly, decision support systems to monitor and guide us, and expert systems that enable computers to make the decisions and avoid altogether the fallible humans.

This book was written to balance the others and takes a different perspective. Here I document human strengths and capabilities that typically have been downplayed or even ignored.

Klein has studied decision making for years and has filled the book with research findings of his own and others, along with story after story of decision making in action.

Despite the fact that we at times and for good reason use “deductive logical thinking, analysis of probabilities, and statistical methods” to inform decisions, Klein reports that in natural settings decisions are rarely analytical, but are usually informed by intuition, mental simulation, metaphor, and storytelling.

The power of intuition enables us to size up a situation quickly. The power of mental simulation lets us imagine how a course of action might be carried out. The power of metaphor lets us draw on our experience by suggesting parallels between the current situation and something else we come across. The power of storytelling helps us consolidate our experiences to make them available in the future, either to ourselves or others.

If you are new to a field, having not yet developed expertise in the domain, decision making must be informed by a relatively slow process of information gathering and evaluation. If you are an expert, however, this research-laden and brain-taxing process is necessary much less often. According to Klein, experts usually make decisions based on what he calls the “Recognition-Primed Decision Model” (RPD). It combines two processes: “the way decision makers size up the situation to recognize which course of action makes sense, and the way they evaluate that course of action by imagining it”. Experts can often look at a situation and quickly recognize it as familiar, knowing intuitively what’s going on and how to respond. When multiple courses of action seem possible, they can take the first from their immediately prioritized list and evaluate its merits by running a quick mental simulation. If problems are discovered during the mental simulation, they proceed to the next possible course of action, until they find one that works, and then without further delay, take action. Are they always right? No, but if they’re experts in the field, they’re right most of the time.

How does this relate to data visualization? People who analyze data, if they are experts in the domain, usually know what’s important and make sense of it based on pattern recognition. They’ve seen it before, or something similar. Nothing presents meaningful patterns that reside in data better than properly chosen and well designed visual representations. More than any other tool, data visualization software can support meaningful pattern recognition for a broad range of people and can enable them to clearly present what they’ve found to others. Good visual analysis software pares information down to its essence in the form of a picture, removing the noise to enable clear focus on the signal. It translates abstract patterns of meaning in the data into images that can be easily perceived by our eyes and discerned by our brains, thereby serving as an external tool of cognition.

Good decisions must be based on clear presentations of data that allow experts to bypass time-consuming and often unnecessary mental gymnastics and software mechanics so they can spend their precious resources assessing the situation and responding while there’s still time. If you don’t have good visual analysis and presentation tools to support this process, you’re wasting valuable time and working partially blind.

Take care,

Signature

Visualizing 360 Data Points

April 9th, 2007

In addition to this blog, Perceptual Edge provides a Discussion Forum for people to exchange and discuss ideas regarding data visualization. It tends to be a place where specialists in visual data analysis and presentation congregate, but from time to time the ideas that are shared cry out for a broader audience. A few days ago Jorge Camoes provided a dataset with 360 values — expenditures in 12 categories for 29 European Union countries, plus an average for all of these countries (30 rows by 12 columns of values) — and seeded a discussion by presenting several ways to represent this information in a single display. The discussion that has ensued presents a rich exploration of data visualization, not only in terms of the solutions that have been proposed, but also in its consideration of the questions one must ask to determine the best way to visualize quantitative data. If this sounds interesting, please take a look and perhaps join in the discussion.

Take care,

Signature

Things That Make Us Smart

March 30th, 2007

Donald Norman is well known in the field of design. For many years he has been helping designers become aware of the qualities that must exist in products for people to use them effectively. As a cognitive psychologist, Norman bases his recommendations on an understanding of the human mind—both its strengths and its limitations. Well designed products (including software) take advantage of our strengths and help us overcome or work around our weaknesses. They don’t complicate tasks by forcing us to do things that we aren’t good at. They help us do things without drawing undue attention to themselves. We who are involved in the field of business intelligence, whether we know it or not, are designers. The reports and dashboards that we create are products. Just like any product, they must be designed in a way that can be perceived and understood by the human mind. Every business intelligence professional can benefit from Norman’s work.

Norman is probably best known for his book The Design of Everyday Things, which is superb. When I first read it several years ago, my neck became sore from the constant nodding that occurred as I recognized one point after another that made so much sense I couldn’t help but nod and smile. Although I’ve read several of his other books and papers since, for some unknown reason I managed until recently to miss his 1993 contribution to design entitled Things That Make Us Smart: Defending Human Attributes in the Age of the Machine (Basic Books). I would like to encourage you to add this book to your library. Allow me to entice you with a few quotes.

Society has unwittingly fallen into a machine-centered orientation to life, one that emphasizes the needs of technology over those of people, thereby forcing people into a supporting role, one for which we are most unsuited. Worse, the machine-centered viewpoint compares people to machines and finds us wanting, incapable of precise, repetitive, accurate actions. Although this is a natural comparison, and one that pervades society, it is also a most inappropriate view of people. It emphasizes tasks and activities that we should not be performing and ignore our primary skills and attributes—activities that are done poorly, if at all, by machines. (p. xi)

The good news is that technology can make us smart. In fact, it already has. The human mind is limited in capability. There is only so much we can remember, only so much we can learn. But among our abilities is that of devising artificial devices—artifacts—that expand our capabilities. We invent things that make us smart. Through technology, we can think better and more clearly. We have access to accurate information. We can work effectively with others, whether together in the same place or separated in space or time. Three cheers for the invention of writing, reading, art, and music. Three cheers for the development of logic, the inventory of encyclopedias and textbooks. Three cheers for science and engineering. Maybe. The bad news is that technology can make us stupid. (p. 3)

My goal is to develop a human-centered view of the technologies of cognition. My theme is not antitechnological, it is prohuman. (p. 12)

The power of the unaided mind is highly overrated. Without external aids, memory, thought, and reasoning are all constrained. But human intelligence is highly flexible and adaptive, superb at inventing procedures and objects that overcome its own limits. The real powers come from devising external aids that enhance cognitive abilities. (p. 43)

A good representation captures the essential elements of the event, deliberately leaving out the rest…A representation is never the same as the thing being represented, else there would be no advantage to using one…Herein lies both the power and the weakness of representations: Get the relevant aspects right, and the representation provides substantive power to enhance people’s ability to reason and think; get them wrong, and the representation is misleading, causing people to ignore critical aspects of the event or perhaps form misguided conclusions…Representations are important because they allow us to work with events and things absent in space and time, or for that matter, events and things that never existed—imaginary objects and concepts. (pp. 49 and 50)

My concern is with the violation of psychological principles, whereby graphs are used in inappropriate ways, sometimes deliberately to confuse but more often out of sheer ignorance. My frequent complaints to friends, colleagues, students, and newspapers are commonly met with the excuse “My computer program did that for me automatically; I had no choice.” Poor reason: Ignorance of the law is not a valid excuse, whether it be a governmental law of a psychological one. (p. 94)

The things we are good at are the things natural to humankind. The things we are bad at are unnatural. And guess what? We can build machines that perform flawlessly many of the things we are bad at. As for the things we are good at, it is very difficult, today usually impossible, to build machines that can do them. “Why that’s wonderful,” you should be saying. “What a marvelous match to our abilities! Between us and our machines, we could accomplish anything, for the one complements the other. People are good at the creative side and at interpreting ambiguous situations. Machines are good at precise and reliable operation.”

Hah! That isn’t what has happened. Instead, technology has decided that machines have certain needs and that humans are required to fulfill them. The things we are good at, those natural abilities, are hardly noticed. Machines need precise, accurate control and information. No matter that this is what people are bad at providing, if this is what machines need, this is what people must provide. We tailor our jobs to meet the needs of machines. (p. 222)

It will take extra effort do design systems that complement human processing needs. It will not always be easy, but it can be done. If people insisted, it would be done. But people don’t insist: Somehow, we have learned to accept the machine-dominated world. If a system is to accommodate human needs, it has to be designed by people who are sensitive to and understand human needs. I would have hoped such a statement was an unnecessary truism. Alas, it is not. (p. 227)

Convinced? Do yourself a favor and read this book.

Take care,

Signature