The notion that “we need more data” seems to have always served as a fundamental assumption and driver of the data warehousing and business intelligence industries. It is true that a missing piece of information can at times make the difference between a good or bad decision, but there is another truth that we must take more seriously today: most poor decisions are caused by lack of understanding, not lack of data. The way that data warehousing and business intelligence resources are typically allocated fails to reflect this fact. The more and faster emphasis of these efforts must shift to smarter and more effective. Although current efforts to build bigger and faster data repositories and better production reporting systems should continue, they should take a back seat to efforts to increase the data sense-making skills of workers and to improve the tools that support these skills.
Even in the sensitive arena of intelligence analysis, where decisions can preserve or end lives and information is often spotty, it is much more important to teach analysts effective skills and give them the best sense-making tools than it is to give them more data. Former CIA analyst Richards J. Heuer, Jr., argues the following in his book Psychology of Intelligence Analysis (1999):
The difficulties associated with intelligence analysis are often attributed to the inadequacy of available information. Thus the US Intelligence Community invests heavily in improved intelligence collection systems while managers of analysis lament the comparatively small sums devoted to enhancing analytical resources, improving analytical methods, or gaining better understanding of the cognitive processes involved in making analytical judgments. (p. 51)
This lack of appropriate funding exists no less and probably a great deal more in the corporate world as well. Heuer cites research findings that additional information rarely improves the accuracy of analyst’s judgments. What really matters is the quality of the mental model that analysts use—the conceptual frameworks that we bring to the process of data sense-making. Additional information only improves the accuracy of analytical judgments when it helps the analyst correct and improve his or her mental model. Heuer writes:
The accuracy of an analyst’s judgment depends upon both the accuracy of our mental model…and the accuracy of the values attributed to key variables in the model…Additional detail on variables already in the analyst’s mental model and information on other variables that do not in fact have a significant influence on our judgment…have negligible impact on accuracy, but form the bulk of the raw material analysts work with. (p. 59)
Unfortunately, even the most expert among us rarely understands their own mental models.
Experts overestimate the importance of factors that have only a minor impact on their judgments and underestimate the extent to which their decisions are based on a few variables. In short, people’s mental models are simpler than they think, and the analyst is typically unaware not only of which variables should have the greatest influence, but also which variables actually are having the greatest influence. (p. 56)
Researchers, especially those who work in the cognitive sciences, have learned a great deal about the way people process information and make decisions, including the flaws in the process that often trip us up. Proper training based on these insights is needed to make us better analysts; good tools are needed to help us work around analytical limitations that are built right into our brains. It is toward these ends that the bulk of our data warehousing and business intelligence investments should be directed. Is this where you’re focusing your efforts? Is this even on your radar?