Big Data and the NSA

In a recent blog post titled “Big data NSA spying is not even an effective strategy,” Francis Gouillart raised concerns about Big Data that are very much in line with mine. Gouillart’s is a refreshing and rare voice of sanity. He’s been around long enough to recognize marketing hype when he sees it, and as an independent thinker with ethics, not a shill for technology vendors, he is one among few who are speaking the truth. Here’s a sample:

The evidence for big data is scant at best. To date, large fields of data have generated meaningful insights at times, but not on the scale many have promised…Yet, for years now, corporations and public organizations have been busy buying huge servers and business intelligence software, pushed by technology providers and consultants armed with sales pitches with colorful anecdotes such as the Moneyball story in which general manager Billy Beane triumphed by using player statistics to predict the winning strategies for the Oakland A’s baseball team. If it worked for Billy Beane, it will work for your global multinational, too, right? Well, no.

The worship of big data is not new. Twenty-five years ago, technology salespeople peddled data using an old story about a retailer that spotted a correlation between diaper purchases and beer drinking, allowing a juicy cross-promotion of the two products for young fathers. Today, most data warehouses are glorified repositories of transaction data, with very little intelligence.

Working with multinationals as a management consultant, I have chased big data insights all my life and have never found them. What I have learned, however, is that local data has a lot of value. Put another way, big data is pretty useless, but small data is a rich source of insights. The probability of discovering new relationships at a local, highly contextual level and expanding it to universal insights is much higher than of uncovering a new law from the massive crunching of large amounts of data.

Read Gouilart’s article in full and pass it on. It’s time to usher in a quiet voice of sanity in this noisy, naive world of “more is better.”

Take care,

7 Comments on “Big Data and the NSA”

By John Kafalas. June 21st, 2013 at 11:09 am

Interesting juxtaposition of this blog post with the one about Eric Siegel’s book (which I haven’t read, but will now!). Sometimes two views that seem diametrically opposed can both be right, depending on exactly what part of the subject you’re talking about. I’m a big fan of Jim Cramer on CNBC (I love to make my friends in the financial-services industry cringe by mentioning his name, because they all hate him), and have no doubt that his record as a hedge-fund manager vindicates his approach. But at the same time, Burton Malkiel’s classic, A Random Walk Down Wall Street, which argues that you can’t beat the market, is also basically right! I think what it amounts to is that there’s no substitute for the Big Data intelligence tool between the ears — software tools are great, and so are investing methods, but success depends on the person using them. The secret of making money in the stock market isn’t really something you can quantify — it’s an instinct the best investors have, not a method per se. And getting insight out of data, big or otherwise, requires another type of instinct. In golf, it’s Ben Hogan’s “the Secret is in the dirt” axiom. Work with it long enough, and you’ll get it; just don’t try to explain how it happens!

By Neil Barrett. June 26th, 2013 at 5:45 am

In some ways it was ever thus. People crave a magic bullet so painlessly solve all their woes…

We give kids vitamin pills to make them more intelligent (but we could give families more support and maybe just teach the kids better), we don’t eat protein at the same time as carbohydrate [insert diet fad of choice] (but we could eat fewer burgers and do more exercise), and we buy magic software that will tell us what the right thing is to do and absolve us of personal responsibility (but we could take the time to understand, rather than thinking that all leadership is about shooting from the hip with charismatic assertiveness!).

Lots of people don’t want to be reasonable though, because it doesn’t feel right. And unless the analytical community can do something that both is reasoned and feels right, we’ll never win any ground. Look at Tableau adding their new woo-woo charts — they don’t help you be reasoned (or at least not well) but they do help you feel as if you’re being reasonable…

By Al Gulland. July 4th, 2013 at 7:33 am

“Today, most data warehouses are glorified repositories of transaction data, with very little intelligence.” Is a bit of a factless statement without any qualification.

Data warehouses themselves aren’t necessarily intelligent but one objective is to collate data so that a business user is able to analyse data to better understand their business or to disprove gut reactions and factless statements. The intelligence is then with the business user and how they use the data within the warehouse.

Similarly big data itself is not intelligence but it is the use and application of big data that is.

Yes, there is a lot of hype around, and every BI vendor is trying to cash in with their own tech solutions, but to say “I have chased big data insights all my life and have never found them” I find a bit incredulous.

By Stephen Few. July 5th, 2013 at 8:11 am


Anyone who has worked with data warehouses for as long as and Gouilart have know that his statement, “most data warehouses are glorified repositories of transaction data, with little intelligence,” is accurate. There are exceptions, but this is true of most data warehouses. Contrary to your claim, this is not a “factless statement.” Perhaps what you meant to say is that this statement of fact lacks evidence or is not true. Unfortunately, you countered Gouilart’s statement of fact “without qualification” or evidence–the very thing that you accused him of doing. If you have evidence to counter his claim, you’re welcome to provide it.

Gouilart’s statement that he has never found insights in so-called Big Data is not incredulous in the least. He is contrasting Big Data (i.e., collect everything you can as fast as you can) with “local data.” To understand what he means and evaluate the merits of his claim, I recommend that you read his work.

By Fernando Falcón. July 9th, 2013 at 1:38 pm

Here in Mexico, I have found that in most companies where I have worked in the area of ​​information management, data warehouse is used for the purpose of making operational reports. These reports only show the information of what is happening at the time or of what has already happened, but do not apply data analysis techniques to gain more insight. In addition, managers and directors seem more concerned about the Look and Feel of the report or dashboard, rather than the information itself. In my opinion, I think the lack of ability to harness the potential of information due to lack of knowledge or lack of faith in current techniques.

Although many companies have capacity at its Data Warehouse to store and process large amounts of data (Big Data), usually this is only used to create operational reports due to the use of analysis techniques is very little known and not to mention applied.

By Stephen Few. July 9th, 2013 at 1:43 pm


What you’ve observed in Mexico is true everywhere that I’ve worked, which includes the most developed countries in the world. Exceptions are rare, even among organizations with reputations for being analytically sophisticated.

By SusanO. August 5th, 2013 at 8:43 am

The concept of ‘Big Data’ has been around for a while but the new trend of trying to integrate, analyze and take action using ONLY technology is a disturbing one. The real benefit of human intervention lies in the unique knowledge a user brings to data. We can program and develop code to analyze and to take action but, using the example of automated reports that produce a list of people to investigate and/or launch an investigation without human review is a risky road. Look at the automated trades in the stock market and how those have impacted investors. Automated decisions can sometimes result in disaster.
Yes, there is a need to tame ‘Big Data’ and integration and analytical tools and algorithms can help us do that.
But, in the end a person with knowledge and experience should have the final say. Using concise information from business intelligence tools or industry-specific solutions, users can create personalized alerts and exception reports with thresholds. They can receive notifications when things go awry and pay personal attention to the issue. They can share information with others and collaborate and they can drill into data and analyze root causes to correct an issue before it becomes a real problem.
We still have a long way to go to tame the Big Data ocean, but good business intelligence tools and analytical tools can be combined with the human knowledge and intervention to more quickly make good decisions. Triggers and automated decisions have a place but not when the business (or in the case of the NSA, the nation and the world) depends on critical decisions. In this age of global competition and rapid changes, decision-making can be overwhelming. There is so much information to sift through and it is tempting to sit back and let computers drive our future. But, we have to blend true business intelligence with the intelligence of users and experts who know their jobs, and give them the business intelligence tools they need to integrate, access and analyze data easily.
We’re definitely not ready to turn the world over to technology and trust that those oceans of data can be properly analyzed and decisions can be made solely with technology.