> They’re having a wild orgy in that bed, rolling in cash, but it is only the end users who are getting screwed

Love it! I too wonder at the supposed job numbers from flurry of these type of articles. Hyperbolic to say the least. I remember at high school being told there wouldn’t be enough IT workers until 2025 at least. What rubbish. All these predictions really don’t take into account how wrong the last gasping predictions were and why; technology changes and shifts in efficiency to reduce the numbers of people actually needed, and of course as you point out, how well it all actually delivers the benefits it claims to. As always, no silver bullet in any of this stuff without the processes and business shift in understanding how to use it.

“What constitutes a Big Data role?”

Easy: having the words “big” and “data” together somewhere in the job title or description. Of course, the actual responsibilities and skill-sets will be no different than any other data-related role.

“Have you noticed that every business intelligence vendor has suddenly become the leading Big Data company without changing anything that they do?”

What annoys me is that these vendors aren’t just selling old solutions under new names, they’re also reinventing the problems with new names. And of course the names need to sound ominous. BIG Data (*crack of thunder*) – who wouldn’t be afraid to tackle that on their own?

As long as businesses trust vendors to identify their problems for them, the vendors will keep identifying problems for which they are always the leading experts in solving.

Seth,

How is Hadoop a qualitative departure from the past? Doesn’t it just do a little better what we’ve been doing for years (distributed systems, integration of structured and unstructured data, etc.)?

The analyst -vendor buzz creating system in business technology seems corrupted and ripely overdue for disruption. Bravo Mr Few, few people lack the courage to flag bullshit so openly, too busy selling their next webinar. respect!

Quite frankly I’m fond of the term ‘Big Data’. And while I agree that it is somewhat vague and generic, the fact of the matter is that it helps to define a problem in some way that we can start to have an intelligent discussion about solutions. I don’t think anyone can dispute that Volume is absolutely growing like never before, so this is a new dynamic. Also, I don’t think any reasonable person would argue that the Variety of information is like never seen before. And, last, but not least, the Velocity is also very obviously a challenge to harness. So put this all together and it paints a picture of the problem at a high-level; without specifics, admittedly.

What’s really exciting, and somewhat new, is that there are reasonable tools to help real people, in all sorts of industries to realistically solve these problems instead of just huge enterprise organizations like IBM or Stanford University, for example, that have gigantic budgets and massive resources. Even the smallest of companies can rent Amazon AWS to do an incredible amount of data crunching that they didn’t have access to just a few short years ago. Or open source software like Splunk gives everyone a chance to build nice solutions without a significant up-front investment. Additionally, with the incredible emergence of Web Services to gather and integrate data from all different sources such as social media feeds makes the world of Big Data extremely interesting to people who wish to exploit this information.

While I understand your, and others, distaste for the term I think it does a sufficient job to describe the opportunity at the 10,000 foot level. Just my two cents.

Stephen,

My point about software such as Splunk is that it enables people to learn, maybe inaccurately as you point out, some of the simple concepts of data aggregation and presentation. Having software such as this is a good thing, in my opinion, for the point of learning general concepts. Also, being open source means that the community can make it better if they wish. Just like Alfresco and many other solutions.

For Amazon AWS, my point is that this is an example of a resource that small companies can utilize to gather volume, variety and velocity, so I would say that in some way AWS can help a small company such as Fliptop, for example (and I just picked one case study at random to illustrate this point), to achieve the following:

“AWS allows the company to scale to hundreds of instances to process massive lookup jobs without the corresponding capital expenditure of a traditional infrastructure. “In addition,” says Chiao, “the built-in redundancy, recovery, and monitoring features of services like Amazon RDS allow our development team to run our production infrastructure without any dedicated operations resources.”

As far as the Gartner prediction, it’s not quite 2015 yet so we might be a bit pre-mature. I agree that with the term being so vague they can find a creative way to justify their prediction but I have personally witnessed more of my industry contacts changing job titles to something with “analytics” or “data specialist” or something similar. Also, it seems like demand for these types of positions is increasing in my unscientific research. Also, the pay for these types of people seems to be increasing so I am personally interested and not necessarily benefiting from the hype, yet.

I was wondering who would be first to mention Cloud :)

Kevin, according to http://www.aiim.org/community/blogs/expert/Big-Data-3d-Big-Problems-3d-Big-Opportunity you ‘detest’ these ‘over-used’ and ‘faddish’ terms such as ‘Big Data’ and ‘Cloud’, yet above you say you are ‘fond’ of them. Which is it? I was also amused by your suggestion that a common way to capture ‘indexes’ is to autogenerate a searchable PDF file. Really? I can’t honestly think of a less efficient way to repurpose data.

Having not been involved in Gartner’s forecasting process, nor having purview into 300,000 client interactions throughout the year, it’s no wonder Stephen is odiously dismissive.

“Big Data” is the #1 term searched on Gartner.com and one of the top inquiry topics among our clients (the vast majority of which are not technology vendors).

True, Big Data is not new. I first defined the “3Vs” (volume, velocity, variety), now commonly used by the industry to characterize Big Data, at Gartner over 12 years ago (http://goo.gl/wH3qG). And SGI Chief Scientist John Mashey actually coined the term several years before that.

As for the jobs forecast, Gartner has a disciplined forecasting methodology that brought together our industry experts, human capital experts, and forecasting specialists. It leverages existing data on skill breakdowns, related technology sales, end-user survey data, service trends for similar technology bubbles and country/region/industry specific data.

The range of job types considered in the forecast include those in computing, HW, storage, DBMS, enterprise content mgt, data prep, data quality, data integration, app dev, governance, security, analytics, etc. Note that our forecast clearly indicates that these will not all be new jobs, but also many existing jobs that will require new Big Data-related skills.

–Doug Laney, VP Research, Gartner, @doug_laney

@Doug “Having not been involved in Gartner’s forecasting process, nor having purview into 300,000 client interactions throughout the year” – so you’re basically saying that because Stephen doesn’t work for Gartner or a similar large analyst he is in no position to criticise?

Fabio,

What is this new technology that “breaks some paradigms and creates a new baseline for analytics applications”? What are the paradigms that it breaks? What is the new baseline that it sets?

Fabio,

In fact, real applications have included unstructured data and real-time analytics (if by this, you’re referring to in-memory databases) for many years. Two BI tools that I’m familiar with–Spotfire and Qlikview–have used in-memory database engines for many years. Tableau introduced theirs several versions ago. I was including unstructured data in applications over 20 years ago. I’m not familiar with “infinite data storage,” but I can assure you that no data storage is infinite.

It is true that the capabilities of these technologies continue to progress. For example, your example involving real-time video is something that technologies have only enabled in the last few years when processing power finally made them possible.

So here’s the question: if we’ve been imagining these capabilities for many years and technologies have eventually enabled them as a result of the same exponential growth that began with the advent of the computer, is it accurate to speak of these capabilities as a qualitative departure from the past or a quantitative continuation of the past?

What we’re calling Big Data, mobile computing, social networking, and the cloud have all evolved as continuations of the past, not discontinuous departures from it. What we’re calling Big Data today we called decision support over 30 years ago, data warehousing over 20 years ago, business intelligence over 10 years ago, and analytics over 5 years ago. The first computer that I owned was the original portable computer, called the Osborne. It was an early example of mobile computing. I began using email about 29 years ago. It was an early example of social networking. When I began working for a large bank, about 28 years ago, our major systems were housed and run at a facility in Fresno, California, to which we connected using T1 telecom lines. It was an early pre-Internet example of what we now call the cloud.

I applaud the fact that technologies continue to improve and extend our capabilities. We need good technologies to solve the problems that we face today. However, I don’t applaud poorly designed technologies, the notion that technologies rather than people (through the assistance of technologies) will solve our problems, or the hype of marketing programs that keep organizations chasing the latest technologies rather than developing the skills that are required to use data effectively, independent of the tools that they use.

Big Data is just a new name for old ideas and for technologies that provide an incremental improvement over the past. For fun, take the time to look at articles, papers, and presentations by business intelligence thought leaders today and compare them to the work of these same people five or ten years ago. You will see, with few exceptions, that they say the same thing with a few new terms mixed in. They are stuck in the past. To get unstuck, they must become more intimately familiar with the real needs of people who struggle to make sense of data and also with the strengths and weaknesses of human cognitive and perceptual systems. Only then will they help to usher in the true “information age” that’s needed.

Elephants like to mate with other elephants. Whether Gartner knows it or not, there is an inherent self selection bias with its audience, and the theatre (analysis) of it all is part of the mating rituals. The job of vendors is to eat up ever expanding Capex/ Opex budgets, all they are doing is creating an avenue for budgets to be spent in full rather than get it taken away from departments. For this, I say don’t hate the player, hate the game.

That eco system, as dysfunctional as it is, i do not feel sorry for customers anymore that should know better than jumping on board the gravy train. Those that exploit the game are quick to share their numerator of success, without highlighting just exactly how big the denominator of failure really is. As much as I share the distaste you have about those that exploit the game, companies that treat technology first before people and buy into the BS do not have my sympathy.

Stephen,

I think we don’t really desagree in our opinion, it is just the height of the step to call a technology as an evolution or a new technology. I agree that sometimes new names are created to allow re-investments on old IT concepts, you just need to be careful to undertand the height of each step, calling it as a new name or not.

The main stuff here is to know that the technology by itself does not solve any business problem and the most important is to have human brains thinking about how to mix all this names to generate useful applications.

regarding your last point in your reply to me, I agree with that sentiment 100%, it is the reason I left consulting. Having to jam a square peg in a round hole over and over until a locked in customer caves in, it truly is awful experience when inefficient operations is the by product of ruthless tech sales and naive customers. Then again, I have been called a purist :)

“Big Data is just a new name for old ideas and for technologies that provide an incremental improvement over the past”

“Have you noticed that every business intelligence vendor has suddenly become the leading Big Data company without changing anything that they do?”

Loud and clear. Make noise and you will get the attention of CIOs and senior executives. Every single white paper on Big Data is nothing more than the BI we had earlier with more latest jargon. I attended a presentation by senior guy from so-called Big Data solution provider “Mu-Sigma”. They are after every single financial institution with deep pockets to sponsor on projects which has a disclaimer at the beginning as “Dont expect short term benefits” and in the long term we are dead!!

They talk about cross-industry convergence concept. This is using Yield Optimization technique for Banks. I was going hang on minute. What?

It is really surprising how they convince CIOs and senior management with their research work. And we were listening to their sales pitch. I hope to see some well written case studies on the failures of these Big Data in the future. That will send a strong message

Hello Doug,

I had given up hope that you would respond. Thanks for picking up the thread of this discussion. To remind readers what this discussion is about, here’s the claim by Gartner that I deemed nonsense: By the year 2015 a total of 4.4 million jobs will be created worldwide to support Big Data.

So, I have a prediction of my own. By the year 2015 a total of 6.3 million jobs will be created worldwide to support Small Data. “On what is this based?” you ask. Oh, I can’t reveal anything about my forecasting process — intellectual property and all that — but you can rest assured that it is viable and trustworthy. Could I have nearly 100,000 users of my services (books, articles, courses, etc.) if my methods weren’t trustworthy?

This prediction is every bit as valid as Gartner’s, even though it was fabricated from thin air. Can you prove otherwise? You can’t, because my prediction isn’t verifiable. Why? Primarily because I haven’t defined what I mean by Small Data jobs. I could have crunched the numbers using the most sophisticated statistical model known to humankind and this prediction would still be nonsense, because the measure — Small Data jobs — hasn’t been defined.

In my work, I explain my claims, back them with evidence, and lay them open for scrutiny. I believe that you are hiding pseudo-research behind the veil of intellectual property concerns. Is it acceptable to charge your clients for expert forecasts that might be nothing more than wild guesses? Is there a wizard or a demented little man from Kansas behind the curtain? People of intelligence, especially data analysts, will not accept your predictions as reliable without inspection. The fact that your clients accept them reveals the sad state of analytical reasoning among the CIOs of the world. In time, this will change (a prediction is based on hope, not on certainty).

You say that I criticize things that I don’t understand and suggest that I am nothing more than a self-aggrandizing “blogger.” To the contrary, as you well know, this is my field of expertise. I suspect that I understand what goes into effective analytics, including predictive models, as well or perhaps better than you. I can assure you that my criticism of Gartner’s prediction is not due to sour grapes. I do not envy your ability to make vaporous predictions. Instead, I have genuine, unadulterated disdain for predictions that are built on pseudo-analytics. I work in the field of quantitative data sensemaking and communication (i.e., business intelligence, analytics, data visualization, data science, and yes, what you call Big Data). I help organizations use data more effectively for decision-making. It is my job to expose pseudo-analytics like this prediction of yours for what they are: pure nonsense. When Gartner and other so-called analyst organizations spread blarney like this, it makes me angry, because it does harm.

As the VP of Research at Gartner, have you instituted a program to confirm the accuracy of your predictions? Have you reviewed past predictions to see if they were on target? How could you? A prediction that “by the year 2015 a total of 4.4 million jobs will be created to support Big Data” isn’t verifiable? You haven’t defined what you mean by a Big Data job as different from other IT jobs. I suspect that this omission was intentional. It is a sly trick that is also used by psychics when they do cold readings. If you invested more effort in this prediction than the nighttime carousing of drunken analysts playing with a Ouija board, as I described the process, you wasted your money — or more to the point, you wasted the money that your clients paid you for credible research.

Let’s consider your business model. You said: “Gartner doesn’t ‘generate revenue’ from promoting IT concepts — rather our clients subscribe to our service annually.” Your biggest clients — those that pay the most for your services — are the very companies that you are supposedly analyzing, monitoring, and reporting on. You support their marketing efforts by praising and making glowing predictions about the technologies that they sell. So yes, you do generate revenue from promoting IT concepts. You directly contribute to the marketing hype in ways that are designed to keep your clients happy. They pay you and you reward them by presenting their products in a good light.

Back to the general topic: Big Data. You said: “The Gartner definition is specific though as to the types of challenges, use cases, strategies and threshold that’s crossed from ‘just data’.” Excellent. Since I’ve scoured the literature and haven’t found anything of substance that separates so-called Big Data from just plain data, I’m eager to hear about these challenges, use cases, and strategies that are substantially new. Please elucidate. As you yourself pointed out, the characteristics that are generally used to describe Big Data — the 3 V’s (volume, velocity, and variety) that you wrote about long ago — are not new. When did we cross this threshold? When, since the advent of the computer, was data not growing at an exponential rate, moving at increasing speeds, and expanding in variety? If a threshold had been crossed, I would acknowledge it, do my best to describe it, and teach people how to work with data in the new ways that were required. As it turns out, however, what people need most to use data effectively isn’t new. Of course, data handling technologies continue to improve, but in fits and starts, much too slowly, and only rarely in game-changing ways. It is true that data continues to grow in volume, velocity, and variety, but this is growth, not an evolution into some new species. No threshold has been crossed.

What’s behind Big Data? The answer can be found by following the money. Now that would be a worthwhile research project for Gartner.

If anything that I’ve written here about Gartner or Big Data is not accurate, please say so, but back your responses with evidence. As the VP of Research at Gartner, especially one who specializes in information technology, if you can’t support your position with data, something is amiss.