Excel 2010: Another Opportunity Missed

In March of 2006 I glimpsed the new charting capabilities of Excel 2007 for the first time and wrote about them in an article titled “Excel’s New Charting Engine: Preview of an Opportunity Missed.” After waiting for years to see how the world’s most popular data analysis software would improve its sadly lacking charting capabilities, I mourned the opportunity for improvement that was almost entirely missed. Essentially, an entirely new charting engine in Excel 2007 replaced the old one, but what it brought with it was a fresh array of flashy visual effects that encouraged us to hide our data behind a thick layer of cheap makeup. Within two days of my article’s publication, I received an email from Scott Ruble, the person in charge of charting functionality in Microsoft Office products. Scott invited me to help the team improve the charting capabilities of the product’s next major release—Excel 2010—which will become available sometime during the first half of next year. We’ve had several conversations since, including a teleconference with the team. Early glimpses into the charting capabilities of Excel 2010 are now beginning to surface, and it appears that the opportunity to improve the product’s data visualization capabilities has once again been missed. Although I haven’t seen an advance version of the product myself, those who have tell me that it includes only one change to charting: the addition of sparklines. What a shame.

Don’t misunderstand me. I’m thrilled that a version of Tufte’s sparklines will be added. Assuming that the implementation is well designed, this will eliminate the need for an add-in product if you want to display a set of time-series values as a simple sparkline, but this is a single grain of sand compared to an entire seashore of need. No single product in the world is used more than Excel for analytics, not because it’s a good tool for data exploration, analysis, and presentation—it isn’t—but because almost everyone in the world who works with quantitative data has it. Just imagine how much the world would benefit if Excel were more powerful and better designed. I was frustrated and upset when Excel 2007 missed the mark, but now with Excel 2010 trying to assuage our misery with nothing but sparklines, I’m inclined to give up on the product entirely as a tools for data analysis. Fortunately, where Excel has failed, alternative products have emerged that deliver effective and visionary analytical abundance.

Will Microsoft play a role in the future of data analytics? Although the company boasts a business intelligence (BI) solution and even declared its commitment with the first annual Microsoft Business Intelligence Conference in May of 2007 (the 2009 conference was cancelled), only its database does anything particularly useful for BI so far. The other pieces that have been awkwardly rubber-banded together into a so-called BI solution suggest the lack of a strategy or a confused one at best, and previews of coming additions, such as Project Gemini, suggest nothing but already dated functionality for the future. I don’t have a bias against Microsoft because it’s huge and powerful; I have a progressively growing disappointment with it because of unfulfilled potential. If Microsoft seriously applied itself to the task, it could probably do some wonderful for the world of analytics. At this point, Microsoft will have to do something big, totally unexpected, and uncharacteristically well designed if it hopes to play a role in the future of analytics. I would welcome this with arms wide open, but I’m not holding my breath.

Take care,

35 Comments on “Excel 2010: Another Opportunity Missed”


By Govert. August 10th, 2009 at 11:53 am

Hi Stephen,
I agree on your opinion(about Excel 2007 that is)
I haven’t seen 2010 yet.
A lot of visualizations in Excel require workarounds. They are fun to do but not very cost-effective.

I wonder, what alternative products do you mean?

Govert

By Mark Underwood. August 10th, 2009 at 12:07 pm

Excel is the one tool that could, in a single release, affect the way that millions of users approach data visualiation. It’s disappointing that the Office management team may have once again missed an opportunity to make more than incremental improvements.

The past year has seen a number of compelling Microsoft initiatives that showed the firm still capable of innovating. But making innovation systemic apparently requires more a casual sip of third party Cool-Aid. Perhaps that’s why the Company acquired Dundas Technology, whose .NET offerings seemed well within the ability of Microsoft to create internally.

-Knowlengr

By Stephen Few. August 10th, 2009 at 1:26 pm

Govert–Commercially available visual data analysis and presentation solutions that are effective include products such as Tableau, Spotfire, and Panopticon for both non-statisticians and statisticians, and JMP from SAS for statisticians. Other good products might exist, but these are the ones that I’ve examined closely enough to recommend with confidence.

Mark–Like you, I believe there are talented people at Microsoft who could develop a useful data visualization solution. Some of the brightest contributors to the field of information visualization research work for Microsoft Research. There appears to be a gap, however, between the folks who could do this and the opportunity to do so. There are people in the Microsoft Office products group who read my work regularly and share my vision, but for some reason their good intentions never make it into Excel. (By the way, saying that Dundas’ technology was well within the ability of Microsoft isn’t saying much, in my opinion. I’ve seen no evidence that Dundas understands data visualization.)

By Dan Murray. August 10th, 2009 at 2:52 pm

Stephen;

Unfortunately I’m not surprised at your early findings regarding Excel 2010. I suppose the truth is that Microsoft just doesn’t feel the need to address the issue in a serious way.

By Stephen Few. August 10th, 2009 at 3:03 pm

Dan,

It very well might be true that Microsoft isn’t serious about business intelligence or data analytics. If this is the case, it would simplify things for everyone if Microsoft would make a statement to this effect. Of course I don’t expect this to happen, but wouldn’t it be nice if vendors actually made clear statements about their intentions (assuming that they have clear intentions)?

By David Gerbino. August 10th, 2009 at 6:27 pm

Another version of Excel and another negative critique by Stephen Few. With so many data visualization evangelists out there like Edward Tufte, Stephen Few, and Naomi Robbins, there is no reason for the most influential software developer out there to be missing the mark.

I really do not understand Microsoft.

By Jon Peltier. August 10th, 2009 at 8:57 pm

I don’t think Microsoft really feels the need to invest development resources in their desktop product. They’re all into cloud and collaboration and other fancy phrases, and they provide only fancy visual features but no significant improvements. They talk BI, but I haven’t seen much fire behind the smoke. Dundas? Spare me.

2007 was two giant steps back for charting. The defaults were slightly improved, but the interface was made much worse, and a lot of the fun old workarounds were broken in the process. And clients yell at ME because of it.

What I’ve seen of 2010 charting is that the sparklines are nice, if rudimentary, and the data bars have been corrected. The regular charting functionality seems little changed from 2007, the dialogs and ribbon interface are just as awkward and unworkable. Maybe a later version of the beta will show improvement, but I’m not holding out much hope.

By N Barrett. August 11th, 2009 at 2:35 am

It would be delightful if Excel included some useful charting tools (or at least some sensible defaults). But Excel is a one-size-fits-all solution and vaste swathes of the user base use it as electronic graph paper or to note various squiggles at the behest of the accountants. Whilst the user base lacks sufficient skills to appreciate how information can help them do their jobs better, usable visualisations will lack the sex-appeal of funky, dramatic, glossy and imaginative pictures that catch the eye and impress the uninitiated.

Microsoft is doing exactly what it needs to, in order to give its customers what they want. Interestingly, if data visualisation can make a real difference, then you’d expect the current niches of data visualisation to expand, even into the likes of Microsoft. Not sure there’s much sign of that except amongst hobbyists, fanatics and consultants!

By Stephen Few. August 11th, 2009 at 9:36 am

N Barrett,

Do you really believe that “Microsoft is doing exactly what it needs to in order to give its customers what they want”? Clearly they aren’t giving their customers what they NEED (you and I agree on that), but I don’t think they’re even giving them what they WANT. It’s hard to imagine that a majority of customers really want to be faced with a list of 19 versions of a column chart to choose from when they want to display data using vertical bars? Did the customers demand that Microsoft replace the old menu system with the new ribbon interface? If so, why have the customers reacted so negatively? I believe that Microsoft’s product decisions are not being clearly directed by either customer need or want. The process of determining product direction seems to be broken.

Regarding the potential for data visualization to make a real difference, as better tools are becoming available and becoming known, it is becoming more broadly adopted. As more and more people are exposed to these good tools, I believe we’ll eventually reach a tipping point when the silly data exploration, analysis, and presentation tools that are being offered by most business intelligence vendors will be recognized for the lipstick-on-a-pig imitations that they are.

By D Casey. August 11th, 2009 at 10:43 am

I totally agree that Excel doesn’t do everything correct, but I would disagree that it is a terrible tool. Mr. Few, you mentioned tools like Tableau, which is amazing, but those can be restrictively expensive and with enough work you can recreate those same visualizations in excel (minus the map tool for Tableau, that thing is just cool). The ribbon interface that so many people have a vast hate for is easier to use once you get the hang of it. They just don’t like change.
I am in no way suggesting that excel is perfect or even close to it, but arbitrary hate for a product is dangerous (like Mac fan boys).
At least they are trying to take steps toward better visualizations, they could just let things go on as they have and people would still buy the product (see Visio). You have to admit the charting upgrades in 07 were a giant leap forward, even if they did go overboard on the options and 3D..and that’s ok really. I don’t use them anyway. Most people who use their product have never heard of Tufte or Few (or common sense). People will probably even complain about sparklines being included cause they won’t understand them.
Excel does its job of being a cheap, easy(ish), tool to display data in simple forms that laymen can use, but it also allows people who have a better idea of what they are doing manipulate things to a higher standard. It still misses many marks, but its a commercial spreadsheet tool with charting features first, data analysis and display tool second (despite what MS says)

By Stephen Few. August 11th, 2009 at 11:57 am

D Casey,

I didn’t say that Excel is a terrible tool. It does many things well. I use it all the time to keep track of financial information. In fact, I even use it more than any other tool to create charts–not because it’s charting capabilities work well, but because all of my clients use it and I believe it’s important to help them analyze and present data more effectively even if they have nothing but Excel to work with. You will find no examples of “arbitrary hate” in my critiques of Excel or another other tool out there. My criticisms are specific and based on empirical evidence related to real-world scenarios.

I don’t agree that the charting upgrades in Excel 2007 were “a giant leap forward”–far from it. Well before the release of Excel 2007, I took the time to provide the charting team at Microsoft with specific suggestions for how the charting capabilities could be improved in simple and practical ways. They thanked me for my suggestions, assured me that I would be very happy with the new charting capabilities of Excel 2007, and then went on to ignore everything that I suggested. A few improvements, such as better color management and in some cases better formatting defaults, were introduced in Excel 2007. Overall, however, Excel 2007’s new charting functionality has made the choices that people must make when selecting and formatting a chart more difficult by increasing the options, and the likelihood of poor chart design greater by adding visual effects that undermine the clear and accurate representation of data.

The fact that people like you and I can “manipulate things to a higher standard” when using Excel by painfully and time-consumingly working around the product’s deficiencies doesn’t constitute a benefit, but a deficit. No amount of work will allow us to do with Excel what we can do with tools such as Tableau–that is, to explore, analyze, and present data effectively. The fact that, with enough time, effort, and skill we can make a single Excel chart look good–perhaps similar to a Tableau chart–is of limited use, because this ability to do things in bits and pieces can never be translated into the kind of analytical flow that is enabled by a true visual analysis tool.

Excel is not a terrible tool. It is good spreadsheet software. It is not good data exploration, analysis, and presentation software. It could be much more than it is, but the track record of Excel 2007 and the coming Excel 2010 does not bode well for its future.

By David Gerbino. August 11th, 2009 at 5:59 pm

@Casey/@Few,

Stephen Few said “Excel is not a terrible tool. It is good spreadsheet software. It is not good data exploration, analysis, and presentation software. It could be much more than it is, but the track record of Excel 2007 and the coming Excel 2010 does not bode well for its future.”

D Casey said “Excel does its job of being a cheap, easy(ish), tool to display data in simple forms that laymen can use, but it also allows people who have a better idea of what they are doing manipulate things to a higher standard. It still misses many marks, but its a commercial spreadsheet tool with charting features first, data analysis and display tool second (despite what MS says)”

My educated guess is that MS Excel is used by more non-accountants then accounts. Most people I have worked with for over 30 years use the Excel tool for some type of analysis. They can be Finance, Sales, Marketing, whatever. They all use the tool thinking that the software leader of the planet Earth is providing them with a best practices tool set. Have data, make chart, use defaults, I have a best practices chart. Have a data table for a report, use standard formating tools and I have a best practices data table. I can go on but WRONG.

Edward Tufte wrote 4 books, Stephen Few wrote 3 books, Naomi Robbins wrote 1 book, William Cleveland wrote 3 books that I know of. So my point here is, people are writing books about how to show data visually for better interpretation and the team at Microsoft does not seem to care. My only hope is if the High School and University System can knock some sense into people so they know how to work with data, create visual displays of data and to interpret data both analytically and visually.

@dmgerbino

By JKardos. August 12th, 2009 at 2:52 pm

An interesting note is that Microsoft is one of Tableau’s biggest customers. Microsoft’s own accounting dept was quoted on Tableau’s website as a very happy user. At this point, Tableau is a complimentary tool to Excel and vice-versa. Until Tableau starts to include more built-in financial functions, stronger query interfaces that allow for cross-platform data sources in one worksheet and spreadsheet row-column navigation options into their product, Microsoft won’t feel compelled to change anything. Their big revenue products are SQL Server, Sharepoint, Visual Studio…..Since Tableau works very well with Excel and SQL Server and Sharepoint, maybe Microsoft is content to let Tableau have this part of the BI Functionality world?

I’ve found that Tableau leadership definitely listens to its users in order to develop new features and enhancements. They have made important and useful updates with each new version of the product. My fear is that once Tableau begins to add features that start to infringe on too many Microsoft product functions, Microsoft would consider buying Tableau. Hopefully, Tableau will remain independent for a long, long time.

By N Barrett. August 13th, 2009 at 8:29 am

Mr Few!

People shouldn’t want Excel as it is — but people seem to love their 3D pie charts and associated flim flamery and the more the better!

I think people do want it, in the mistaken belief that it is helpful (because it looks engaging, pretty and clever) — and microsoft pampers to that want. And in the same way that people (shouldn’t) want to be faced with 19 different bar chart options, they (shouldn’t) want to be faced with fifteen different types of rice or 100 different brands of beer. But they are still enamoured by the choice, because choice is (apparently) Good and more choice is simply Better… Lunacy, but great for sales!

And whilst the benefit of data visualisation is a truth I currently hold as self evident, I’ve not seen any research that has shown that better data visualisation improves the performance of an organisation. Would love to see any case studies on this topic that are out there!

By Stephen Few. August 13th, 2009 at 10:17 am

N Barrett,

You have raised an important issue: the lack of studies that measure the impact of data visualization in the real world. Studies of this type are more difficult and expensive than laboratory experiments, so they’re rarely done, and often when they are, they are poorly designed. I’m not aware, offhand, of any such studies, which is a shame. We know the value of data visualization because of a great deal of experimental evidence that confirms what it does perceptually and cognitively to support data exploration, analysis, and presentation, which can’t be done in any other way, but studies to confirm how all this adds up to measurable results in the real world seem to be lacking. I suspect that they haven’t yet been attempted. This would be a great project for a university or some other research organization. Unfortunately, tiny organizations like mine lack the resources to pull it off.

By Richard. August 15th, 2009 at 7:58 pm

I don’t think anyone would argue that data visualization is important under the right conditions. The problem is that visualizations are often poorly designed or implemented where they are not needed.

Data visualizations are just another level of abstraction required when our brains can no longer effectively process the raw data. But there are so many situations where the raw data is sufficient and visualization techniques are superfluous. When visualizations are warranted you often get inexperienced BI implementation teams creating dashboards that are ineffective in addressing operational or analytical needs.

If I were a pilot I would want a carefully designed dashboard that allows me to instantaneously determine the state of the aircraft. As a business owner I would want the same thing within the context of my business. The design of an airplane cockpit benefits from millions of dollars in usability research and industry evolution. Also, designers know what pilots need to know to fly. Most corporate dashboards are just “best guesses” developed by technologists with no real understanding of the underlying business problems.

The real problem arises from not adequately understanding the actions that a business owner can take to alter outcomes and what type of information he/she needs to make appropriate decisions. Data visualization are undoubtedly one of the tools that can be used to present data but it in itself is not inherently useful without proper context. But when properly applied they are invaluable.

By Lorre. August 22nd, 2009 at 5:56 pm

I’d like to comment on a number of strings in the conversation above.

On Choice:
N Barrett - Heavens! There are dozens of studies showing that the more choices a consumer has, the faster they give up without purchasing. It’s well documented that you’ll get far better sales if you can reduce choices to three, two of which are similar. If you truly care about detailed studies, considering a quick scan through press releases at the Journal of Consumer Research for the last year, and click through to the studies.

On Dashboards / Data Visualization / True Stories on Value:

In general the discussion between dashboards and data visualization seems to be a little confused. Our firm fixes many failed dashboard implementations, but I wouldn’t consider most of what we do data visualization.

We fix dashboards that have failed typically because they don’t reflect the organization’s key drivers for their strategy’s success. In our work we try to do three things:

1. Focus the information on the elements that truly drive success

2. Limit the number of visualizations we present. Humans brains are limited in what they can keep in mind - most peak out between three and five, the very best at seven. We tell our clients they can have two to three, and allow creep to no more than five

3. Make the visualizations future-focused. When data is historically focused, the nature of management in most organizations, and in particularly in large organizations, is to feature self-aggrandizement, lay blame, and engage in other very human, natural, but ineffective political behaviors. By making visualizations future-focused, we skip over most of that and jump the conversation from “what happened” to “most effective next action”.

The results are dramatic because they shift the conversational focus. For example, for a global sales organization of a Fortune 500, their sales (> $1 billion) increased by more than 20% in just over 6 months. In addition, they saved approximately 6 weeks of staff time from no longer preparing the additional reports that primarily generated arguments and deflected attention from goals.

We have many similar examples, including a tripling or more of sales for many clients with sales in the low millions.

We sometimes very effectively demonstrate the value of presenting the right data in the right way by selecting key questions clearly facing the organization, and plotting the “natural” answer from Excel’s graphs. Then we contrast that with other “pictures” that will actually answer the question or drive the appropriate behavior. As they say, a picture is worth a thousand words. Most executives have no idea what they’re missing, or how much time they’re wasting.

On Microsoft:
I am on one of Microsoft’s advisory board. I agree that there are many bright people at Microsoft who “get it”. But the organization is incredibly hierarchical and bureaucratic. For example, they were very excited about the “power” of a toolbar icon that expressed app/system status in red/green/yellow. When I asked why they would ignore the 10% of their audience that is red/green colorblind with this “fantastic selling point”, I was told incredulously that one of the most senior execs was colorblind, and could not tell the status from the icon either. Yet no on had ever considered making a change because it would be too difficult for their minions of programmers. I suggested a simple flip of the black/white background with the red/green status. You would have thought I had suggested they flatten the earth. Later several people from the development team approached me to praise the idea and express frustration that they would never be allowed to implement such a radical change.

Microsoft cares about knowing the answer, but nobody dares risk the political capital to try to implement. In this economy it seems even more unlikely that anyone would want to risk job security for the good of a company that doesn’t seem to care. On the bright side, our skills will enable us to continue to create compelling advantage for our clients for years to come.

By Jeff Weir. August 24th, 2009 at 1:07 am

Stephen: saying “No amount of work will allow us to do with Excel what we can do with tools such as Tableau–that is, to explore, analyze, and present data effectively” is an incorrect and dangerously emotive statement.

How exactly are you quantifying “No amount of work” here? You mean ‘No amount of work’ by yourself? Or are you extrapolating this to cover the skills of every expert analyst/Excel user out there? Are you sure? There’s some pretty qualified users out there – whereas you can only pass judgement on what is and what isn’t possible to you.

Sure, right out of the box, Excel doesn’t explore, analyze, and present data effectively for the user automatically or intuitively. But reading your latest book, I know that someone with a very good understanding of Excel (including perhaps some VBA,a bit of SQL, and advanced charting stuff) and of information analysis/design principles can explore, analyze, and present data just to the ‘best case’ standards you set out – perhaps as proficiently as someone using Tableu even. Maybe as efficiently as well, maybe not.

In several instances in your latest book where you say ‘Excel can’t do this’ or ‘This other tool is the only one I know of that does that’, you are mistaken. Just because you can’t do it - or haven’t seen it done – doesn’t mean someone else can’t or hasn’t. Excel is programmable, flexible, and easily customisable WHEN YOU KNOW HOW. So much so, that I believe a very cleverly well designed excel interface can – at a pinch - hold its own against many of the other products you cover.

And as the quote goes, we go to war with the army we have, not the one we want. I’ve never had to tell a client ’sorry, I can’t explore, analyze, and present your data effectively because you only have Excel’. I do an excellent job DESPITE the fact I have to do it on Excel.

Don’t get me wrong…I hate many of the choices that Microsoft designers have made regarding this product. But I’m pretty good at finding workarounds, and (like all of us) I’m way more intelligent than any software package is.

By Stephen Few. August 25th, 2009 at 7:52 am

Jeff,

My statement that “no amount of work will allow us to do with Excel what we can do with tools such as Tableau” addresses the capabilities of Excel, not my own. Excel lacks many of the capabilities that are at your fingertips in a well-designed tool like Tableau. For example, Excel does not provide dynamic filters, visual crosstab displays, linked highlighting among multiple charts, geo-spatial displays, or automated switching between different types of charts based on the nature of the data, to name a few features that are necessary for data exploration and analysis. For data analysis, these features don’t just need to be possible, they need to be at your fingertips, without the need to spend hours or days writing code to create a specific instance of the feature. If it takes hours or days of sophisticated effort to create an instance of something in Excel that could be accessed in an instant with a tool like Tableau, it won’t get done–certainly not in the context of ad hoc data analysis, which only works when you can get to the next view of the data that’s needed within seconds. The money that would be lost in time trying to create one of these views in Excel that are automatically available in Tableau could be used to buy a copy of Tableau. Given this fact, the argument that “we go to war with the army we have” loses its ring of sensibility.

Excel is “programmable”, but to a limited degree, “flexible”, but only with a great deal of programming, but definitely not “easily customizable”, except in minor ways. I count as my friends several of the top Excel experts in the world. Their value to the world as experts stems from the fact that much of what we need from Excel is not available and not easy to produce through workarounds and programming. It’s likely that much of the work that you do for your customers could be done by them without your help if they had a better tool.

You are clearly more intelligent than any tool. It’s unfortunate that so much of your intelligence must be applied to working around the limitations of a poor tool, however, rather than the real work of data sense-making.

By Jeff Shanahan. August 25th, 2009 at 11:34 am

I’m sure most reading this post are already aware, but there’s a handy Excel add-in product called Microcharts which addresses a lot of Excel’s visualization deficiencies. It can even do the ‘bullet graph’ invented by Stephen. It’s worth looking into. (Note: I have no association with Microcharts, just a casual user)

By Jeff Weir. August 25th, 2009 at 4:39 pm

Hi again Stephen. Thanks for taking the time to respond. I wrote my comment because I believe that simply saying “No amount of work will allow us to do with Excel what we can do with tools such as Tableau–that is, to explore, analyze, and present data effectively” is a fairly unhelpful blanket statement…the verbal equivalent of using a rotated 3d chart that obscures some other relevant stuff that we should probably also consider.

I take from your reply that you are effectively saying that “Excel requires a significant amount of work to set up a lot of the stuff that we can more easily do with tools such as Tableau with regards to exploring, analyzing, and presenting data effectively”; and not simply that it can’t be done. That is a rather different statement, and one that I could agree with, with qualifications.

I’d like to comment on a few extra things you’ve raised in your reply. Sorry about the length of this, but I think it’s a worthwhile conversation. Here goes:

You said that ‘Excel lacks many of the capabilities that are at your fingertips in a well-designed tool like Tableau.’ I really think that you’ve got to preface that statement with ‘Out of the box…’.

I disagree that Excel is programmable only to a limited degree. My view of Excel is that it is programmable to a remarkably large degree – and in fact many people use customised instances of Excel (i.e. fully fledged applications) with no idea that they are using Excel at all. However I agree that Excel is flexible ‘ only with a great deal of programming’, and that it is definitely not “easily customizable”. Granted, most Excel users can’t do this kind of development themselves. But most Tableau users couldn’t assemble the code that runs Tableau’s interface from scratch either. Thankfully they don’t have to, because developers can do it for them.

I’m positive that Excel IS capable of most of the things you mention, with some exceptions that may or may not matter depending on the requirements of the specific analysis at hand. Granted, out of the box these capabilities are nowhere near your fingertips…you really have to roll your sleeves up and get your hands dirty in the innards of the beast in order to implement this kind of functionality. You need VBA as well as excellent knowledge of what is possible, and lots of time to play around to implement it. But once you’ve worked out how to do something, then not only is your knowledge extended, also what you’ve just built is easily replicated, and hugely customizable.
Using Excel, a sophisticated user can set up tools like dynamic filters and visual crosstab displays (I’m assuming this means a matrix of say scatterplots that look at 2 variables per scatterplot over a multitude of variables…I don’t have any reference books immediately to hand to check my terminology).

I don’t see any reason why you couldn’t set up linked highlighting among multiple charts if you are clever enough with some VBA, dynamic ranges, and pivot tables…I’ve been thinking about how cool it would be to implement this functionality since I read about it in your book recently, and once I’ve finished my current bit of work I’ll see if I’m Excel-savvy enough to implement it.

Granted, Excel is no geo-spatial tool. But then I’ve never done any Geo-spatial analysis. I agree I’ll likely need to step up if and when the time comes to a more dedicated bit of kit.

As for automated switching between different types of charts based on the nature of the data, you could probably write some VBA if you wanted…but then I don’t want this kind of functionality…I prefer to change things myself than rely on some preprogrammed logic. I think that automated switching might be nice if you want it, but I hardly think it’s so important that it’s ‘necessary for data exploration and analysis’.

I also don’t agree with your premise that if something takes longer than mere seconds (as can often be the case in Excel) then it ‘won’t get done–certainly not in the context of ad hoc data analysis, which only works when you can get to the next view of the data that’s needed within seconds’.

While I DO agree that the faster you can get to the data you need, the better, I disagree that adhoc analysis should be measured in seconds in the first place in order for it to be worthwhile and efficient. Ad-hoc means “for this purpose”…It doesn’t mean “right this instant”. Much of the adhoc analysis I perform for organisations takes days, if not weeks. The time taken to switch data views is not a significant limiting factor, if a limiting factor at all (dynamic graphs and pivot tables and the little known ‘custom view’ functionality built into Excel renders this pretty much well painless). Rather, the limiting factor in terms of my time is usually firstly getting the management to be very clear about what they are wanting, and secondly documenting the existing information systems so I can have some sort of confidence that when I finally get access to some data for analysis purposes I can be reasonably confident that I know it’s pedigree…i.e. exactly what the data is, and what it isn’t. This has on many occasions taken waaaay longer than any analysis itself…and the choice of analysis tool up to this point is irrelevant. Furthermore, some of the systems I’ve been working with have practically zero documentation, the IT people have very little recall as to what is what, and the management…well, let’s not go there. Compared to delays from these things, then switching data views faster than I already can is not high on my frustration list.

You’re absolutely correct that it takes hours or days of sophisticated effort to create an instance of something in Excel that could be accessed in an instant with a tool like Tableau. You’re probably right from the viewpoint of the organisation I’m working with currently that ‘The money that would be lost in time trying to create one of these views in Excel that are automatically available in Tableau could be used to buy a copy of Tableau.’ But this ignores the fact that while there may be a sunk cost in terms of time taken in working out how to do something tricky in excel, the cost of replicating it on the next job I do is pretty close to zero thanks to the fact that I now know exactly how to it, and can cut and paste from the last example in a matter of seconds, and then customise it to suit. So the organisations I work with get quite a bit of fiddly stuff at little cost, because I once put in the time to learn how to do it. I don’t have to reinvent the wheel for each new job – and any sunk cost of additional Excel fiddling is very minimal compared to the overall project budgets they are spending…including on the time I need to put in prior to any actual analysis. As I said above, the actual analysis is usually the quickest bit.

Given this, switching to Tableau isn’t going to save the companies I work with significant amounts of money. If they did switch, they will likely find it more difficult to recruit and retain an analyst that uses Tableau at reasonable cost, compared to one that is very proficient in advanced Excel. I’m from New Zealand, and it’s a very small job market here – we only have 5 million people in total. For every person that uses Tableau, there’s gotta be at least 100 people that use Excel to a good enough standard to achieve whatever analysis the average organisation needs. And that’s assuming that you can even persuade the organisation to do the analysis on something other than Excel in the first place. They’d just say “well, can you do the job with what we’ve got or not?” and I’ve have to answer ‘yes’. They already have Excel loaded on damn near every PC. Their IT departments already support it. It’s just too damn easy to stick with the status quo. Luckily, that status quo gets them past the finish line.

So I think we can sensibly go to war with the army we have, and still kick ass if we have smart soldiers. Yes, Tableau might help get to ‘Mission Accomplished’ a bit faster. But it’s not a critical factor.
I don’t think for a moment that much of the work i do for customers could be done by them without my help simply if they had a better tool. To me that’s like saying that if every senior manager had a copy of ‘Now You See It’ on their desk then they’d suddenly start building reliable econometric models themselves. I don’t get hired because of the tool I use. Just as well, because I’m not even a black belt compared to many of the excel ninjas out there. I get hired because I’m a good problem solver, and because I’m good at presenting complex information to my bosses in a manner that they can then relay to THEIR bosses, and because I don’t have the distractions they have, and/or because they simply don’t have the resources in house to do something.

Despite using Excel for the analysis I do, not much of my intelligence ‘must be applied to working around the limitations of a poor tool’. Yes, SOME of my intelligence must be applied to working around the limitations of a less than perfect tool. But I think that helps me become a smarter guy, generally…not just a smarter Excel guy. Unfortunately, given some of the hurdles mentioned above, not enough of my intelligence actually gets used in the actual activity of data sense-making. Most of it gets used in stuff that is largely irrelevant to whatever tool – other than my brain - is at hand.

Where you say ‘much of what we need from Excel is not available and not easy to produce through workarounds and programming’ I’d be inclined to replace ‘Excel’ with ‘good data analysts’.

Regards,
Jeff

By Michael Nagy. August 25th, 2009 at 5:57 pm

Stephen,
Icon sets have been improved in Excel 2010. For each of the conditions you set you can now select one of 50 odd icons or you can choose NOT to have an icon so you don’t need an icon in every cell that you conditionally format. That means you can now highlight bad results only without having to highlight the good results as well - as you show in your book on Information Dashboard Design. I think this has changed eye candy into a useful feature.

By Tom Ryder. August 26th, 2009 at 12:54 am

“By Jeff Shanahan. August 25th, 2009 at 11:34 am
I’m sure most reading this post are already aware, but there’s a handy Excel add-in product called Microcharts which addresses a lot of Excel’s visualization deficiencies. It can even do the ‘bullet graph’ invented by Stephen. It’s worth looking into. (Note: I have no association with Microcharts, just a casual user)

Jeff,

Unfortunately, MS have also managed to break microcharts.

Microcharts is of course font based, so by and large when distributing microcharts content, you have to pdf it (how many clients are going to install the font range across their systems?)

A recent “security update” from MS causes excel to crash whilst it pdfs.

The security update? Designed to prevent people doing useful stuff with fonts.

Strangely, it came through automatically at about the same time that MS announced the sparklines in excel, but I’m sure that’s just coincidence…

Any way, if you use microcharts, remove KB961371 and KB961371-v2 and avoid installing any service packs that bundle this.

Apologies for the unsolicited tech support!

Tom.

By Stephen Few. August 26th, 2009 at 9:30 am

Jeff,

I appreciate your efforts to help people stretch Excel beyond its limits through workarounds and custom programming. Similarly, I help those who read my books and attend my courses to analyze and present quantitative information as much as possible with Excel when it’s the only software they have for these purposes. I don’t encourage them to extend Excel beyond its limits, however. Excel is an electronic spreadsheet, not a data analysis tool. Rather than encouraging people to work around the limitations of Excel or to pay a consultant to develop custom programs to extend its capabilities, I encourage them to work smarter, more effectively, and more cost effectively by using a tool that was actually designed for the task.

I chose my words carefully when I wrote that “No amount of work will allow us to do with Excel what we can do with tools such as Tableau.” Applications that you’ve written in Visual Basic to significantly customize or extend the capabilities of Excel are no longer Excel in any practical sense. Even if we count them as Excel, however, my statement is still accurate. I spent months working with two of the cleverest, most sophisticated Excel experts in the world to develop an add-in product that I designed to simplify, correct, and extend Excel’s charting capabilities through any means possible. Eventually, I abandoned the effort because their phenomenal work was never able to produce a product that I found acceptable. This failure was entirely due to limitations in Excel. To extend Excel through custom programming, you must work through the application programming interfaces (APIs) that are provided for the various versions of Excel. When you interact with the product at this level you discover that, under the hood, Excel is very messy and quite limited. To approach the capabilities of a good data exploration and analysis tool, you would be better off writing an entirely new application from scratch rather than trying to customize Excel.

Jeff–you are clearly a smart and talented fellow, but your experience appears to be limited. Based on what you’ve written, it appears that you’ve never worked with a good data analysis and presentation tool. Your tenacity is heroic, but Sisyphian in nature. Sticking to Excel is too much like insisting to push a huge boulder up the same hill every day. If your analytical skills were applied using a tool such as Tableau, you would spend much more of your time actually solving problems with data rather than problems with software.

You made it obvious that you haven’t had a chance to work with Tableau when you wrote: “Switching to Tableau isn’t going to save the companies I work with significant amounts of money. If they did switch, they will likely find it more difficult to recruit and retain an analyst that uses Tableau at reasonable cost, compared to one that is very proficient in advanced Excel.” It is much easier to learn how to use Tableau than it is to learn how to use Excel. The interface is elegantly designed. Nothing is in the product that doesn’t work well and isn’t of great value. Perhaps you’ve forgotten the full extent of the effort that you’ve put into becoming proficient with Excel. With a tool like Tableau, rather than expending enormous effort learning the software, people are encouraged to spend more time learning data analysis, which is as it should be.

Another of your statements reflects a tragic lack of experience. Regarding your customers’ exclusive use of Excel you wrote: “Luckily, that status quo gets them past the finish line.” In this, you are wrong. Your clients who are using only Excel for data exploration and analysis have barely crossed the starting line. They have yet to catch a glimpse of the finish line and never will with Excel.

By Jeff Weir. August 26th, 2009 at 3:46 pm

Hi Stephen. Me again. I have to admit that there’s little in your reply that I don’t agree with. For instance:

- I’m sure I would be better off using Tableau or (if I could) building another application from scratch to approach the capabilities of a good data exploration and analysis tool, rather than pushing Excel beyond what it was intended to do.

- My experience IS limited (although there’s nothing ‘tragic’ about that : a true tragedy with regards to experience is an inability or unwillingness to gain more of it than you currently possess).

- Excel has definitely been a major undertaking to learn so far, and I’m still only scratching the surface of it in many areas.

- I certainly do feel like I’m pushing that rock uphill on many occasions when I work with Excel.

- I haven’t worked with a dedicated data analysis and presentation tool.

But what I don’t agree with is the sheer ‘absoluteness’ of some of your prepositions. For instance, take your statement that ‘Excel is not a data analysis tool’. Following this kind of logic, one could say that because a bit of bone was fashioned by its maker (i.e. the body of the animal it comes from) to support the body but not to be a fishhook, then it can’t be a fishhook.

Tools are defined (sometimes differently by each of us) by the use that we make of them. But to me, some of your language reads like you are saying no one can use Excel as a tool to analyze, and present data effectively, and no one ever will. You don’t seem to be limiting this statement as a comparison between Excel and Tableau (in which Tableau would win, I’m sure). But good ideas, analysis, and recommendations are good whether they were conceived on good tools or great ones…or even poor ones…provided there’s no lost opportunity to follow up on that advice due to the time of arriving at it on one tool versus another.

Looking at it another way, 2 different analysts working independently with the same data could come up with the same recommendations whether they used Tableau or Excel; and their report could influence decision makers the same way within the same project timeframe. How can we say here that the analyst that happened to use excel didn’t cross the finish line? (and how are we defining ‘finish line’, anyway? Either we managed to cross it with both Excel AND Tableau, or we didn’t managed to cross it with either.

A really smart analyst using an abacus and graph paper can provide meaningful advice and insight for a particular problem. A poor analyst using Tableau can provide really bad advice for the same problem. A really smart analyst could save himself a lot of pain by upgrading to a product like Tableau when working on that problem. But that analyst might come up with the same advice as they did with the abacus and graph paper, only easier.

Unfortunately, not very many organisations here in New Zealand use Tableau (yet). An analyst would have to take it on faith that the time cost of becoming proficient in Tableau would be worth it, as well as the dollar cost of purchasing additional kit. Payoffs from doing so are less than certain, given there are zero jobs currently advertised for Tableau on the premier job board in New Zealand. In fact, I’ve never seen Tableau advertised, and I keep an interested eye on this job board. But then, as with Excel, practicing your problem solving skills on any tool brings its own reward…and someone who can demonstrate abstract problem solving is half way to getting employed no matter the tool. So I’d certainly urge an analyst to check Tableau out. I’ll certainly download it, and give it a spin…and I’m sure I’ll like it from what I’ve seen in your book. But unfortunately that doesn’t mean I’ll get to use it on someone else’s PC other than mine any time soon.

By Matt. August 26th, 2009 at 5:54 pm

@Jeff Shanahan; @Tom Ryder
- You can make bullet graphs in Excel without 3rd party add-ins or VBA; e.g., the Charley Kyd method, as well as others.

re: Tableau vs. Excel
Echoing some previous posts, many of the Tableau charts in “Now You See It”, such as crosstabs or weekend/weekday backgrounds, are possible to make in Excel without add-ins or VBA. Advanced charting skills and (initial) set-up time are needed. Certainly not as dynamic as Tableau, but the static visualization is the same.

What would be interesting is to see some commentary on how hard it is to make Tableau Reader available to corporate environments outside of your own and over which you have no control.

Internal adoption of Tableau is easy enough, but what about the report you’re sending to multiple persons in multiple external corporations who don’t have Tableau Reader installed, who don’t have admin rights on their pc, who have to go through a bureaucratic maze to install anything, etc.]?

And, even supposing everyone outside your office did have Tableau Reader, I can still imagine the scenario of an outside person wanting a report in Excel because that’s they’re program of choice and they “have to” have the report in Excel.

By Stephen Few. August 27th, 2009 at 12:36 am

Jeff and Matt,

I want to respond thoughtfully to your comments, but I’m on vacation this week and next and must rush off to the next location where I might not have Internet access for a few days. Please be patient; I’ll get back to you in a few days. This discussion is keeping my brain sharp in a beautiful place (Sardinia) where I would otherwise be tempted to let it wander aimlessly.

Steve

By Stephen Few. August 28th, 2009 at 7:24 am

Jeff,

If, in a pinch, I pick up a stone and use it to drive a nail, I don’t call the stone a hammer; it is still a stone. If I use Excel to clumsily explore and analyze data to the degree that it allows, I don’t call Excel a data analysis tool; it is still a spreadsheet. Interacting with data to discover and understand the meanings that reside therein, using only Excel, is much like seeing the world while confined within a burka–our field of view is narrow and a thin layer of fabric always clouds our vision.

Many women in the Middle East who have been induced from adolescence to wear the burka choose not to give it up when given the opportunity. They cling to the familiar. Similarly, many people who work with data choose to hold tight to Excel for ill-suited purposes, even when offered greater freedom and power through better tools. Excel feels comfortable and safe, but it imprisons them. It is a refuge from a changing world of possibility. Women who wear the burka might fear giving it up for good reason–zealots are determined to keep them down by violent means–but those who need a better data analysis tool than Excel have no real reason to fear but fear itself.

Excel is a fairly good electronic spreadsheet. Because almost everyone has it, Excel also provides a convenient means to exchange small data sets. For these purposes, it is a handy tool. It is tragic, however, to insist on using it for data exploration and analysis when true data analysis tools are now available. Why would anyone choose to work clumsily and inefficiently, wearing blinders, when they could soar? Data analysis software worthy of the name invites us to become immersed in information, experiencing analytical flow, seamlessly interacting with information to uncover its stories; the software doesn’t get in the way. With Excel, however, we are constantly interrupted by the mechanics of using the software, never allowed to keep our minds in the data for long. When forced to work in this way, we’re inclined to settle for easy answers, because our desire for the truth is worn down by the one-step-forward-two-steps-back effort to reach it.

Neither Excel the product nor MIcrosoft the software company have earned our loyalty. Why do we wear ourselves out trying to work around Excel’s limitations, correct its problems, and extend its capabilities? By doing so, we stand in the way of our own progress. We willingly don the burka and encourage others to do the same. No one who has had a chance to work with a true data exploration and analysis tool would ever again choose to use Excel for anything but spreadsheet tasks.

The analytical finish line isn’t the completion of a single analytical task. A simple task can certainly be accomplished with Excel, just as a single nail can be driven into a board with a stone. Compared to what people could accomplish by supporting their analytical skills with a good tool like Tableau, however, those who exclusively use Excel have barely crossed the starting line, yet they might believe they’ve won the race.

Jeff–you’re in a great position to show your clients better ways of squeezing value from their data. If all they have is Excel and they’re not looking for a better tool, then you should provide them with an Excel-based solution, but you can also introduce them to better alternatives along the way. If you do this, you’ll be doing them a favor; you’ll be providing them with a higher level of service. It is through the efforts of people like you that we’ll reach the tipping point that will usher in a new world of evidence-based decision making. You have a chance to be a part of this. Your work in New Zealand could make a big difference there. The shift is beginning to occur here in the United States, but it will still take a while to turn this huge ship. New Zealand is smaller and more agile. Why not help your country lead the way?

By Jon Peltier. August 28th, 2009 at 7:35 pm

I’ve been following this discussion with great interest. I’d like to describe my experience in corporate life, and why I’m such an expert in Excel.

In a rare few forward thinking organizations, I’m sure Tableau and other fine tools are available, and there is sufficient time given to learning their proper use.

Most organizations have barriers to such tools. 1. Excel has been used for so long and everyone has it already installed, so the decision makers can envision nothing else. 2. The IT people warn of problems installing, maintaining and supporting these tools. 3. The initial cost of acquisition of these tools may quickly be made back through improved productivity and improved decision making, but it’s still a barrier that prevents their purchase. 4. The learning curve, however short, keeps many from using these tools. If it requires only half a day away from normal activities, people can’t spare the time.

In corporate America, I had access to no advanced data tools, but I had Excel. I was already using it as a platform for regular worksheet-related tasks and for small utilities that retrieved information from external files, so tweaking the data and charts was a natural extension. If I needed a probability scale, I had to figure out how Excel could do it. A Gantt chart? Excel again to the rescue.

Excel has two great strengths: it is everywhere, and it is as flexible as the user is imaginative. I know that if I build a utility in Excel, it can be used by almost anyone with Excel on their desktop. I also know that I can usually figure out or Google a solution for whatever problems I might encounter.

Excel’s main weakness is the lack of interest of the Microsoft higher-ups to support real development in areas where Excel is lacking. Another weakness is the lack of appreciation at Microsoft for the flexibility they built into Excel, and for the myriad ways Excel is extended. The continued development of the pervasive and invasive graphical features could be considered another weakness, or at least a major distraction.

By Elissa Fink. August 28th, 2009 at 7:38 pm

Matt, to answer your question about how hard it is to make Tableau Reader available to corporate environments outside of your own and over which you have no control, I know of several Tableau customers who are in fact doing that. (I’m the VP Marketing at Tableau by the way.)

For example, analysts at The Martin Agency, an interactive advertising agency, produces weekly dashboards for their clients. Producing the dashboards now takes a small fraction of the time it used to take using Excel. But as importantly, clients now get a highly interactive, visual set of dashboards and reports they can use easily understand and use. We actually have a screenshot on our website (http://www.tableausoftware.com/learning/applications/studies/the-martin-agency-client-reporting-business-dashboards).

Steve Wexler of I4CP (the Institute for Corporate Productivity) has been producing amazing analytics of I4CP’s surveys for quite awhile now. A primary mode of distribution of these analytics to clients is Tableau Reader. You can check out a couple of samples at http://www.i4cp.com/company/interactive-data.

Cyon Research is doing something similar with its survey of engineering software users. They’re providing their incredibly detailed survey in Tableau Reader to customers (mostly business executives and managers) so customers can slice and dice the data rapidly and easily on their own.

Of course, it can be a challenge to ask a third party to install a free software application. But for the customers we know about, it certainly seems like they’ve had success.

I’d love to hear more informtion - good and bad - from other Tableau customers who are working with or want to work with external constituencies using Tableau analytics and Tableau Reader. Please email me at efink at tableausoftware.com.

By Matt. August 29th, 2009 at 9:49 am

@Jon
Thanks for the sharper details about Excel’s weaknesses and strengths.

@This discussion as a whole
It seems to me this discussion has some implicit assumptions that have skewed the conversation’s direction. This has been bothering me because, from experience, one assumption in particular fails to ring true.

Is Tableau a superior tool for “multi-facted data analysis”? Yes.

Excel allows for data exploration as well, but is it as easy to implement the exploration as in Tableau? No, Tableau wins - no contest.

Can you make charts in Excel exemplifying good or even excellent data visualizations? Yes.

Can the visualizations you make in Excel look as good as the visualizations in Tableau? In some cases, No. In some cases, Yes - including cross-tabs.

Can you make professional looking dashboards in Excel and distribute them to customers without using VBA or add-ins? Yes.

Now, to the heart of the matter: Does every dashboard or report you make have to allow for data exploration and in-depth analysis? No. Support for this claim comes from pages 40-42 of “Information Dashboard Design”, most notably pg 41:
—————————————-
“Dashboards for strategic purposes

The primary use of dashboards today is for strategic purposes. The popular “executive dashboard”, and most of the dashboards that support managers at any level in an organization, are strategic in nature. They provide the quick overview that decision makers need to monitor the health and opportunities of the business. Dashboards of this type focus on high-level measures of performance, including forecasts to light the path into the future…

…these dashboards don’t require real-time data; rather, they benefit from static snapshots taken monthly, weekly, or daily. Lastly, they are usually unidirectional displays that simply present what is going on. THEY ARE NOT DESIGNED FOR THE INTERACTION THAT MIGHT BE NEEDED TO SUPPORT FURTHER ANALYSIS…[emphasis added].”
—————————————-

In cases such as these, Excel can produce excellent visualizations in a distributable format where the only time-intensive process is the initial idea/set-up of the .xls template.

By Stephen Few. August 29th, 2009 at 3:24 pm

Matt,

In my original blog post and all of my comments that have followed, I’ve been speaking specifically about Excel’s failure as a tool for analytics. If you know what you’re doing and buy an add-in product such as Microcharts, you can design an effective dashboard (that is, a monitoring display) with Excel. The result is not a robust dashboarding solution, because the database interactivity, scheduling, and distribution possibilities are limited, but it’s not a bad way to start if you need a dashboard and can’t afford betters software.

So, for some forms of reporting, including dashboards, you can produce reasonable results with Excel, provided that you are skilled in chart and dashboard design. For data exploration and analysis, however, you can do little with Excel. To say that “Excel allows for data exploration as well” is misleading. Excel’s data exploration capabilities are almost non-existent. Data exploration requires easy access to data and the ability to modify the visual display immediately (filtering, drilling, aggregating to various levels, small multiples, automatic highlighting across multiple views of the same dataset, etc.), none of which are supported by Excel.

The primary reason to move on from Excel is not its clumsy and poorly designed data presentation capabilities, which we can work around without killing ourselves, but its almost total lack of data exploration and analysis capabilities. Without these, having the ability to present data alone doesn’t give us much to go on. Before you know what to present, you must first make sense of your data. For that, there’s little that Excel can do to tame the deluge of data that most of us face.

By jeff weir. August 30th, 2009 at 9:48 am

thanks for your reply Stephen. I guess a large part of Excel´s success lies in the network effects of its widespread deployment…and certainly not in its inherrant capabilities.

I´m much more comfortable with your turn of phrase that “Compared to what people could accomplish by supporting their analytical skills with a good tool like Tableau…” as it leaves me some wiggle room!

Jon - when you say Excel “…is as flexible as the user is imaginative” I think you REALLY meant to say that it “…is as flexible as the user is persistant.” ;-)

Tell you something, learning Excel may be hard, but learning Spanish is much harder. AH well, can`t really complain about holidaying in the Canary Islands, can I?

I hope you enjoy Sardinia as muh as I`m loving Tenerife, Stephen.

By Matt. September 24th, 2009 at 5:54 pm

Just came across this post on the Excel Team blog: http://blogs.msdn.com/excel/archive/2009/09/23/easy-and-even-fun-data-exploration-introducing-excel-2010-slicers.aspx

Maybe someone knows if these “slicers” are a real advancement or just an insignificant step for Excel’s “data exploration” capabilities?

By Stephen Few. September 24th, 2009 at 6:01 pm

Matt,

I believe that these slicers (filters) are part of Project Gemini, which will be sold by Microsoft as an add-in to Excel 2010. Compared to the dynamic filters that exist in products such as Tableau, Spotfire, Panopticon, and a few others, these slicers are primitive in design and limited in function.