Federal CIO Dashboard: We Can and Should Do Better Than This

Over the past few months, the Obama Administration has worked to apply technology to our nation’s problems and opportunities. I applaud the efforts of our recently appointed Federal CIO, Vivek Kundra, to invest more wisely in technology and to make useful data more available both within government and to the public. While welcoming and encouraging these efforts, it is important that we critique their effectiveness as well and speak up when they could be significantly improved. It is in this spirit of patriotism that I would like to point out flaws in the new Federal IT Dashboard that is currently available in beta release. As someone who has designed a great many dashboards, I can say without reservation that the Federal IT Dashboard is about as useful in its current form as a typical business dashboard, and this isn’t a compliment. Others have written about the Federal IT Dashboard in articles and blogs with nothing but praise. Although it’s tempting to rain nothing but praise on a child who’s performed poorly in the past when he makes an effort to improve, it’s important to supplement that encouragement with instruction as well, if you really care. Kundra states on the website: “We tapped the brightest and most innovative minds from Federal agencies, Congress, independent oversight organizations, and the private sector as we built the IT Dashboard.” The project team apparently failed to tap anyone who has expertise in quantitative data analysis and presentation—data visualization in particular. On the dashboard’s website, Kundra invites suggestions. I think it’s time for us who have the expertise that appears to be lacking in the dashboard’s design to lend a hand.

When we initially access the Federal IT Dashboard, here’s what appears on the site’s home page:

The pie chart and its three companion bars on the right automatically morph every few seconds to display a few measures of a different government agency’s IT projects. Unfortunately for those of us who might actually like the time we spend on this page to produce something useful, neither the slices of the pie nor the segments of the bars are labeled, so we have no idea what we’re seeing. Perhaps the home page was meant to function only as an opening splash page of sorts and we must go elsewhere for actual information.

Let’s select the Investments tab at the top and hope for something useful.

Aha! Here we see the pie chart and bars from before, but this time the parts are labeled. Now we’re getting somewhere. Well, actually, we’re not getting anywhere without a great deal of unnecessary effort. Why are the charts three dimensional? Despite their unfortunate popularity, three dimensional displays of two-dimensional data are not only superfluous, they also undermine the simple task of graph perception and comprehension. As Edward Tufte would say, this is “chartjunk.” It breaks one of the basic rules of data presentation: “Do no harm.”

Those of us with expertise in quantitative data displays almost unanimously despise pie charts. The one thing they have going for them is the clear message that they’re displaying parts of a whole. It would help, however, if we could actually compare those parts by comparing the slices of the pie, but visual perception isn’t tuned to compare areas effectively. It is, however, highly tuned to compare the lengths of bars. Had the percentages of the projects that fall into the three categories of “normal,” “needs attention,” or “significant concerns” (see the legend at the bottom) been displayed as three separate bars with a common starting position and labels to the left, rather than pie charts, we could have easily compared these percentages. As it is, to make sense of the pie chart we must keep referring to the legend and then read the numbers that appear next to each slice, because the pie doesn’t do the job on its own.

We’re faced with a similar problem when we try to use the three stacked bars to understand “project costs,” “schedules,” and “CIO evaluations,” because we can’t effectively compare segments of a bar arranged end to end. Three separate horizontal bars for each set of measures (for example, “Costs”) arranged one above the other with a common starting point, on the other hand, would be easy to compare.

Even if the information were displayed using appropriate graphs, it would still be of little use because we derive meaning from quantitative information primarily through comparisons, but for any of these measures we can only compare values related to the three qualitative states of projects—”normal,” “needs attention,” and “significant concerns”. At any one moment we can only see either all agencies combined or a single agency, but never multiple individual agencies which prevents us from comparing them, and we can only see one point in time, which prevents us from comparing what’s going on now to the past to observe how things have changed.

If we wish to compare service groups and agencies, however, we can move to another page, which displays IT projects in the form of a treemap.

Using this treemap, we can roughly compare projects among different service groups by using the sizes of rectangles to compare one measure (total IT spending in this example) and the colors of rectangles to compare a second (% change in IT spending in this example). If the treemap were better designed, we could now get a fairly good overview of how projects among service groups compare, but a couple of problems make it tough going. In the treemap above, projects are organized into four service groups: “Services for Citizens,” “Management of Government Resources,” “Service Types and Components,” and a truncated category that begins with “Support Delive…”). Unfortunately, if we want to identify individual projects in these categories, we must hover with the mouse over each in turn to get the name to appear in a tooltip window.

If we drill down into a particular service group by clicking it, we can see projects in that service group organized by agencies (”Defense and National Security,” “Health,” etc.).

Based on this view, however, can you actually see the boundaries that separate one agency from another? For some reason, the borders that separate them have become partly obscured. Eventually we can drill down to a level in the hierarchy where a treemap is no longer the best way to view the projects because the number of them could be more easily compared using one or more bar graphs, but this option isn’t available. And finally, when we’ve drilled down to the lowest level—a single project—the treemap view is entirely useless, as you can see below. The unlabeled big gray rectangle tells us only that spending on this project—whatever it is—didn’t change much since the previous year. Perhaps it didn’t even exist in the previous year.

Below the treemap in the bottom left corner we have the ability to change the colors that are currently being used to display percentage change in IT spending ranging from -10% (blue) to +10% (yellow). This ability is useful for ad hoc data analysis when flexibility is needed to respond to unanticipated conditions , but on an analytical application like this, which has been designed to display a specific set of measures for a specific set of purposes, it would make more sense to select a color ramp that works well and resist complicating the dashboard with choices that aren’t necessary.

If we wish to see how spending on federal IT projects has changed over the years, we can proceed to the Analysis section of the dashboard and select Trends. The first of two displays that are available for viewing time-series data is an animated bubble chart, which attempts to use the method popularized by Hans Rosling of www.gapminder.org.

The strength of this approach is when it’s used to tell a story. When Rosling narrates what’s happening in the chart as the bubbles move around and change in value, pointing to what he wants us to see, the information comes alive. Animated bubble charts, however, as much less effective for exploring and making sense of data on our own. I doubt that Rosling uses this method to discover the stories, but only to tell them once they’re known. We can’t attend more than one bubble at once as they’re moving around, so we’re forced to run the animation over and over to try to get a sense of what’s going on. We can add trails to selected bubbles, which make it possible to review the full path these bubble have taken, but if trails are used for more than a few bubbles the chart will quickly become too cluttered. Essentially, what I’m pointing out is that this is not the best way to display this information for exploration and analysis. A simpler display such as one or more line graphs would do the job more effectively. Perhaps you’re concerned that a line graph couldn’t display two quantitative variables at once, such as “Total IT Spending” and “Percent Change in IT Spending,” which appear in this bubble chart. Assuming that two quantitative variables ought to be compared as they change through time, two line graphs—one for each variable—arranged one above the other, would handle this effectively. One of the fundamental problems with the bubble chart above, however, is that the two quantitative variables that appear in it really don’t need to be seen together. There is no correlation between total IT spending and percentage change in IT spending from year to year, so there’s no reason to complicate the display by viewing them together.

Even if this animated bubble chart were a good visualization choice in this case, several problems in its design would undermine its usefulness. When I first look at it, I was puzzled for awhile about what “03. % Change in IT Spending” meant. I couldn’t understand the significance of “03. %…” It took awhile to figure out that each variable that appears on the graph was numbered, beginning with “01.” and ending with “05.”, which was completely meaningless and confusing.

Unlike the intuitive use of colors to that we saw in the treemap, the rainbow of colors that appear in the bubble chart are ineffective. The order of the various hues as they change from red to blue is not intuitive. Take these colors and ask people to put them in order from high to low and you’ll get a variety of answers.

Also, the ability to switch the quantitative scales from linear to logarithmic certainly makes sense to people who have been trained in statistics, but is confusing to most of the folks who would use this dashboard. For this reason, I believe this feature should be removed. While it is appropriate to include such functionality in a general purpose data analysis tool, custom analytical applications like the Federal CIO Dashboard should eliminate features that aren’t commonly useful and are potentially confusing in an effort to keep the application simple. Even those who understand how to use a log scale don’t need it available on this dashboard, because few of them would be satisfied using this bubble chart, but would rather download the data and explore it using a better analytical tool.

For those who recognize the limitations and flaws of the bubble chart, an alternative in the form of a bar graph is available. For our entertainment pleasure, when switching between the two, the bubbles morph into bars before our eyes and line themselves up along the horizontal axis.

The bar chart version is just plain silly. None of the bars are labeled until you click on them one at a time to make labels such as “Education (Dept of)” and “Homeland Security (Dept of)” appear. Knowing only the identity of the selected bars (the others remain unlabeled) and watching the bars move around as spending changes through time is eye-catching but almost totally meaningless. Once again, simple line graphs for comparing changing values for the selected items would do the job much better.

Because I wanted to learn something more useful about federal IT spending, I decided to take advantage of the data feeds that are provided, but once again ran into a wall. Unfortunately, the information that can be downloaded is limited to a current year’s snapshot, which includes three variables—total spending, new/upgrades spending, and maintenance spending—broken into three time-based categories: last year’s actual spending, the current year’s enacted spending, and next year’s budgeted spending. Time series aren’t available nor is there a way to compare actual to plan. In other words, the comparisons that I would have found most meaningful couldn’t be made based on the information that’s available.

I want to encourage Vivek Kundra to complement his fine intentions with more effective designs. There’s no need to duplicate the mistakes that most businesses still make when working with information. Data analysis and presentation best practices are not a mystery and aren’t difficult to learn. Several of us who know and care about this are available to help. I suspect that others would be willing, as I am, to assist free of charge. America can do better than this. We have a great opportunity to use information technology to make the world a better place. Let’s not miss it.

Take care,

8 Comments on “Federal CIO Dashboard: We Can and Should Do Better Than This”


By Zach Gemignani. August 8th, 2009 at 7:40 pm

Steve,

Thanks for this critical review of the Fed IT Dashboard. I agree that we can simultaneously be grateful for the effort and intentions of this project while offering our information design expertise. In fact, I had an opportunity to meet with a couple of the folks responsible for it and provided some of the same feedback.

That said, I’d like to take a shot a defending the treemap implementation. (Disclaimer: the Fed IT Dashboard uses our open source JuiceKit for this visualization.)

1. SF: “Unfortunately, if we want to identify individual projects in these categories, we must hover with the mouse over each in turn to get the name to appear in a tooltip window.”

ZG: There is a design choice they made to not clutter up the visualization with labels. I’ve seen lots of treemaps that label everything and end up unreadable. A treemap is good at drawing your attention to the big and growing elements; I don’t think it is asking too much for the user to scan their mouse over the boxes as they investigate these most important items.

2. SF: “Based on this view, however, can you actually see the boundaries that separate one agency from another? For some reason, the borders that separate them have become partly obscured.”

ZG: Juice has to take the blame for this one as our first release of the treemap didn’t bring the same boundary formatting down as you drill in. That is being fixed in our next release.

3. SF: “And finally, when we’ve drilled down to the lowest level—a single project—the treemap view is entirely useless, as you can see below. The unlabeled big gray rectangle tells us only that spending on this project—whatever it is—didn’t change much since the previous year. Perhaps it didn’t even exist in the previous year.”

ZG: The image shows that the project is called “Strategic National and Theatre Defense” and the value changed by -0.37%, indicated it must have existed in the previous year. Interactive visualizations like this may not be quite as useful when zoomed way in; however, it is a difficulty requirement to expect the visualization to morph into something new as you get to a specific level of investigation.

4. SF: “This ability is useful for ad hoc data analysis when flexibility is needed to respond to unanticipated conditions, but on an analytical application like this, which has been designed to display a specific set of measures for a specific set of purposes, it would make more sense to select a color ramp that works well and resist complicating the dashboard with choices that aren’t necessary.”

ZG: This section in the IT dashboard is explicitly labeled the “Analysis” section, so I don’t think it is misguided to offer some flexibility. Personally, I’d have taken out the color selector as you suggestion. I would have left the range selectors as I’ve found it very useful for understanding the distributions — particularly with all the different metrics available for selection.

By Stephen Few. August 9th, 2009 at 11:00 am

Hi Zach,

Thanks for sharing your thoughts about the Federal IT Dashboard treemap. Of the four items that you mentioned, it appears that our opinions differ on two.

Regarding the lack of item labels (that is, labels for each of the small rectangles), I agree that being able to view a treemap without item labels is useful at times for particular objectives. However, it is also useful to view it with item labels for other objectives that are relevant to this implementation. Without the labels, we can get a overview of relative spending across the four service groups, the number of projects in each service group, the mix of large vs. small projects, and increases or decreases in spending since the previous year. Beyond this, we wish to know something about the actual projects themselves and how they compare to one another. This requires labels. Why should we be forced to hover over each rectangle in turn to identify each of the projects? Wouldn’t it make more sense to turn the labels on or off with a click of the mouse? When forced to view one label at a time, we can never make meaningful comparisons, because we simply can’t remember more than two or three of the labels at any one time and therefore must go back to those that we’ve identified previously again and again in an effort to make comparisons. By placing the service group labels in the middle of each large service group rectangle, you have made it difficult to label the individual project rectangles as well. By including a small title border at the top of each treemap group (that is, large rectangles that contain an entire group of smaller ones) where the name of that group can appear, you could also label the individual rectangles (except for those that are too small) in a way that clearly differentiates group labels (service groups in this case) from individual item labels (projects in this case).

Regarding the usefulness of drilling down to a single item (that is, a single rectangle), you mentioned that the treemap image provides the following information at this level: the project name and the percentage change in spending since the previous year. Actually, the image doesn’t provide this information; this information must be read from the text fields that appear above the treemap. The visualization itself is useless. Everything that’s provided by the text fields could have been provided at the prior hierarchical level by making the information available as text in the individual rectangles or via the tooltip window that appears when we hover over a single rectangle with the mouse. There is no reason for the treemap to drill to the lowest level.

By Zach Gemignani. August 10th, 2009 at 7:01 am

Regarding the labels, I’ve found that adding all those labels can add more confusion than insight. Check this one out: http://media.juiceanalytics.com/images/macrofocus_treemap.png

Also, adding a title bar on top of each group can be mildly deceptive. I wrote about this in my post about treemap design (http://www.juiceanalytics.com/writing/10-lessons-treemap-design/). Space means something in a treemap; when labels take up space intended to represent a value, it is breaks the model of the whole thing.

That said, I like your idea of turning on and off labels.

I also agree that drilling to a single item isn’t terribly useful. Unfortunately, the information above the treemap represent the whole of the current treemap view. This seems to me the logical right model, i.e. summary statistics for what you see. You could provide these stats on roll-over, but this doesn’t offer a permanent view if the user wants to go elsewhere on the screen. Trade-offs.

By Stephen Few. August 10th, 2009 at 9:57 am

Zach,

Labeling in a treemap can be done poorly and it can be done well–it’s a matter of design. The example that you provided is definitely confusing, because the labeling is poorly done. It doesn’t make sense to place labels in rectangles that are too small to display them. Here’s an example of labeling that works:

Treemap Labels

Labeling could definitely be applied to the projects that appear in the Federal IT Dashboard treemap. Those rectangles that are too small to display labels should automatically omit them. This might be difficult to do from a technical standpoint, but it’s what’s needed for an effective display.

By Zach Gemignani. August 10th, 2009 at 11:07 am

It is all about good design — and the right choice of visualization.

On the former, the labels look ok for the largest boxes (though poor contrast), but quickly break down after the top 10. In my opinion, they’d have been better served just trying to show the company name. And when you can only show the first two letters of a company name, it is time to stop labels altogether.

On the latter, this example isn’t well-suited to a treemap given that it doesn’t have any hierarchy. It would be better served as a bar chart or table with bars (http://www.juiceanalytics.com/writing/lightweight-data-exploration-in-excel/)

By Stephen Few. August 10th, 2009 at 11:32 am

Zach,

Actually, the example that I’ve shown above is well suited for a treemap. For the sake of simplicity and because of the severe size limitations for images shown in this blog, I’ve included only a single section (technology stocks) of a larger treemap (the entire stock market). In the original treemap from which this image was derived, I can view the entire stock market in a single high-level view grouped by sector and the labels for many of the stocks that are readily available and easy to read. I can compare these stocks with awareness of their identity without having to hover over each rectangle individually.

I agree that the when only two characters of a label can be seen, it is rarely useful. If I were designing the default labeling behavior for the treemap above, I would probably set the threshold for when the labels cease to appear to exclude them sooner. To illustrate the amount of useful information that can often be included as part of the text labels without making the treemap dysfunctional, I’ve included five pieces of information in addition to the stock name, but I could easily turn all but the stock name off.

You get my point. When labels are needed and can be displayed in the rectangles of a treemap, it is useful to include them. The ability to easily turn the labels on and off in the Federal IT Dashboard would make the treemap more useful.

By jerome cukier. August 26th, 2009 at 6:07 am

I’d like to take a step back to a higher level. I think the dashboard could use some very basic bar charts that could make some equally basic comparisons useful. In less than 10 graphs which could fit on 1 page, one could compare total IT spending or change in IT spending by service group, function or agency. Now the treemap allows a user to drill down to bureaus and to individual projects and see their relative importance to the whole, but it doesn’t help making high-level comparisons.
now you’re going to tell me, you can make bar charts with the trendalyzer widget in trends. but then I share your opinion on its usefulness when not supported by a presentation.

By Larry Keller. September 22nd, 2009 at 12:46 pm

When one considers the information consumer who has access to the visuals in debate, there is still a portion of those consumers that have not gone through the cultural adoption process from tabular to visual. These are code words for folks that may be 50 to 60+ years old and did not grow up in a visual world. I have found that turning labels off and on is absolutely critical to conveying a message in corporate America and is a must for the public. The clutter of a poorly designed tree map presents a bigger problem given the need to convey insights with very limited real estate if one needs labels.