Major kudos to the country’s new CIO, Vivek Kundra for setting a vision for open and transparent information and getting right to work.  The amount of time between talking about his vision and starting to execute on it with this Federal IT Dashboard has been weeks, not months or years.  I thought government was supposed to be slow?  Not in this case.

Given the speed and the fact that the dashboard is labeled “beta,” I can cut the team that put this together a lot of slack.  As an organizing framework for my comments, I used the structure of the CIO Dashboard series from my last 4 posts: Audience, Attributes, and Dashboard Type.

Audience

Audience is an interesting topic as this IT dashboard is not for internal organization consumption.  Rather, the stated audiences are federal contractors and citizens in general.  In the commercial sector, it would be like publishing your CIO dashboard to any interested consulting and sourcing firms and your shareholders!  Can you imagine the headaches that would be created if you shared your IT portfolio spending plans to the universe of IT consultants?  A tough job Federal IT has, for sure.

After reviewing the dashboard, I would recommend separate views for agencies/contractors and the general public as each are quite different in their needs and IT sophistication.

Attributes

In my first CIO Dashboard article, I proposed 5 attributes of a CIO dashboard:

1. One Page in Length
2. Easy to Read
3. Doesn’t Require a Decoder Ring
4. Accurate
5. Up-to-Date

Given that the tool is on-line, I’m not so worried about being a page in length.  What I am looking for, however, is how well the individual chunks of information are organized.  On the homepage, there is a nice master-slave relationship between the agency summary on the left and the investment details on the right.  Subsequent detail pages are well organized and easy to read.  Also, I am a big fan of the Google data analysis gadget in use under the Analysis tab under Trends.

My main issue with the dashboard is that it needs a pretty significant decoder ring.  On the right side of the homepage, two charts are presented for a particular agency.  The first is a pie chart that shows the number of investments rated by a 3-level scale, red, yellow and green.  Problem is that the red-yellow-green scale is not easily understood.  It took me about 10 minutes to find an explanation of the evaluation criteria and scoring scale.  Granted, the FAQs for Agencies do a decent job, but the charts should explain themselves (and I’m not an agency).  I don’t blame Mr. Kundra or his team with the cryptic initiative reporting.  However, this will be a significant issue in making this information truly accessible beyond the Beltway insiders.

In terms of accuracy and timeliness, the only dates and times that appear are the current time stamp.  I remember in my report writing days, we would have an “as of” date and a “printed” date.  These data snapshots do not seem to have the “as of” date, which is a significant omission.  That said, there may be the necessary dates in the detailed “Exhibits 53 and 300″ documents, but they are not shown in the dashboard.

Dashboard Type

This is a great example of a portfolio style dashboard and does a good job of showing investment portfolios for several massive organizations in a pretty simple way.  Unfortunately, the dashboard uses the term “investment” throughout, which I would use to mean new projects and initiatives but seems to be the term used for all spending categories, projects, maintenance, etc.  I’d like to see a better distinction between project and maintenance type investments.  There is a “tree browser” that you can use to look through a budget, but it is not very intuitive and only shows percentages.

Summary

A major congratulations should go to the Feds for a fast dashboard deployment and some very innovative features.  I expect many of the early issues to be worked out with as much effort and openness they are demonstrating.  We can all learn from this effort.

Pros:

• Dashboard up and running very quickly
• Open data feeds
• Simple charts and graphs
• Ability to share chart and graph links via social media channels (eg, Twitter)

Cons:

• Investment scoring criteria not easily found and understood
• Comments provided by agency CIO’s regarding their rating are sparse, even for those deemed complete.  When this data is complete, it will be some of the most useful information available through the dashboard
• Needs to simplify and clarify the relationships between “projects” and  “maintenance” and provide separate views for each
• Several minor viewing and mouse over glitches (in Firefox)

]]> http://rocketpanther.com/ciostage/metrics-and-measurement/federal-it-dashboard-decoder-ring/feed 4 http://rocketpanther.com/ciostage/metrics-and-measurement/the-problem-with-it-benchmarks http://rocketpanther.com/ciostage/metrics-and-measurement/the-problem-with-it-benchmarks#comments Wed, 01 Jul 2009 17:16:37 +0000 http://www.ciodashboard.com/?p=818 Tweet

18th Century Surveyor's Benchmark

A few years ago, I had the opportunity to meet with the enterprise CIO of a Fortune 50 company in the chemicals and industrial products industries (and many others, as I guess is the case for many giant companies).  The subject was justifying enterprise-level IT investments.  IT investments that span business units seem to be a tricky thing for companies without a CEO who isn’t a strong IT champion and those without an investment culture that handles investments where everyone must chip in.

To get a sense for how they thought about their investments, I described a top-down benchmark from MIT CISR that we use (see VIII-2B for the latest).  It categorizes IT spending in 4 categories: Infrastructure, Transactional, Informational and Strategic.  The data is collected annually as part of a survey MIT does with Gartner. This is a great framework and one I use to this day to discuss not only investment types but also the idea that you shouldn’t assign the same kinds of metrics and expectations to all investments – a network upgrade will have a different return than an investment in a new in-store customer service kiosk.

Anyway, I thought that presenting this benchmark for the relevant industries would be a good way to start the discussion.  Furthermore, I collected publicly-available IT spend as a % or revenue for the company and presented it too.

Well, this took a typically mild-mannered into a tirade about benchmarking and how it’s not very useful to him.  When asked why exactly, he offered:

“I can never figure out what exactly is in IT benchmarks, so I don’t use them.”

When I pressed further, he provided an example.  He said that the IT spending benchmarks he looks at are inconsistent regarding how end-user computing is accounted for:

• Are end-user PC costs all in IT costs or is some allocated to business units?
• Do the benchmarks include all hardware, software, labor and support?
• How is 3rd party end-user support, Tier-1 help desk and other external services accounted for?

He said that when he wants to see what other companies are spending for certain technology components or services, he just picks up the phone and calls his peers.  This conversation took up the rest of the time we spent together and really shaped how I think about benchmarks.

IT Benchmarks – Internal vs. External

Why would you use an external benchmark?  Some reasons:

• Increase comfort that you aren’t overspending
• Get input into an area you are redesigning
• Convince your boss or peers that you are not any worse than others

Another issue with many benchmarks is that you have no way of including only the companies you are most like and are the exemplars you would like to benchmark against.  Rather, you are stuck with all of the companies who respond in one aggregate number.  A very reputable IT benchmarker told me recently that the response rates are down 50-75% for some surveys as many of the “data junkies” who typically respond on behalf of their companies have been laid off or re-assigned.

I would recommend thinking about internal benchmarks as the primary way to benchmark your performance.  What I mean is take a metric you want to improve – maybe help desk calls resolved on the first call – and get a baseline measure.  Then, keep measuring it every day or week and watch the trends and experiment with ways to improve it.  Once you have some data, then look to 3rd party research and benchmarks to see where you stand – only to give you more ideas to improve yourself.  Not to tell you when to stop improving.

Catch up on the CIO Dashboard Series: Part 1, Part 2, Part 3

As a wrap-up to the CIO Dashboard series, I’ve summarized some of the key points and added a few new ones to this list of 10 CIO Dashboard tips.  Thanks to two of my colleagues at Diamond, Jim Quick and Adrian Vern, who are experts in measuring and communicating IT value through dashboards and provided their ideas.

10 CIO Dashboard Tips

1.  Know the questions your dashboard is trying to answer before building it

A financial services company we worked with a few years ago asked for help to review their IT dashboard and help them figure out how to make it better – that said, they were pretty proud of it.  IT WAS A 20 PAGE POWERPOINT FILE!  The basic issue they had is that they were trying to answer every conceivable question someone might have.

2.  Make sure you can actually collect the data you want to measure

You can do some phenomenal business driven design for a dashboard, but if you don’t have the data, you are dead in the water.  An insurance client wanted to measure productivity on his dashboard but didn’t have reliable time tracking.  We had to come up with a proxy for productivity instead (using project budget, deliverables and milestones…).

3.  Know your audience and understand how they consume information

Resist the temptations (or encouragement from staff or vendors) to use a fancy graphical dashboard tool for your early dashboard efforts.  Bottom line is that busy executives and managers don’t have the time to browse your dashboard online.  Know your audience(s) and how they consume their information.  If they have a long Connecticut – NYC commute, maybe a podcast is the best way to deliver it!

4.  Begin by summarizing and analyzing data you already collect

The easiest place to start is reporting on data you already collect and trust.  Think about recent system or process that has been improved and its data scrutinized.

5.  Your first dashboard should never use a dashboard tool – that should come later

Focus your initial efforts on defining the questions, understanding the audience and determining the credible metrics to report.  You probably won’t get it right at first, so why waste the expense of a fancy tool until you learn more?

6.  Dashboards should always have a printable version

If you want to make decisions using your dashboard, it should be portable and available when and where you need it.  Once the dashboard is established, some consumers of the dashboard will hopefully start carrying it too. [if you deliver your dashboard on a mobile device, that will work too]

7.  Incorporate application instrumentation into systems design process

This tip is a little more involved, but can create a lot of options for you.  Basically, the idea is to create a standard way to collect business metrics and embed the collection into the related systems.  I have helped a few clients do this specifically for metrics that were used in their business cases – sort of an automatic reporting feature to track the actual value their systems are helping to deliver.

8.  Make sure those responsible for creating the dashboard understand who is reading it – it will increase quality

One of our insurance clients was having some data quality issues with an IT performance dashboard.  So, they decided to make a big deal of letting everyone know who (the COO, CFO and CIO) was reading the dashboard and that they were all pretty disappointed with the unreliable information.  This helped to improve the data almost overnight.

9.  Create a report to perform checks and balances on core dashboard data to increase credibility

Another idea for improving dashboard data quality is to build a companion report to perform some checks and balances to increase confidence.

10.  Keep a list that tracks the decisions and changes made as a result of dashboard analysis; attach quantitative and qualitative benefits that result

We all want to get value from our business investments.  So, why not track the decisions made using a dashboard and the outcomes/results from the decision?  See if this whole CIO Dashboard thing is worth it or not.

That’s it for the CIO Dashboard series.  I hope it made you think a little.  Please leave me some thoughts in the comments section.

Further Reading on Dashboards:

• Breaking Free of the One Page Dashboard Rule by Zach Gemignani
• Dashboard Confusion by Stephen Few
• Using Analytical Dashboards to Cut Through the Clutter by Kurt Rivard and Doug Cogswell, DM Review, April 2004

Catch up on the Building a CIO Dashboard Series: Part 1, Part 2

Now that we have a few questions answered – do I really need a dashboard, what questions will it answer and what type of dashboard we need, we can discuss how to deliver it.

This post will cover the 3 channels through which a dashboard is presented:

1. Devices – Blackberries, iPhones and other mobile devices
2. Desktops – browser based dashboards and other stand-alone dashboard tools
3. Dead Trees – paper-based dashboards

Some automatically move to the desktop delivery of dashboards since it is the most data-rich and visual of the options (and most fun for the developers to play with?).  However, if our objectives are to use the data for decision-making and to influence behavior, then we need to think about delivering it in a way that will maximize the chances of it being regularly read.

I don’t know about your organization, but I don’t know many whose senior leadership has the time or patience to sit down and explore a desktop-based dashboard.  So, I propose looking at the options by the audience as follows.

In summary, consider mobile device delivery for non-IT audiences.  For IT audiences, the use of intranet or Sharepoint sites will determine how much emphasis is put on desktop delivery.  Always consider paper dashboards for easy carrying and sharing with stakeholders.

BOTTOM LINE:  Don’t let vendors tell you that you need a fancy graphical tool to build a dashboard.  That will hurt you more than it will help.  Especially for a non-IT audience.

In the fourth and final CIO Dashboard post, I will cover some challenges and final thoughts.

Sample CIO Dashboards

Business Value Creation Report – Healthcare

• Purpose:  show progress in delivering business capabilities identified during a strategic enterprise architecture planning project
• Audience:  Non-IT, business unit leaders, functional/process heads
• Characteristics:  longer than 1 page and could be easier to read
• Delivery:  paper (via Powerpoint)
• Comments:  uses language that business stakeholders are familiar with and little IT jargon; in use for 3+ years

IT Service Dashboard – Insurance

• Purpose:  show overall progress toward stated IT objectives
• Audience:  IT steering committee (Non-IT and IT)
• Characteristics:  1 page and visually appealing, requires some decoding
• Delivery:  paper (via Powerpoint)
• Comments:  contained over 30 metrics, many of which were compiled using custom metrics and formulas which made this very manually-intensive

Critical Transaction Dashboard – Banking

• Purpose:  highlight processing status and issues with key customer oriented functions
• Audience:  business and IT function heads
• Characteristics:  simple and uses business language
• Delivery:  Blackberry (primary), browser (secondary)
• Comments: mobile dashboard was very critical in a high-volume 24×7 financial markets business; ability to drill into detail for any dashboard item.

IT Portfolio Dashboard – Insurance

• Purpose:  provide a broad view into project investments managed by IT
• Audience:  COO, CFO and IT leaders
• Characteristics:  simple layout and visually appealing; too many tabs and detail to promote summaries
• Delivery:  browser and specialty tools (eg, Business Objects, Crystal Xcelcius, Compuware Changepoint)
• Comments: extensive effort to develop this dashboard but once done, the information was used daily by most stakeholders

Read Part I of the Building a CIO Dashboard Series

I think the biggest challenge in building a meaningful CIO dashboard lies in the question of what to measure.  As my partner Jim Quick says it’s either feast or famine when it comes to measuring IT.  You either have so much data you don’t know where to focus or you are paralyzed by the idea of figuring this out.

In keeping with the best iterative design practices, I’ve altered the structure of this series of posts about building a CIO dashboard.  The design topic actually deserves two posts, so this one (part 2) will center on what to measure and the next one will look at different display techniques.

Audience Matters

Given the range of questions that can be addressed, maybe the most important way to narrow down your measurement choices is by considering your audience.  I don’t think there will be much argument that there are some things important to IT that the non-IT audience won’t care about or understand.  So let’s define two broad audiences that way.

When I think of a dashboard, I think of a broad set of gauges that tell me how my car is performing, how it’s using its resources, the status of the interior and exterior environment and if there are any issues.  Today’s car dashboard is actually a great metaphor for the CIO dashboard because of the degree of abstraction required to summarize all of the on-board computers and sensors.  But since there are so many things to measure, creating a meaningful AND broad dashboard is a significant, but doable challenge.

So, a second way to discuss the choices of metrics is to determine the scope to be measured – broad across all of IT or more specific to a program, business area or set of business objectives.  With audience and scope as our two dimensions in the ubiquitous 2×2, we can discuss the 4 types of CIO dashboards.

Portfolio Dashboard

For those organizations who have adopted the portfolio management philosophy, a portfolio dashboard can make a lot of sense.  In fact, the country’s new CIO, Vivek Kundra, is planning a portfolio dashboard for the government’s technology projects.

There are two real challenges in this kind of dashboard – actually they are challenges with IT portfolio management in general:

1. It is critical to embrace the philosophy that there are different types of IT projects (asset classes), each with a separate benefits (return) profile.  I will write a separate post on this subject, but as an example, measuring an email upgrade using an ROI metric doesn’t make much sense.  A portfolio with one asset class defeats the purpose of portfolio management.
2. You must have the discipline to measure actual benefits regularly after the project has finished.  I would guess that 20% of IT organizations do this sufficiently well.

Sample Measures

• Planned Projects: Projected Spend ($, %) by  Project, Project Type; Risk/return by Project, Project Type
• In-flight Projects: Projected Return by Project, Project Type; Baseline vs Estimated Budget by Project
• Completed Projects: Projected vs Actual Return by Project, Project Type

Linkage Dashboard

Probably my favorite way to think about and communicate IT metrics is to explicitly link them to some business metric that is important and well-known across the company.  The challenge with this is to credibly link the business objective to the IT metric.  There are two ways to do this:

1. Develop the explicit linkage yourself.  One technique for this is using Diamond’s Business Architecture approach, but other approaches exist too – Balanced Scorecard? Others?
2. Rely on academic or industry research to prove a causal link between a business objective (eg, improve customer service) and an IT capability (eg, CRM system performance) and then just measure the IT performance metric as a direct impact to the business objective.  This is easier but less credible.  MIT’s Center for Information Systems Research has done the best job I have seen in linking business performance to various IT practices (governance, architecture, etc).

Sample Measures

The measures will depend on the business objectives, but here is an example of the linkage to give you an idea of how this could look.  This is from a presentation by Peter Weill at MIT CISR.

Service Dashboard

Even though the service dashboard is oriented toward an IT audience, it if focused on organizing and presenting metrics that impact service to IT’s internal constituents.

Sample Measures

• Throughput: Business (not tech) transactions per hour/week/month and peak times
• Quality: Number and severity of feature/bug fix requests
• Satisfaction: average time to close helpdesk tickets; qualitative customer sat measures

Improvement Dashboard

I’m not a fan of a general purpose IT dashboard because it will lead directly to the “metrics feast” and a lack of focus.  So, a broad IT dashboard should be oriented around IT’s 3-5 improvement goals, whatever they may be.  Popular improvements IT shops seem to be focusing on these days include KTLO cost reduction, applications TCO and software reuse.

Sample Measures

• Projects: earned value metrics
• Processes: # exceptions in governance processes
• Maintenance: size and severity of backlog
• Skills: % of staff trained in core skills

I wouldn’t be offended if your dashboard mixed and matched types and metrics in the columns (Portfolio + Linkage, Service + Improvement).  But I think it’s pretty important to keep things clean within each audience type.

I look forward to your feedback and comments.  The next post will look at some actual examples of CIO dashboards.

Read Part 3 – Device, Desktop or Dead Tree?

I figured it was about time for me to tackle the CIO dashboard topic since you’ve been good enough to read along so far and were probably wondering if I would ever get around to it. I think this is a pretty substantial topic, so I plan to use a series of posts to cover it over the next few weeks.

Part 1 (this post) explores the rationale for and attributes of a CIO dashboard.  Part 2 will discuss dashboard audience.  Part 3 will cover dashboard design considerations and offer some examples.  The last post, Part 4, will summarize things in a 10 tips list.

Why Do You Need a CIO Dashboard?

Come on, aren’t you already swimming in data?  Most people are, but there are a few paralyzed by reporting – sort of a feast of famine situation.  Nevertheless, the idea of a dashboard is appealing.

Here are some of the reasons I have heard for taking on this kind of project:

• “I need a summary of what’s going on across IT”
• “I want to show progress that IT is making toward our goals”
• “I need to better justify IT investments to the organization”
• “I need a way to track a specific target set by the CEO/my boss”

Consider a comment from JP Morgan Chase CEO Jamie Dimon on how he and his leadership team work (from Fortune, June 8, 2009):

You have to be very disciplined, and this is even more important in a crisis. Our management team, 15 to 30 people, talk every day at 7:30, 12 noon, and five o’clock – what’s going on, sharing information, making some decisions on the spot, reviewing facts and information.

In plain English is a requirement for all of his team to be armed with the right information every day that can be used to make timely decisions.  This is what I would want my CIO dashboard to do for me.

5 Characteristics of a CIO Dashboard

These are the five as the core attributes of a CIO dashboard, or any management dashboard for that matter:

1. One Page in Length
2. Easy to Read
3. Doesn’t Require a Decoder Ring
4. Accurate
5. Up-to-Date

One Page in Length

In keeping with the simplicity theme, a management dashboard should contain no more information than can fit on 1 8.5×11 page (or 1 window).  Furthermore, while we would all like to think we live in a 100% digital world, the dashboard should be printable – a requirement if you have senior management as your audience.

Easy to Read

To quote Edward Tufte in his seminal The Visual Display of Quantitative Information:

Data graphics should draw the viewer’s attention to the sense and substance of the data, not to something else.

A good dashboard needs to be low on the chartjunk and generous with white space.

Doesn’t Require a Decoder Ring

Not only should a dashboard be easy to read, the data must stand on its own and not required a legend or key.  For example, one of my insurance clients has a composite metric to measure and report on the “health” of a major replatforming program – red, yellow, green.  Much quantitative and quantitative data combine to create the measure, but it is not simple or transparent.  Instead, they would be better served by reporting metrics like budget to complete and earned value measures.

Accurate

One of the biggest challenges in management reporting is data accuracy.  A CIO of a regional bank I have been working with wants to understand how much of her staff are working on strategic projects.  The problem is that the time tracking system only provides maintenance and trouble-ticket related categories and some generic project buckets.  The resulting data is virtually meaningless.

Up to Date

The dashboard metaphor often leads to a desire for real-time data.  I purposely use the term up-to-date instead because real-time data is both extremely difficult to deliver accurately and almost never needed.

In the next post, I will lay out some ideas about tailoring your dashboard to your audience and some design considerations, including online versus printed dashboards.  I look forward to your comments and suggestions.

Go to the next post: 4 Types of CIO Dashboards