The Problem With IT Benchmarks

Surveyor's Benchmark

18th Century Surveyor's Benchmark

A few years ago, I had the opportunity to meet with the enterprise CIO of a Fortune 50 company in the chemicals and industrial products industries (and many others, as I guess is the case for many giant companies).  The subject was justifying enterprise-level IT investments.  IT investments that span business units seem to be a tricky thing for companies without a CEO who isn’t a strong IT champion and those without an investment culture that handles investments where everyone must chip in.

To get a sense for how they thought about their investments, I described a top-down benchmark from MIT CISR that we use (see VIII-2B for the latest).  It categorizes IT spending in 4 categories: Infrastructure, Transactional, Informational and Strategic.  The data is collected annually as part of a survey MIT does with Gartner. This is a great framework and one I use to this day to discuss not only investment types but also the idea that you shouldn’t assign the same kinds of metrics and expectations to all investments – a network upgrade will have a different return than an investment in a new in-store customer service kiosk.

Anyway, I thought that presenting this benchmark for the relevant industries would be a good way to start the discussion.  Furthermore, I collected publicly-available IT spend as a % or revenue for the company and presented it too.

Well, this took a typically mild-mannered into a tirade about benchmarking and how it’s not very useful to him.  When asked why exactly, he offered:

“I can never figure out what exactly is in IT benchmarks, so I don’t use them.”

When I pressed further, he provided an example.  He said that the IT spending benchmarks he looks at are inconsistent regarding how end-user computing is accounted for:

  • Are end-user PC costs all in IT costs or is some allocated to business units?
  • Do the benchmarks include all hardware, software, labor and support?
  • How is 3rd party end-user support, Tier-1 help desk and other external services accounted for?

He said that when he wants to see what other companies are spending for certain technology components or services, he just picks up the phone and calls his peers.  This conversation took up the rest of the time we spent together and really shaped how I think about benchmarks.

IT Benchmarks – Internal vs. External

Why would you use an external benchmark?  Some reasons:

  • Increase comfort that you aren’t overspending
  • Get input into an area you are redesigning
  • Convince your boss or peers that you are not any worse than others

Another issue with many benchmarks is that you have no way of including only the companies you are most like and are the exemplars you would like to benchmark against.  Rather, you are stuck with all of the companies who respond in one aggregate number.  A very reputable IT benchmarker told me recently that the response rates are down 50-75% for some surveys as many of the “data junkies” who typically respond on behalf of their companies have been laid off or re-assigned.

I would recommend thinking about internal benchmarks as the primary way to benchmark your performance.  What I mean is take a metric you want to improve – maybe help desk calls resolved on the first call – and get a baseline measure.  Then, keep measuring it every day or week and watch the trends and experiment with ways to improve it.  Once you have some data, then look to 3rd party research and benchmarks to see where you stand – only to give you more ideas to improve yourself.  Not to tell you when to stop improving.

  • http://www.measuresw.com Grant (PG) Rule

    As a benchmarker, I have to agree with Chris. The most useful benchmarks and comparisons are those you make with yourself.

    It doesn’t really matter what the chap next door, or two streets over, is doing. What matters is that you are being more effective today at achieving your strategic goals than you were yesterday.

    Of course, external benchmarks can be useful as marketing material. They may help you convince others that they are standing on a burning platform, and create that sense of urgency that is imperative if any kind of transformation is to take place. But internal comparisons of what you could do if you removed the waste from the process, with what you are currently achieving, will always be more convincing than comparison to 3rd parties.

    You need a quantitative view of where you are, a vision of where you want to be, an understanding of the gaps, a route map of how to get there from here, and the will to act. That’s all. It’s easy really. Just make it so.

    “The ways of the military are five: measurement, assessment, calculation, comparison and victory”… to paraphrase Sun Tzu.

    Regards,
    Grant (PG) Rule
    MD, Software Measurement Services Ltd

  • http://www.measuresw.com Grant (PG) Rule

    As a benchmarker, I have to agree with Chris. The most useful benchmarks and comparisons are those you make with yourself.

    It doesn’t really matter what the chap next door, or two streets over, is doing. What matters is that you are being more effective today at achieving your strategic goals than you were yesterday.

    Of course, external benchmarks can be useful as marketing material. They may help you convince others that they are standing on a burning platform, and create that sense of urgency that is imperative if any kind of transformation is to take place. But internal comparisons of what you could do if you removed the waste from the process, with what you are currently achieving, will always be more convincing than comparison to 3rd parties.

    You need a quantitative view of where you are, a vision of where you want to be, an understanding of the gaps, a route map of how to get there from here, and the will to act. That’s all. It’s easy really. Just make it so.

    “The ways of the military are five: measurement, assessment, calculation, comparison and victory”… to paraphrase Sun Tzu.

    Regards,
    Grant (PG) Rule
    MD, Software Measurement Services Ltd

  • http://www.telmaco.com Bernard Londeix

    Wouldn’t it be the moment to recommend GQM?
    All the best,
    Bernard

  • http://www.telmaco.com Bernard Londeix

    Wouldn’t it be the moment to recommend GQM?
    All the best,
    Bernard

  • Pingback: The CIO Guide to Dashboards — CIO Dashboard()

  • Pingback: Louie Ehrlich, Chevron CIO – Future State CIO Journey — CIO Dashboard()