18th Century Surveyor's Benchmark

A few years ago, I had the opportunity to meet with the enterprise CIO of a Fortune 50 company in the chemicals and industrial products industries (and many others, as I guess is the case for many giant companies).  The subject was justifying enterprise-level IT investments.  IT investments that span business units seem to be a tricky thing for companies without a CEO who isn’t a strong IT champion and those without an investment culture that handles investments where everyone must chip in.

To get a sense for how they thought about their investments, I described a top-down benchmark from MIT CISR that we use (see VIII-2B for the latest).  It categorizes IT spending in 4 categories: Infrastructure, Transactional, Informational and Strategic.  The data is collected annually as part of a survey MIT does with Gartner. This is a great framework and one I use to this day to discuss not only investment types but also the idea that you shouldn’t assign the same kinds of metrics and expectations to all investments – a network upgrade will have a different return than an investment in a new in-store customer service kiosk.

Anyway, I thought that presenting this benchmark for the relevant industries would be a good way to start the discussion.  Furthermore, I collected publicly-available IT spend as a % or revenue for the company and presented it too.

Well, this took a typically mild-mannered into a tirade about benchmarking and how it’s not very useful to him.  When asked why exactly, he offered:

“I can never figure out what exactly is in IT benchmarks, so I don’t use them.”

When I pressed further, he provided an example.  He said that the IT spending benchmarks he looks at are inconsistent regarding how end-user computing is accounted for:

• Are end-user PC costs all in IT costs or is some allocated to business units?
• Do the benchmarks include all hardware, software, labor and support?
• How is 3rd party end-user support, Tier-1 help desk and other external services accounted for?

He said that when he wants to see what other companies are spending for certain technology components or services, he just picks up the phone and calls his peers.  This conversation took up the rest of the time we spent together and really shaped how I think about benchmarks.

IT Benchmarks – Internal vs. External

Why would you use an external benchmark?  Some reasons:

• Increase comfort that you aren’t overspending
• Get input into an area you are redesigning
• Convince your boss or peers that you are not any worse than others

Another issue with many benchmarks is that you have no way of including only the companies you are most like and are the exemplars you would like to benchmark against.  Rather, you are stuck with all of the companies who respond in one aggregate number.  A very reputable IT benchmarker told me recently that the response rates are down 50-75% for some surveys as many of the “data junkies” who typically respond on behalf of their companies have been laid off or re-assigned.

I would recommend thinking about internal benchmarks as the primary way to benchmark your performance.  What I mean is take a metric you want to improve – maybe help desk calls resolved on the first call – and get a baseline measure.  Then, keep measuring it every day or week and watch the trends and experiment with ways to improve it.  Once you have some data, then look to 3rd party research and benchmarks to see where you stand – only to give you more ideas to improve yourself.  Not to tell you when to stop improving.