IT Benchmarking: A Tool for Measuring IT Performance

In the quest to demonstrate value to top management and other stakeholders or convince them to act, the IT leadership may assess the current state of IT. This assessment covers various dimensions including:

• Budgets
• Development practices
• Infrastructure
• People
• Vendor management
• Service delivery

These assessments measure, analyze, and understand the status, behavior and performance across IT dimensions. They highlight well-performing areas and identify gaps or concerns that require improvement. By comparing with expectations, IT functions gain better insight into their current position and determine what needs enhancing to get to the next level.

In this article, we will discuss the definition of IT benchmarking, the two main approaches, and the benefits and drawbacks of this method of asses the state of an organizations IT performance.

What is IT benchmarking?

IT benchmarking is a popular method for assessing the current state of IT across various parameters. Gartner defines benchmarking as the comparison between an organization’s performance and the performance of designated organizations or indexes.

Indexes refer to publicly (or privately) available indicators for a factor that is associated with a measurement element such as:

• Price
• Customer satisfaction
• Industry standard

IT benchmarking involves measuring the performance of an organization’s IT products, services, or practices against those of a similar organization, usually an industry competitor or peer. By benchmarking against a superior organization, an IT team can identify improvement initiatives that could propel them to the top of the class.

(Related reading: IT service management).

The ITIL 4 Direct Plan and Improve publication states that benchmarking serves as a valuable tool for motivating culture change, based on the premise that organizations can set the standard by which others measure themselves.

IT benchmarking approaches

Where should IT organizations start with benchmarking? Context matters here. IT organizations can use two fundamental types of benchmarking: internal and external benchmarking.

Internal IT benchmarking

Internal IT benchmarking compares performance within an organization, usually between departments or functions that may or may not carry out similar activities.

For example, a product team might compare its development metrics, such as on-time delivery and team velocity, with those of other internal product teams. Knowledge bodies like APQC provide IT benchmarking measures that IT functions can use as starting points to gain insights into the efficiency and effectiveness of their processes.

Examples of such measures include:

• Cost effective measures: Total IT cost per IT Full Time Employee (FTE); Cloud Services costs as a percentage of total IT costs.
• Cycle time measures: Time in weeks from concept to major release; Time in weeks to close an identified IT skill/capability gap.
• Process efficiency measures: Net Promoter Score (NPS) for IT services.
• Productivity measures: Number of end users serviced per IT FTE.

External IT benchmarking

External IT benchmarking compares an organization’s performance with industry peers using data from various sources, including third party sources or standards. An organization, regulators, industry associations, or contracted third parties may carry out external benchmarking. Ideally, companies share anonymized data to use for benchmarking.

Examples of external IT benchmarking services include:

Gartner offers a benchmarking service called IT Score for CIOs that covers 30 functional activities across 7 functional objectives. Organizations can perform a self-assessment to measure their maturity level relative to Gartner’s best practice research and prioritize improvement areas.

Gartner IT Score for CIOs

IDC Global through its research arm provides an IT benchmarking service that generates insights based on six key parameters that are analyzed as follows:

• Cloud economics – cloud billing, rates and resource utilization
• Deal and bid support – market intelligence and vendor selection
• IT service cost management – internal IT cost and performance
• IT staffing assessment – staffing, structure, and skills
• IT supplier performance management – contracted services against market performance and costs
• Contract lifecycle management – contract creation to completion control, risk, and cost management

Info~Tech Research Group provides an IT spend and staffing benchmarking service that involves a comparative analysis of IT’s financial and human resource data with aggregate data from industry participants. The analyzed information is then presented based on stakeholder views showing insights and opportunities, and thereafter a cost and staffing optimization plan is provided. The example below shows sample outputs of the benchmarking analysis.

Sample IT Benchmarking Analysis (Source: Info~Tech)

IT benchmarking benefits and drawbacks

When selecting IT benchmarking as an assessment methodology, organizations must consider some of the benefits and drawbacks.

Benefits of IT benchmarking

According to the ITIL 4 Direct Plan and Improve publication, the benefits of this method include:

• It focuses on ideas for improvement. IT benchmarking highlights specific areas where peers have made progress or where the organization has overlooked opportunities for improvement.
• It provides quantitative and explicit standards for comparison. IT benchmarking centers on data, offering IT teams standardized measures that simplify comparison and help plan incremental improvements.
• It enables competitive analysis or potential partnering. External benchmarking offers competitor ratings, allowing IT providers to identify capabilities they can leverage to win more business or to approach high-performing organizations for potential collaborators in competitive bids.
• It allows comparison across multiple industries. IT benchmarking data can provide valuable insights into sectors beyond the IT provider’s own environment.

Drawbacks of IT benchmarking

However, IT benchmarking also has some drawbacks including:

• It does not always transfer well between organizations without context. No two IT providers are identical. Even if organizations share similar revenue, staffing, and industries, they may operate under different models or serve different markets. Therefore, IT benchmarking data should serve as a guide for improvement, not as a strict target.
• It does not measure effectiveness. While effectiveness relates to the value created, some IT benchmarking parameters may not directly reflect benefits for an organization. For example, Company A’s IT costs as a percentage of revenue might exceed the average benchmark, but if profitability and customer satisfaction are high, the company may have deliberately chosen to invest more in IT than its peers.
• It can introduce industry bias based on revenue rather than practice. IT benchmarking data can skew in favor of the larger players with bigger market share due to first-mover advantages, acquisitions, or other factors. As a result, peers with better processes, staffing or technology may appear inferior because the revenue perspective overshadows other factors.
• It aims to identify industry leaders, which may standardize behaviors that aren’t ideal. For example, an IT provider may be rated highly on response and resolution times, but the IT benchmark may overlook customer satisfaction. As a result, the company might close tickets faster than its competition, but customers cold still suffer — a phenomenon known as the “watermelon effect” in service management circles.

Final thoughts

Top performing organizations attract customers, media and investors among other stakeholders. IT benchmarking allows organizations to measure their performance and standards against their peers. This process helps them identify what works and their Achilles’ heel is from a competitive perspective.

Areas such as digital transformation, operations, and cybersecurity are all suitable for IT benchmarking. Industry comparisons can uncover practices that organizations should optimize to gain an edge by improving efficiency and effectiveness.

FAQs about IT benchmarking