Benchmarking in High-Security Environments

Benchmarking is certainly important (Goetsch & Davis, 2021), but making it happen in a sector where secrets must be kept involves “creative” solutions, or only making extremely broad comparisons that do not have value to opponents. For example, Gebicke & Magid’s (2010) global study doesn’t compare specific defense systems, but they do compare force size, tooth-to-tail ratio, and so on.

Another type of comparison that doesn’t involve sharing secretive information is between military education institutions. For example, V. Kravets (2024) compares Ukraine’s higher military educational instructions, but instead of comparing technical proficiencies in military science, her goal is to determine the feasibility of including management activities into those institutions. Aren't things going bad enough for Ukraine?

Because of the need to avoid classified information becoming public, the benchmarking that could be used by companies like Rheinmetall AG against BAE Systems or General Dynamics is fraught with difficulties that aren’t encountered in civilian industries. For example, in any company that has IT infrastructure (which means all companies), benchmarking various IT components (like databases or servers) is possible because different industries use the same IT components, and so the benchmarking partners need not be competitors. For example, Google’s Gmail and the fictitious Gaggle dot Com’s GaggleMail are competitors, so no mission-critical information should pass between them. Gaggle dot Com is not a competitor of X (formerly Twitter), so it is OK to benchmark their databases, for example. And this database benchmarking can involve direct comparison of databases made by the same company (like Microsoft) or comparisons of databases made by different companies (Microsoft vs Oracle).

It's not clear whether the same benchmarking would happen in the IT departments of artillery manufacturers since there are all sorts of proprietary IT components. But one can benchmark various systemic quality measures like lean or six sigma standards against other companies, without giving away classified information.

In a 1999 paper by Yarrow & Prabhu, three different modes of benchmarking are presented: metric benchmarking, diagnostic benchmarking, and process benchmarking. Metric benchmarking is the comparison of “apples with apples” performance data. Process benchmarking “involves two or more organizations comparing their practices in a specific area of activity, in depth, to learn how better results can be achieved.” And diagnostic benchmarking “seeks to explore both practices and performance, establishing not only which of the company’s results areas are relatively weak, but also which practices exhibit room for improvement.”

I would like to make guesses of the types of benchmarking done at companies like Rheinmetall AG, without knowing anything about artillery manufacturing! Metric benchmarking could be done on an IT component level (everybody uses databases), but exact benchmarking may be precluded because proprietary software is used. In that case, process benchmarking would still be possible. Diagnostic benchmarking seems to best describe comparisons of six sigma measurements. But I don’t know anything about six sigma, either!

For metric comparisons, then, companies like Rheinmetall AG must enter into consortium agreements with other defense manufacturers as you stated. It isn’t clear how security would be maintained, even if the data is anonymized. In the absence of consortium agreements, one would have to look to different industries. For example, for information about specific tolerances, one would have to compare data from, say, civilian pipe manufacturers. This is probably a one-way transfer of information.

References

Gebicke, S. & Magid, S. (2010). Lessons from around the world: Benchmarking performance in defense. McKinsey & Company. https://www.mckinsey.com/~/media/mckinsey/dotcom/client_service/public%20sector/pdfs/mck%20on%20govt/defense/mog_benchmarking_v9.pdf

Goetsch, D. L. & Davis, S. B. (2021). Quality management for organizational excellence: Introduction to total quality (9^th ed.). Pearson.

Kravets, V. (2024). Development strategies for higher military educational institutions of Ukraine: analysis based on benchmarking. Честь і закон, 2(89), 74-82. https://chiz.nangu.edu.ua/article/download/309198/300732/714412

Yarrow, D. & Prabhu, V. (1999). Collaborating to compete: Benchmarking through regional partnerships. Total Quality Management, 10(4-5), 793-802. https://doi.org/10.1080/0954412997820

Benchmarking in Brick-and-Mortar Stores

According to Jenkins (2025):

Reverse logistics is the reverse of the standard supply chain flow, where goods move from manufacturer to end consumer. Reverse logistics includes activities like returns management, refurbishment, recycling, and disposal. It’s an important part of supply chain management, often involving the return of products due to damage, seasonal inventory, restock, salvage, recalls, or excess inventory.

Benchmarking has several interesting twists when applied to the problems of reverse logistics in brick-and-mortar stores, such as hardware stores.

In the context of a hardware store, internal benchmarking of reverse logistics is possible. For example, comparing rates of customer returns by manufacturer would be valuable to the customer, with the idea of minimizing the number of returns. Based on that information, manufacturers making products with high return rates can be dropped from the hardware store’s offerings (Jenkins, 2025).

In cases where customers do return a product, the speed at which the vendor provides credit can also be tracked. Even if products from the slowest vendor are maintained in the store’s offerings, knowing the expected delay in credit could be useful for accounting purposes. For example, if a particular vendor takes 60 days to provide credit, then that credit cannot be used to cover any expenses until 60 days.

Other information that can be gleaned from benchmarking returns includes measurements to identify and reduce slow-running processes like return processing, reentry into the inventory system, and coordinating with the vendor (Goetsch & Davis, 2021).

Internal comparisons aren’t the only route to using benchmarking for process improvement, of course. Benchmarking partners are also available, and in a way that is different from benchmarking between the information technology (IT) departments of various companies.

By being brick-and-mortar, hardware stores can engage companies in the same business but are separated by geographic distance so that they aren’t direct competitors. For example, a hardware store in Hawaii can form a benchmarking partnership with a hardware store in Philadelphia, say. Because these stores rely on foot traffic, there is extremely little chance that a customer of the Philly store would travel to Hawaii to pick up a hammer!

So, information learned by forming a benchmarking partnership is extremely relevant and valuable to both partners (since they are in the same business), but that information cannot be used against each other (since they are brick-and-mortar stores located on opposite sides of the globe)!

It is also possible to set up a reuse supply chain (Atterblad & Blomkvist, 2023). With this, used and returned products are shunted to a used hardware store. It isn’t clear from Atterblad & Blomkvist how profitable this is to the original hardware store, but it at least avoids a 100% loss. Besides “return to point of origin,” it is also possible for the customer to sell or donate products to a “second-life retailer” (Beh et al, 2016). In that situation, the original hardware store does not benefit at all.

References

Atterblad, R., & Blomkvist, H. (2023). Challenges and recommendations for product reuse: Exploring the reuse supply chain of in-store hardware: A case study. https://www.diva-portal.org/smash/get/diva2:1770268/FULLTEXT01.pdf

Beh, L. S., Ghobadian, A., He, Q., Gallear, D., & O'Regan, N. (2016). Second-life retailing: a reverse supply chain perspective. Supply Chain Management: An International Journal, 21(2), 259-272. https://doi.org/10.1108/SCM-07-2015-0296

Goetsch, D. L. & Davis, S. B. (2021). Quality management for organizational excellence: Introduction to total quality (9^th ed.). Pearson.

Jenkins, A. (2025, 1 May). A guide to reverse logistics: How it works, types and strategies. Oracle NetSuite. https://www.netsuite.com/portal/resource/articles/inventory-management/reverse-logistics.shtml

Process Benchmarking

Introduction

In this discussion post, process benchmarking is defined, and it is compared with seemingly related concepts as well as other types of benchmarking. The application of benchmarking in the information technology (IT) industry is discussed, and then it is applied to a fictitious software company. Finally, it is noted that while all companies and organizations can benefit from benchmarking, contemporary industrial manufacturing (Industry 4.0) companies can especially benefit from it, due to Industry 4.0’s similarity to the IT industry.

What is Process Benchmarking?

Goetsch & Davis (2021) define benchmarking, specifically process benchmarking, as “the process of comparing and measuring an organization’s operations or its internal processes against those of a best-in-class performer from inside or outside its industry.” (p. 360)

This is different from competitive analysis, which is what customers do. Benchmarking is not about comparing the final product or service of two companies. Rather, benchmarking involves comparing the underlying processes used to produce a product or service, distribute it, and support it. The goal is to find a “secret sauce” another company uses and adopt that to one’s own company. That other company is called a benchmarking partner. If possible, the benchmarking partner should be “best-in-class,” at least in the particular process under comparison.

Benchmarking should also not be confused with so-called “best practices.” In software development, best practices are frequently arbitrary constraints not backed by any measurement. Best practices’ popularity is simply based on the insecurities that many IT employees and managers have.

Yarrow & Prabhu (1999) present three different modes of benchmarking: metric benchmarking, diagnostic benchmarking, and process benchmarking. The first concerns comparisons of performance data. They state:

So long as we are comparing `apples with apples’, metric benchmarking can serve a useful purpose as a `call to action’. However, its emphasis is on the `what’ rather than the `how’. This form of benchmarking can help an organization to pinpoint aspects of performance that need to improve, but on its own it cannot help them to learn how to improve. (Yarrow & Prabhu, 1999, p. 794)

Process benchmarking “involves two or more organizations comparing their practices in a specific area of activity, in depth, to learn how better results can be achieved.”

The third approach, diagnostic benchmarking, “seeks to explore both practices and performance, establishing not only which of the company’s results areas are relatively weak, but also which practices exhibit room for improvement.”

Benchmarking in IT Departments

In large companies with multiple independent software products, it is possible for different product teams to become benchmarking partners, if one of the teams has some metric that is significantly better than the other team’s. Even then, benchmarking with other companies could still be needed to achieve best-in-class quality.

Companies or organizations which have IT departments (which means all companies and organizations) all have certain problems in common. These problems include: what is the best database to use? What is the best way to test software? What is the best way to make websites publicly available? What is the best way to secure a computer system? and so on.

Because of this, it is possible to have benchmarking partners from widely varying industries.

The solutions to these problems are exchanged on an informal basis. IT workers frequently participate in user groups, either in-person or online. There are also online forums, product-specific support groups, etc., that can be used as sources of information.

Information gained through user groups or forums is purely anecdotal. Once this (unverified) solution to a problem is found, it must be evaluated and reviewed. This usually starts with a competitive analysis. Once it is determined that the proposed solution should indeed solve the specified problem, the solution will be implemented on a trial basis. If the trial is successful, the solution will be deployed on a wider basis. This process is called A/B testing.

Benchmarking in a Fictitious Organization

Software companies like Gaggle dot Com often find that there are two different ways to improve some aspect of a company’s IT department. To be concrete, suppose a certain database’s performance has been subpar. This can be improved by either changing some configuration property of the database or by replacing the database with one made by a different vendor.

By reading documentation and the forums of the database vendor, a configuration change may be found that improves the database’s performance. Or the problem can be described in a forum post along with a request for help. This is not benchmarking, however – it is more like self-improvement.

Real benchmarking would happen when Gaggle dot Com finds a benchmarking partner, say at a conference or user group meeting. This would be an informal and ad hoc relationship and would be the situation when a replacement database can be recommended. This is again anecdotal evidence, and it must be verified using A/B testing as described above.

In situations like this, a metric comparison with the benchmarking partner does not make sense simply because the entire processes (database plus everything else) are too dissimilar, like apples-to-oranges.

There are two quantitative comparisons that can be made, however. The first is to compare the performance of the database, acting alone, used at Gaggle.com and the benchmarking partner. In the classification that Yarrow & Prabhu (1999) present, this would be process benchmarking, where the “process” is strictly limited to the database.

The second comparison involves measuring the performance of Gaggle.com’s entire system both with the old database and with the new database in place. This is what Yarrow & Prabhu would call diagnostic benchmarking. As Yarrow & Prabhu (1999, p. 794) note, “diagnostic benchmarking is more akin to a `health check’ for the company, helping to identify which practices need to be changed and the nature and extent of performance improvements which should follow.”

In using either process or diagnostic benchmarking, Gaggle dot Com om has improved its database performance, perhaps even exceeding that of the benchmarking partner, and so have delivered a higher quality service to our customers.

Conclusion

The approach to benchmarking described here – combining process and diagnostic benchmarking – is not limited to the IT industry. As Wolniak & Grebski (2023) notes, any company can benefit from competitive benchmarking. In particular, Industry 4.0 is a transformation of traditional manufacturing processes using digital technologies, data exchange, and automation. As such, Industry 4.0 companies share the tools of IT companies, and the process and diagnostic benchmarking should be applied there, too.

References

Goetsch, D. L. & Davis, S. B. (2021). Quality management for organizational excellence: Introduction to total quality (9^th ed.). Pearson.

Wolniak, R., & Grebski, W. (2023). The usage of benchmarking in Industry 4.0 conditions. Zeszyty Naukowe Politechniki Śląskiej. Organizacja i Zarządzanie, 188, 665-676. http://dx.doi.org/10.29119/1641-3466.2023.188.40

Yarrow, D. & Prabhu, V. (1999). Collaborating to compete: Benchmarking through regional partnerships. Total Quality Management, 10(4-5), 793-802. https://doi.org/10.1080/0954412997820