by Patrick Campbell, CPCU, CLU, FLMI
Introduction:
While serving on an assignment as a director of quality (implementing a quality improvement program), I had the opportunity to develop, edit and mentor many team members on building service levels and the supporting measurement programs. Later, working in a center of excellence concept, I conducted quality reviews and audits of several IT and non-IT measurement programs. Finally, I had to execute measurement programs as director of a major operations center. During those periods, and even today, I have seen some measurement programs struggle and others flourish. Both the failures and successes seem to follow predictable patterns. In this article I would share with you what I have found to be some major reasons for less than satisfactory measurements and their associated risks.
First, let me mention some fundamental principles. It is common axiom that the path to improved performance is through measurement. You cannot improve what you cannot measure. From a supplier performance perspective, suppliers are improving, declining or staying the same. It is vital to know this, especially with your most important suppliers. Each condition calls upon specific action plans.
This leads to a second principle. Measurements are about continuous improvement. Measurements should not be viewed as an event, but as incidences within a pattern of trends. Performance measurements are about tracking trends. Programs need to be built for longevity with elements of repeatability, regular monitoring and feedback.
The third principle, for this discussion, is measurement programs need to be client (business units) usable. They need to be developed to support and assist clients to perform better and ultimately contribute to improved corporate performance. The question to ask is can the measurement be tied (directly or indirectly) to corporate performance?
Unfortunately, many measurements and measurement programs are not successful. There are several reasons for this which include lack of training, inadequate attention and insufficient planning. For example, I had a discussion with one supplier of a major bank, who told me that one of their clients installed a fairly impressive and comprehensive measurement program. However, since the inception of the program the client has, yet, to meet with the suppliers to have a mutual conversation on what is going well and what is not going well. This particular supplier wanted an opportunity to collaborate and improve, but the feedback mechanism was not there. A feedback process is essential and needs to be continually followed. What you make important becomes important
Let me share five issues that I have found to be common causes measurement programs to fail.
Inconsistent executive support
Performance programs fail, when there is dissimilarity between what management wants and the support design. Building a program takes time, money and talent. Additionally, executives need to provide consistent direction, both during the development and execution. They need to agree to the program goals and support those goals. When support does not occur, the program loses creditability and can flounder.
One of the most glaring examples occurred in the case where a supplier was chronically missing their targets. They were in jeopardy of being removed, according to the service agreement. Being concerned, they sought counsel with a senior executive. Upon hearing the supplier’s dilemma and explanations, the executive gave the supplier forgiveness and requested the measurement targets to be reduced. The rippling effect of that decision caused irreparable damage.
Lack of confidence in results
When there is a discrepancy in the results, there is an obvious creation of uncertainty and doubt. I have seen this, most often, in two situations: (1) the published measurements did not match the actual experiences of the client and (2) the computations contained errors.
In the first case, when the customer’s actual experience did not correlate to the published measurements, doubt was caste on the measurement. The most notorious situation occurs with availability and response time. A specific supplier may have a reported 100% availability, but the customer reports a different experience. One reason for this is that many suppliers (such as telecom) may be involved in the chain events providing availability, but not all in the chain are being monitored. Thus, an individual supplier could be performing well, but others in the chain are not doing well. The customer is experiencing substandard performance. The entire chain needs monitoring.
How could there be computation errors? In the second situation published data contained erroneous summations and computations. Why could this even occur? It is usually carelessness created when there is a rush or time driven requirement to produce a report. In a particular case at a large health insurer, the measurement developers did not conduct a quality walk-thru or test runs prior to information distribution.
Lack of clarity
Don’t speak “geek.” This is heard often, especially in the technology world. The language of the measurements must be understandable by the end-consumer. MIPS, MTTR, GIGABYTES, might be familiar terminology to the data creators. However, such abbreviations and technical terminology has little meaning to the customer. The Information Technology culture is saturated with “TLAs” (three letter acronyms). We need to use customer language, not the language of the technologists.
Sometimes, calculations can also be so complicated it takes an advanced accounting skill to interpret them. It can take many hours reconciling and validating. There needs to be simplicity to computations
Too many objectives
What is important? Suppliers do want to be successful and please their customer by satisfying their expectations. How many objectives are enough? Caution should be taken to avoid generating too many objectives. With too many objectives the supplier’s efforts become too diluted and there can be confusion on what is important. It is better to have a few objectives with high focus. Obviously, they can and will be modified and changed over time.
Silo thinking can lead to an inordinate list of objectives. Objectives grow, because every client has their special interests on what to measure. For example, if 5 clients are interviewed and they each identify 5 objectives, the count can quickly grow. Without a prioritization process, the objective portfolio can become too overloaded.
There needs to be a purging process for obsolete objectives. Otherwise, a measurement portfolio can grow and become uncontrollable. As measurement programs mature, new and improved measurement objectives will be created and older measures become less important. Those that are no longer as important need pruning. A lack of pruning can waste valued resources and provide information that no longer relevant, has little value and can confuse importance.
Inadequate Staff Continuity
Lack of staff training and experience are certainly important concerns with any program. However, I have observed inattention to staff continuity can be a major problem for measurement programs. Clear responsibility and accountability need to be identified.
But, what happens when that person in-charge moves to another assignment? It has been my observation the measurement process can “fall into the cracks” Either no clear successor is available, or there is a lag time finding a replacement. Regardless of the reason, measurement programs falter when they are allowed to wander and go unattended.
Summary
Supplier performance is either improving or declining. How do you know unless you measure that performance? Lack of an effective measurement programs can become highly emotional and frustrating. They can have detrimental effect on the business.
We can learn from other’s mistakes. If we do not learn from the past we are likely to repeat it.
Bibliography
Cope, James. “12 Ways to a Better SLA,” Computerworld, November 12, 2001.
Erickson-Harris, Lisa, Multi-tiered SLAs: Juggling Layers of Commitments, Research Director, Enterprise Management Associates, May 2002
Kass, Elliott, Director of Marketing, The Do’s and Don’ts of SLAs: When they’re executed correctly, service-level agreements help enhance IT’s credibility with the rest of the business by iCan SP, Inc., a Computer Associates Company, May 2003
Kuhn, Janet, ITSM: A Customer-Focused, Process-Oriented, Cost-Justified Approach to Success, ITSM Product Manager, InteQ Corporation Mauer, W., R. Matlus and N. Frey, “Guide to SLA Development,” Strategic Analysis Report, R-11-3353, Gartner Group, October 16, 2000.
Peterson, Brad. Partner, Mayer, Brown, Rowe & Maw, Ten Key Questions for Developing Effective Service Level Agreements, – Everest Partners, L.P.
Stanley, Shally Bansal. “Strengthen service-level agreements,” Network World, May 06, 2002
Strum, Rick. President and CEO, Enterprise Management Associates, Inc. “Are You Ready for SLAs?” Network World Fusion, February 2002
Underwood, Ryan. “10 essential ingredients for a solid service-level agreement,” 2003, DocumentIQ.com
Wickman, Robert, Director of Enterprise Monitoring Managing Service Level Agreements with C.A.R.E , Solutions, Empirix, 2003
Van Grembergen, Ph.D., Wim. “The Balanced Scorecard and IT Governance” Information Systems Control Center, Governance Institute, Rolling Meadows, Illinois.
Callaghan, Gary. “Project Dashboard,” DPI – Project Management Seminar, Public Works and Government Services, Canada, February, 2002.
Qaymumi, Mohammad. “Balanced Scorecard at CSU Northridge,” California State University, Northridge.
Neubert, Gilles, and Laure Pichot. “Performance Measurement for the Supply Chain – Mythology to define a supply chain dashboard”. University Lyon II – IUT Lumiere, Department O.G.P., Born, France, October 22, 2002.
Wang, Greg, PhD. “Value Learning: The Measurement Journey,” James Madison University, Educational Technology (2003), 43:1, 32-37.
Brown, Mark Graham, “Customer Satisfaction Measurement Mistakes,” Panorama Business Views, 2001
Performance Measurement & Dashboard Sampling
• Dashboards or Scorecards – What’s the Difference?
http://www.intelligententerprise.com/online_only/smith/010907.shtml
• Executive Dashboard – An interactive performance management & knowledge tool
http://www.anabasis-straub.com/dashboar.htm
• Digital Dashboard Business Process Assessment Guide from Microsoft and Spectria
http://www.spectria.com/docs/wp_bpag/BPAG.htm
• Snippets Active Dashboard
http://www.snippets.com/
• Defining e-Business Performance Management
http://www.cio.com/analyst/112000_hurwitz.html
• @McKinsey – ePerformance: Benchmarking global operating performance
http://www.atmckinsey.com/services/eperformance.html
• Keynote – The Internet Performance Authority
http://www.keynote.com/