Business, Technology, Culture, & Ideas That Matter

The Corrupting Influence of Metrics

There is, every year, it seems, a bigger focus and greater reliance upon metrics.  Metrics can, of course, provide a much-needed focal point for management objectives, and provide valuable performance feedback.  One of the continuing issues with metrics is the  ‘gaming” behavior that metrics can drive.  Metrics inadvertently become the end, rather than a measure of the means.   Stories abound of unintended consequences.   What we see time and time again is a corrupting effect of metrics on what the metrics are purporting to measure.  Has our mad rush to quantify and measure everything resulted in the unintended consequences becoming the rule rather than the exception?

I have spoken with many companies about metrics, in the course of advising them about using software and technology more effectively in their operations to optimize their business processes.   In nearly all of these conversations, the issue of gaming comes up.

We all know what ‘gaming the system’ is.  If we are not directly in involved in work tasks in which we need to meet specified metrics, most of us are familiar with the strategy of under-promising and over-delivering – gaming our expectation-setting to make our subsequent results look better.   Exceeding expectations has become so important that it, well, has come to be the expectation.

Metrics in the Healthcare Industry

Two (2) recent articles have caught my attention.

These two articles, appearing together within a week of one another, highlight these issues in a very poignant way.

Janet Adamy, in her article, enlightens us on how the results of  a 27-question survey will be used to decide how some $1B of payments are allocated among hospitals.   The survey, administered by the Hospital Care Quality Information from the Consumer Perspective (HCAHPS), in Baltimore, MD., has been in use since 2006, but will only now impact hospital revenues.

What is the survey measuring?  Well, one question is:  “During this hospital stay, how often was the area around your room quiet at night?   (Never / Sometimes / Usually / Always ).”   Really?  Are nurses now responsible for enforcing a quiet time at the hospital time now?

With hundreds of thousands to millions of dollars at stake, who among us does not think that hospitals will manage to these 27 questions?   As patients, do we want our medical staff focusing and attending to our medical needs, or monitoring noise levels at night?

Full disclosure: I have no medical background, training, or experience. 

Hospitals are also complaining about the lack of relevance of the metrics and the unintended consequences.  Doctors who feel that they are in the life-saving business are now being told they are in the customer service business, requiring the staff to reallocate time spent providing medical treatment to time spent delivering customer service – to a customer that does not generally understand the product or the implications of the metrics.

At least one hospital has adding flat screen TVs for patients to solicit higher scores.   Another hospital has installed mini waterfalls in some patient rooms. Another hospital added ESPN because the consumption of pain medicine goes down during football games.

One hospital nurse in the Wall Street Journal story relates the story of a patient, who, after receiving treatment for a life-threatening stroke, dinged the hospital for cold meals.  Is a meal warmed to the proper temperature as important as a life-saving medical treatment?

The healthcare industry may prove an exceptional case study on the questionable use of metrics and the law of unintended consequences.

A study earlier this year discovered, in fact, according to its abstract, that patients with higher customer satisfaction scores experienced “greater inpatient use, higher overall health care and prescription drug expenditures, and increased mortality.”  (Subscription required for full access to the study. )

The other article, from the Boston Globe’s Robert Weisman, alleges that Blue Cross Blue Shield of Massachusetts is boosting up its customer service processes for the one week that coincides with a survey of its practices.  While  Blue Cross may claim that its revised customer service practices are pilots for new processes, the timing is certainly curious, to say the least, leading to the unavoidable suspicion that there is an effort afoot to juice the survey results by boosting customer responsiveness.

We see reports all the time of aberrations like this, most often when there is an inordinate focus on the end results, and a “no excuses” pressure on “making the numbers”.

Metrics in the Broader Context

The healthcare industry is, of course, a special case, particularly so given recent extraordinary changes in the industry.

In the broader context, Andrew Stein points out, in The Trouble With Metrics, that there is a significant distinction between “change”, and “transformation”.

Andrew Stein argues that if you want to drive behavior a certain way to achieve specific results, better to think in terms of  transformation, and use metrics to measure the behaviors over time. Focus on metrics as results, and expect behaviors to change short-term to achieve the specified metrics, without any long-term transformative effects.

While there is often an almost irresistible urge, particularly for management, to focus on and measure near-term results, and manage by the numbers, longer-term success requires that a focus on quantitative results be balanced with metrics that focus on shaping the cross-functional process behaviors that drive excellence.

There are other things to watch out for when instituting a set of metrics:

  • Having too many metrics
  • Understanding what the metrics meant and were trying to measure
  • Anticipating how specific metrics are likely to drive, rather than measure, behavior
  • Cultural resistance over the definition of metrics
  • Constant churn in the metrics being used
  • Data availability to measure the metrics quickly and consistently
  • Stovepiping: metrics that encourage siloing
  • People gaming the system to generate the desired metrics, compromising sound practices

Metrics and The Customer Experience

Both the Wall Street Journal article and the Boston Globe are addressing customer-oriented processes.  And when it comes to customer-oriented processes, quantitative measurements may not even be the best way to measure the effectiveness of these processes.

There has been increasing focus over the past several years on something called “the customer experience”.   For a variety of reasons, and in response to several trends, there has been an increasing focus on adopting a formalized “outside-in” view and discipline in order to orient and refine internal processes to be more responsive to the customer.

As Forrester Research has stated, however:

“While NPS (Net Promoter Score) and other quantitatively focused measurement techniques help firms understand the current state of their customer experience, they’re simply not the right tools to understand what kinds of experiences customers truly want and need.” (2012 Customer Experience Predictions, Forrester Research, by Kerry Bodine, January 6, 2012)

It is all the more interesting, then, from multiple perspectives, why more companies are not more actively adopting a strong approach to the Customer Experience.  This has been mystifying to me for as long as I can remember.

Forrester has been reporting that the amount of churn in the top ranks of people overseeing the Customer Experience seems to indicate that many companies are still not as serious about this Customer Experience focus as they could be.

If the Wall Street Journal and Boston Globe stories are any sort of indication, companies might be better served addressing the underlying processes themselves, rather than trying to set up measurements that result in such efforts to game the results.

Best practices for Using Metrics

There is no single set of Best Practices than can address all of the possible uses of metrics.  However there are a set of general Best Practices that can help to maintain a set of metrics that have more integrity and more accurately measures the targeted behaviors:

  • For best results, use metrics to drive longer-term behavior transformation. In the shorter-term, metrics may encourage more short-term behavior modification and gaming of the system, with less long-term sustained results.
  • Do not make the mistake of thinking that all metrics are results-oriented.  The best metrics may not only be process-focused, but cross-functionally process-focused.  Results-oriented metrics are the metrics that may be most susceptible to being co-opted as the numbers to be achieved at all costs, rather than as a target or  measure of the results to be achieved.
  • Think long and hard about the behavior(s) that you are trying to drive, and then structure metrics in such a way that gaming the system is at least as difficult as the target behaviors.
  • Use metrics that are measurable, and measurable in an easy and consistent a manner as possible.
  • Anticipate how your metrics will encourage responsibility-shifting and -dumping, and then try to refine the metrics to encourage a more responsibility-assuming behavior, where appropriate.
  • Focus metrics, where appropriate, on the ideal client experience, and how metrics can be used to drive behavior that is more focused on continual improvements to the client experience.
  • Be sure to apply the metrics at the point where there is responsibility and authority to transform behavior.  Applying a metric to a result or behavior over which someone has no control or authority is likely to be very counter-productive.
  • Follow up your metrics regularly and frequently early on to assess whether the metrics are shaping  behaviors as intended.   Refine the metrics where necessary to minimize or stop instances where results are being compromised by efforts to game the metrics using unwanted behavior(s).

While there is a lot of effectiveness and efficiency that can be driven through an organization through the use of appropriate metrics, the effort should not be underestimated.   Metrics need to be established by people who know what they are doing, who are proficient in understanding and managing efforts to game the system, and who can recognize and quickly respond to unintended consequences.  Metrics will seldom be any substitute for a sound focus on the Customer Experience.


October 21, 2012 Posted by | Business | , | 1 Comment

10 Best Practices For a Better Website

Reposted from The SMB Research Blog

Have you read enough pieces yet on what makes a great website?

You have probably seen quite a few articles and posts addressing this.  We here at SMB Research know we have seen a lot of material. (SMB Research includes at least one fairly good article on SMB Research’s Top 20 Favorite Reads of 2010.)

Many of these articles seem to take a cookie-cutter approach:  it should go without saying at this point that a website needs to have good navigation, and that a website needs a “Call to Action”.

We think there are a number of other critical website features and best practices that get too little attention.

Continue reading

February 17, 2011 Posted by | Technology | , | Leave a comment

Great Learnings from Boston’s Small Business of the Year Awards

SMB Research was very pleased to be able to attend the Greater Boston Chamber of Commerce’s 26th Annual Small Business of the Year Awards yesterday (Oct 14, 2010) at the Colonnade Hotel. Emceed by WBZ Radio’s co-Anchor of “WBZ Morning News” Ed Walsh, the event itself was flawlessly put together and efficiently executed, for a very enjoyable event. Better yet, the winner’s stories and remarks were compelling and inspiring.

Continue reading

October 15, 2010 Posted by | Business, Ideas | , | Leave a comment

Steal This Blog Content

SMB Research, where I spend much of my time these days, has now had our blog content illegally copied multiple times, and we are, sadly, hardly alone.  It seems apparent that too many people either do not bother to learn the ins and outs of intellectual property and copyright, and how to properly blog, before they start blogging, or people are so utterly incapable of creating their own website content and disrespectful of other people’s intellectual property that they simply covet and steal other people’s IP. Continue reading

April 27, 2010 Posted by | Ideas | , | 1 Comment

The New Complexity Management

I have had a working assumption for awhile – we all have one or a few of these, don’t we? Something to keep the mental gears oiled and in good working order when our minds are not otherwise actively or productively engaged?

One of my working assumptions – stay with me on this – is that we are increasingly over-estimating our collective intelligence, and over-designing our world, our processes, our gadgets – our things – as a result. Continue reading

February 18, 2010 Posted by | Ideas, Technology | , | Leave a comment

Your Bureaucracy is No Excuse (a rant)

How is the customer service you have been getting lately?

Our customer service should be better than ever, shouldn’t it? After all, the slackers and underperformers have all been laid off.  This should result in our getting better customer service than ever.

Sadly, the tale I am about tell is, I fear, more common than not. At the risk of painting with a broad brush, I must keep the parties involved unidentified for the simple reason I would prefer to avoid the rath of any Kafkaesque bureaucracy right now who could make my life miserable by losing a check, my paperwork, or my record. Continue reading

January 21, 2010 Posted by | Business | , | 2 Comments

Do I Know You? Identity Authentication in the Real and Digital Worlds

I had occasion recently to sit down to talk with Reliable Identities Wes Kussmaul, accomplished entrepreneur, author, and prolific Internet tinkerer.  Wes experienced early success founding the Delphi Internet service.  (Some of the more seasoned among you may remember Delphi as being one of the early online services, along with CompuServe, Prodigy,  and AOL.)

These days, Wes is talking about another frontier – identity authentication.  It is interesting to think that we are as inattentive to establishing and authenticating everyone ‘s digital identity as we are focused on proving and authenticating people’s identities in the real world. Continue reading

November 11, 2009 Posted by | Technology | , | 1 Comment

Passwords (Not) As Easy as 123

Wired’s Kim Zetter reports over at the Threat Level blog (“Most Common Hotmail Password Revealed!“) that “a researcher” did a study of the compromised Hotmail, MSN, passwords and found that “123456” was the most common password.

Are you kidding me?  C’mon people, we can do better than this. If this wasn’t so damn pathetic, it would be funny. Continue reading

October 7, 2009 Posted by | Technology | , | Leave a comment

David vs. Goliath

In the latest article by Malcolm Gladwell in the New Yorker, How David Beats Goliath , in the May 11th issue, Gladwell once again offers up some pithy observations and insights, this time on the sometimes advantage of effort over ability. Much of the article talks about how TIBCO‘s Vivek Ranadivé used the full court press strategy with his daughter’s basketball team to overcome many more skilled opponents. Continue reading

May 20, 2009 Posted by | Business, Ideas, Technology | , | Leave a comment


%d bloggers like this: