Metrics Myth Nine: NPS is Whatever I Say It Is

I’ll just say it right now: I love the Net Promoter Score.  It may not be perfect, but in my experience, it’s the most powerful tool that Service and Support executives have to demonstrate their value to the entire enterprise, become more strategic in how they think about their organization’s mission, and focus the entire company on the customer and their experience.  The NPS is an All Access Pass to the boardroom.

Here’s a brief definition for those unfamiliar with Net Promoter.  Industry research led by Fred Reichheld and his team at Bain & Company determined that customers fall into one of three buckets: they’re promoters who advocate for your brand, detractors who will badmouth you in the marketplace, or neutrals—people who may be satisfied, but who don’t rise to the level of advocate.  If we think of the companies we’re customers of, we can probably easily slot ourselves into one of those three buckets relative to them.

After much quantitative analysis, Reichheld and team determined that the most accurate way of predicting whether a customer is a promoter, detractor, or neutral is to ask them how likely they would be to recommend the company, on a zero to ten scale.  Nines and tens are promoters; sixes and below get classified as detractors; and sevens and eights are neutrals.  If you survey your customers, take the percentage of promoters and subtract the percentage of detractors, you get your Net Promoter Score—that is, the number of promoters net of the number of detractors.  Neutrals don’t count for or against you.  A company that scores all 9s and 10s gets a 100%; a company with all low scores is -100%.  Bain’s research suggests that most companies are in the low positive territory, while leaders can be plus 50% or higher.  (If you’re in negative territory…sell your stock.)

Unfortunately, because of its power, NPS has also gotten trendy.  And trendy measures tend to be misused, or to put it more nicely, “reinterpreted.”  It’s these creative interpretations that are the NPS myth.  Here are some examples I’ve seen:

Redefining the Scale

It’s a small tactical issue, but Net Promoter is based on the zero to ten scale with break points between six and seven, and eight and nine.  I see companies talk about the “NPS,” but pick entirely different scales or endpoints.  There can be good reasons for picking a different scale, but you can’t call what you calculate from it Net Promoter Score.  It’s something different, and the research on NPS may not apply directly, and comparisons with actual NPS scores aren’t valid.

Measuring NPS for Support

The Net Promoter Score is designed to “measure how well an organization treats the people whose lives it affects—how well it generates relationships worthy of loyalty.”  What we’re measuring is the customer’s impression.  And customers don’t care about your internal divisions.  They don’t have one impression of support, another of product development, another of your CFO, and yet another of HR.  By and large, they think about the company overall, and they’ll certainly recommend the company overall (or not.) They can’t say, “Well, buy your accounting software from Company X, but use Company Y’s support organization.”

So why would we measure the Net Promoter Score for Support?  “How likely are you to recommend Company X’s Support Organization to a Professional Colleague?” is an unhelpful question.  It doesn’t work that way.  You either recommend Company X, or you don’t.

As we mentioned earlier, it’s understandable to want to try to measure things that are only under your control, but that’s not how customers experience things, so it’s not a very productive exercise.

Measuring NPS Transactionally

Even more extreme than measuring Net Promoter for Support overall, is measuring Net Promoter for a single support transaction.  It’s a great thing to send (very brief, recurrence-limited) surveys to people who have engaged with your support staff or online resources—but let’s not pretend that their willingness to promote your brand is primarily based on their last support interaction.  Transactional and relationship surveys are different: in transactional surveys, you ask about the transaction, and only in the relationship surveys should you ask about overall impressions.

In transactional surveys, stick to asking about the case, or the web session, and leave it at that.

Compounding this error, I’m sorry to say that I’ve seen companies that selectively measure transactions, exempting entire classes of cases from surveys and giving individuals the option to strike particular troublesome interactions from surveys as well.  Needless to say, there’s enough sample bias and nonresponse bias in most survey programs that adding more with malice aforethought is really a terrible idea, and ultimately self-defeating.  This almost always is a result of focusing too much on the numbers.

Focusing on the Numbers

The root cause for many of these myths is a mistaken belief that the NPS numbers you report, by themselves, mean anything.

NPS, as you’ll recall, was developed as a way of measuring loyalty.  That is, higher NPS scores mapped into higher customer retention, lifetime value, and referenceability.  Since those are future behaviors, they can’t be measured directly, so NPS at its best is a predictor of good things happening in the future.

There is no NPS Bowl where companies or executives get rewarded because their numbers are high.  Whatever credit an executive hopes to get when he presents his “82% NPS” falls apart as soon as a skeptical audience member smells a rat and starts asking questions—at which point, credibility is irretrievably lost.  And, more importantly, although independent research has been mixed in supporting the predictive power of NPS, at least there is some.  Gamed numbers (based on modified scales, limited to support transactions, selectively sampled) have no research supporting their predictive power.

Fred Reichheld himself has said that the important thing about NPS isn’t what the number is; it’s what you do about it.  In the case of Support leadership, it can help line up the entire company on Support’s mission of the customer experience and customer success.  This is a Good Thing.  If we instead create unreliable, Support-only numbers to try to win a nonexistent NPS Bowl, we’ve done our customers, our companies, and ourselves a disservice.

Little Data

You can’t escape hearing about Big Data right now—it has replaced “cloud” as the current hot buzzword in tech.  The idea is pretty simple: it’s easy to gather huge volumes of data from Internet clickstreams, scientific instruments, roadway sensors, and all the other collectors of digital information in the world…but it’s not always easy to process it.  Big Data techniques are designed to rapidly sort through these piles of bits and bytes in order to gain meaningful, actionable business insights…such as the fact that you like your friends’ cat videos on Facebook.

So, the technical definition of Big Data is all about how to do correlation and trend-spotting in near real time on petabytes and exabytes of data.  But in the popular imagination, I think it has come to mean something a bit broader.  Big data evokes a sort of automated Big Brother—a bank of servers tirelessly poring through tweets, searches, mobile location data, and YouTube comments.  Except, unlike Big Brother, most of this surveillance is designed to figure out how to send you the perfect marketing pitch, all without human intervention.

It’s exciting technology, but it’s a little bit creepy.  In our business, I think we need a complementary approach.  We need to get really personal with our data.  Let’s re-discover Little Data.

Most service and support leaders’ questions are hard to answer just by crunching the numbers.

  • What one product change would help customers the most?
  • How much is knowledge shortening resolution times?
  • How effectively is self-service helping customers?
  • What’s the loyalty impact of a particular support initiative?
  • How well are we collaborating?
  • How many contacts are we deflecting, and are they the right ones?
  • What’s the ROI of knowledge?

These Little Data questions require judgment as well as correlation coefficients; nuance as well as regression analysis.  They require estimations, surveys, interviews, and other less-than-perfectly-precise and human-intensive inputs.

But, as measurement guru Dean Spitzer says, it’s better to measure the important things imprecisely, than irrelevant things with perfect accuracy.

I’m excited to have just wrapped up one of these projects, focused on voice of the customer.  We have lots of Excel sheets, and StatPlus got a workout.  But the interesting part was thinking about the right questions to ask.  To me, the most satisfying results were the stories that emerged from the numbers…and the discussions and innovation triggered by those stories.

Something we hear all the time is, “we have all this data, but we haven’t had a chance to really dig in to it.”  Maybe it’s time for a Little Data project.

ps – registration is now open for our next KCS Foundations Workshop.  Anyone up for February in California?

Summer Reading! The Intention Economy

This blog has no mandatory terms of service, installs no beacons or cookies (that I know about), and vends no third party ads (I hope—let me know if you see one.) That’s just fine with our second Cluetrain Manifesto co-author of the summer reading list, Doc Searls.

(I do plug our workshops, but I think Searls would be OK with that.  I think.)

The Intention Economy: When Customers Take Charge.  Doc Searls. (@dsearls)

Doc Searls is the leading light behind Project VRM, a movement designed to tip the balance of power away from vendor companies and back towards the customers they serve.  (VRM is a twist on CRM—vendor relationship management rather than customer relationship management.  We should be managing the vendors, rather than the other way around.)

Searls is a keen and wry observer of all that is absurd about the current situation.  He takes particular aim at big data efforts designed to collect information to target advertising to us—leaving aside the privacy invasion, it’s not at all clear that we want or respond well to targeted ads.  He also delivers a withering criticism of “contracts of adhesion,” the “click to accept” terms and conditions we acquiesce to at every turn that we don’t have time to review and that the vendor can change at will.  There’s a terrible imbalance of power, and customers are being badly treated. Amen.

If this book’s intention had been to draw attention to the problem, I’d give it full marks.  Unfortunately, he’s also proposing a solution, in the form of a scattershot of ongoing VRM projects.  When stacked up against Facebook, Google, Acxiom, and Apple, VRM looks like pretty weak beer indeed.

Searls clearly believes that the unbalanced relationship between vendors and their customers is bad for the vendors, and that enlightened self-interest (coupled with some good hacks on the technology side) will cause vendors to lower their drawbridges, fill in their moats, and embrace customers as equal partners in value creation.  The book is full of inspiring phrases like “free markets require free customers”—a fact that’s so blindingly obvious to Searls that it requires no further justification.  Ironically, he cites the notoriously secretive and intentionally non-transparent Trader Joe’s as just this new kind of company.  (Who really makes that Trader Darwin’s vitamin pill?  They aren’t saying.)

The thing is, no serious vendor I know buys this.  None is looking for new ways to cede control to their customers.

Searls’s defenders would say that this is unfair: that it’s early days for VRM, and revolutions take time.  But I think the premise is fundamentally flawed.

Fundamentally, Searls is betting on individual consumers to care more about their rights and the personalization of their experience than they care about their convenience.  From what I’ve seen, betting against customers choosing price and convenience first is wrong 100% of the time.  He’s also imagining a time when Apple, or whatever company takes its place, is going to be open to negotiating legal terms with customers.  I’ve worked with Apple Legal, and…no.

I’d deeply love to be wrong about all this.

I’d personally love a world where VRM was a reality.  I’d also like a world with unicorns.  Given the pace of genetic engineering, I’m guessing I get the unicorns first.

Just A Little About Surveys

Surveys are top of mind for me right now.  We’re in the middle of a couple of survey projects, and surveys inevitably come up whenever we talk about loyalty, satisfaction, and self-service effectiveness.  Doing customer service and support surveys is a topic that deserves its own book—as a matter of fact, it already has its own book, by Dr. Fred Van Bennekom, which I strongly recommend.

In this blog post, let me share just a few points that I think every customer service and support professional should know.

Keep relationship and transactional surveys separate.  Fundamentally, there are two different kinds of customer surveys with very different purposes.  Relationship surveys are taken by specific customers on a periodic basis, generally once a year, and seek to understand their overall perception of a company, its products, and services.  (For a B2B business, the customers are generally the economic decisionmakers or key purchase influencers, not everyone in the company.) The survey is often managed by Marketing, although we’d prefer that the Services organization drove it.  It’s about the company relationship as a whole, not one person, issue, or event.  It’s where you get to ask the big-picture questions: do you trust this company? Is it responsive and easy to do business with?  Would you recommend this company to a friend or business colleague? (This is the Net Promoter Score “ultimate question.”)

Transactional surveys are about a single interaction, full stop.  They need to be extremely quick and easy, and only ask questions about the interaction.  It’s not fair to ask about the company overall, or to try to calculate a Net Promoter Score.  The person who had the interaction gets the survey, whether or not she is the ultimate decision maker.  You should throttle transactional surveys so people don’t get hit too frequently, but once a quarter or twice a year is generally OK.

Transactional surveys let you know about your customer’s experience; relationship surveys let you know about your customer’s perceptions and intentions.  Both are important, but don’t mix them up.

Take a stand against bias. “Bias” is survey jargon for the degree to which a survey’s answers don’t accurately represent the answers you’d get from all possible customers.  There are many causes of bias—asking only web users or people who call you, for example, because they may not represent your customers as a whole.  Or, bias can come from simply not asking enough people.  But for most service organizations, the real danger is non-response bias—that is, the fact that the only people who take your surveys are the ones who really care, generally because they’re applauding…or apoplectic.  The silent majority isn’t counted.

Non-response bias is a particular concern for “did this article help you” questions at the bottom of knowledgebase articles.  In our experience, these usually get 0.2% to 2.0% response rates, and it’s easy to see that the other 99%, give or take, may be pretty different from their more opinionated brethren.  Don’t extrapolate from the people who answer you.

Test your surveys. Just as you’d test software or a new web site, you need to test survey instruments before you roll them out.  We’ve asked questions that were clear as the azure sky to us, and resulted in a “huh?” from customers…or, “well maybe you mean this, but maybe you mean that.”  It’s best to fix this before you launch.

Consider picking up the phone. I know, I know—this Internet thing is going to take off.  And we use web surveys all the time.  But there’s just no substitute for actually talking with customers, asking not only the scripted questions but open ended ones, too.  I think as an industry we don’t do nearly enough of this, and a “customer success on the web” survey is a great excuse to do it.  Here’s a simple script:

  • Thinking about the last time you came to our site, and thinking about the reason you came there, were you successful in accomplishing your goal?
    • (If yes) If you hadn’t been successful, are you entitled to open a case with us, and would you have done so?
    • (If no) Did you eventually open a case with us for that same reason?
  • Is there anything else you’d like to tell us about your self-service experience with us?

For more on using surveys as part of estimating contact deflection, see Simple Techniques for Estimating Contact Deflection.

Ask fewer questions, and never ask if you’re not going to do something about it. Tell people you’re doing a survey, and it’s like you’re giving away free cookies or something—everyone has their hand in the jar.  “Ask about the website.”  “Ask about our RMA policy.”  “Ask if they liked the hold music.”

Resist.

I’d like a transactional survey to be no more than three questions, and fit easily on one page.  One question would be better.  Five questions and you’re pushing your luck with me; show me seven questions and two pages, I’m outta here.  Unless I’m really cranky, in which case I’ll respond but you won’t like it.  Long transactional surveys make for low response rates and high bias.

Get only what you need.  And if you think the answer is interesting, but not something you’re going to do something about—for example, if you’d like to know how people feel about your ending support for a product, but you’re going to end of support it no matter what?  Don’t ask the question.  Customers want to know that what they’re telling you matters—that you’ll actually take action based on what they say.

What “aha” moments have you had in doing your surveys?  Please share in the comments below.

<shameless commerce> There are still slots available at the One Day Introduction to KCS immediately following TSW in Santa Clara, May 10.  Our KCS Foundations Workshop in Plano is sold out, but we have openings in the Bay Area Foundations Workshop in July. </shameless commerce>