REPORT: The Pros & Cons of Net Promoter Score

While easy to understand, Net Promoter Score is not the final answer in customer satisfaction measurement.

From Inside the Trenches:
A Practitioner's Perspective on Net Promoter Score

Net Promoter asks customers how likely they would be to recommend a friend to a particular company. Mostly, we find that customers do not engage with this question or respond to it in a reflective or meaningful way.

Think about it like this: Imagine for a moment that you're at a cocktail party with some friends and associates. The conversation turns from politics, work and upcoming vacation plans to a discussion of various local rental car providers. Jim mentions a recent business trip, and tells of the $19.99 a day rate he received. "They even let me upgrade to a luxury car for free," he raves. "On your next trip, you just have to rent your car from them." Can you ever imagine this happening? It seems unlikely at best that anyone would devote much time to thinking, much less talking about or referring their friends to AVIS, Budget or Enterprise.

In a cultural climate that values authenticity, we find the Net Promoter question to be strikingly fake. When asked a similar question, one customer we surveyed exclaimed, "Well, I can tell you this: I just wouldn't go out of my way to have any sort of conversation about that company. It simply wouldn't come up." This customer expressed what we believe is a common response to the Net Promoter question.

Strategies To Consider

Net Promoter Score Strategy #1

What Net Promoter has going for it is its simplicity and simple metrics are a great way to unify staff toward important goals. Therefore, find a few factors (this could be wait-time, attire, accessibility, etc.) that link fundamentally with profitability. Monitor these metrics on a regular basis and post the results in a visible location for all employees to see.

Net Promoter Score Strategy #2

Net Promoter is an outcome metric. In addition to providing outcome metrics, ask your researchers and consultants to identify the process metrics (quality of answer, attentiveness of staff, etc.) that are causing the outcomes you have achieved.

Net Promoter Score Strategy #3

Stop asking "would you recommend…?" Get a better question. Propose a question that is thoughtful, branded and tailored to your industry.

Academic Roundtable

The History of the Net Promoter Score

Harvard Business Review, 2003: Loyalty consultant Fred Reichheld presented a new way to measure customer satisfaction. He claimed that his new measurement method would give companies valuable data about customer repurchase rates. In fact, he even said that "it would be 100% accurate in determining revenue growth" and that it was "the single most reliable indicator of a company's ability to grow."[i]

Reichheld was so impressed with his metric that he ambitiously entitled his article "The One Number You Need to Grow."[ii]

Since this initial publication, the popularity of Reichheld's number—AKA Net Promoter Score or NPS—has exploded among executive management teams. On the heels of his success, Reichheld came out with a book called "The Ultimate Question: Driving Good Profits and Growth" and it quickly became a best-seller. By this point even Net Promoter critics had to admit that "it would be difficult to overstate the impact of Net Promoter on management."[iii]

Net Promoter Score: Controversy Abounds

Recently, however, a debate has erupted over the true value of using Net Promoter for accurate growth predictions. In 2007, Timothy Keiningham—a Senior Vice President at Ipsos Loyalty—and a group of researchers published a study in the Journal of Marketing that appeared to refute the NPS as a successful predictor. Keiningham's study—the first attempt to replicate Reichheld's original results—found relatively low correlations between NPS and a change in revenue. Keiningham then boldly asserted that "Net Promoter in no way would be categorized as the ‘single most reliable indicator of a company's ability to grow.'"[iii] 

Reichheld Admits the Question can be Irrelevant

After the publication of Keiningham's research others began to propose more conceptual concerns with NPS. In a January 2007 article in Marketing Management, D. Randall Brandt (Ph.D. and Vice President at Maritz Inc.) argued that the Net Promoter metric "isn't always the best predictor of customer behavior or business results; some measures perform better in particular markets and sectors, or for particular types of customers" (22).[iv] Reichheld himself even admitted that the question is "simply irrelevant" in certain markets and sectors (qtd. in Brandt, 22).

The takeaway is: sometimes it just doesn't make sense to ask the referral question. In B2B situations where individuals have no control over purchasing decisions or in monopolies, Net Promoter will have little impact on repurchase rate. And sometimes it just doesn't make sense to refer anyone to a particular company.

Net Promoter is Strictly an Outcome Metric

Net Promoter does NOT tell management teams what's working and what needs change. According to Brandt, while Net Promoter may have some predictive capacity for broad customer populations, it says little about what motivates individual purchasing choices. Additionally, knowing the overall Net Promoter Score doesn't give any information about the makeup of the Promoter and Detractor populations. Because of this, Net Promoter Scores have little to offer management teams in the way of actionable next steps. "Without looking at the promoter and detractor scores, grossly incomparable scenarios could be treated as the same—and either inappropriate or no managerial action might ensue" (24). Brandt instead suggested a "longitudinal" approach in which managers examine multiple candidate measures" (24). This way, researchers could apply a variety of metrics aimed at rooting out the sources of resonance and dissonance between customers and companies. With these results management teams could then make process and policy changes with certainty.

Summed Up: Net Promoter is like many other metrics that executives have at their disposal: Sales per Day, Conversion Rate, Order Size, etc. Like high sales numbers, high Net Promoter Scores are terrific, but to manage growth and success, executives must toil with the underlying causes that determine their successes and failures–otherwise luck is the strategy and a win is as likely as a loss.

Reichheld Responds

Reichheld has responded to the controversy over Net Promoter, admitting some potential inaccuracies but continuing to defend his metric against critics. "[My] critics are missing the forest for the trees. Net promoter is effective, because it forces top executives, and other managers, to focus on creating happy customers. Too much market research is based on complicated formulas and long questionnaires that few customers complete. The simplicity of net promoter—one question, one number to track—rankles market researchers vested in other approaches."

Keiningham Remains Unconvinced

Responding to a posting on the Journal of Marketing blog, Keiningham reemphasized his assertion that Reichheld's claim to success was based primarily on research bias and that the support for Net Promoter derived largely from anecdotal "testimonials" with no validity in a scientific framework. Keiningham seemed to get somewhat emotional in his post, writing that "While we can all agree for the need to have measures that are easily understood and used by managers, that is completely irrelevant to the issue at hand. Regardless of whether or not one chooses to believe in Net Promoter, we all must insist that the evidence used to support the metric be unbiased...[otherwise] there is no reason to believe anything we say."[v]

Regardless of the Forecastability of Net Promoter, Interaction Metrics Proposes Three Other Research Considerations

Net Promoter Response Bias: Research in other industries has shown that the most satisfied customers (including patients and clients) are the ones most likely to respond to surveys, leading to artificial inflation of satisfaction ratings. A 2002 study of response bias in health care patient surveys found that those "who use patient satisfaction surveys should be aware that response biases may impact the results of such surveys, giving the impression that patients are more satisfied than they in fact are."[vii]

Net Promoter Question Bias: The Net Promoter question presupposes a positive response. For example, "How likely would you be to refer a friend" assumes from the outset that you would be at least somewhat likely to do so. Studies have shown that wording questions differently often leads to vastly different results for "favorable" and "unfavorable" responses.[vi]

Net Promoter Question Validity: This issue is apparently seldom addressed with respect to Net Promoter Scores. The Net Promoter question is by nature close-ended, and these types of questions have various benefits and disadvantages: They are easier for measurement purposes but force a binary choice on the respondent, when the actual answer might be somewhat grey or open-ended.

Interaction Metrics Free Solution

Net promoter isn’t right for everyone. For free, get an expert opinion on whether the Net Promoter question is right for you. If it’s not, we’ll give you a better question that’s customized for your company.

References

  1. Netpromoter.com. 2005, 2006. Satmetrix. 8 February 2008.
  2. Reichheld, Frederick F. "The One Number You Need to Grow." Harvard Business Review 1 Dec. 2003. 2 February 2008.
  3. Keiningham, Timothy L., Bruce Cooil, Tor Wallin Andreassen, and Lerzan Aksoy. "A Longitudinal Examination of Net Promoter and Firm Revenue Growth." Journal of Marketing 71 (2007): 39-51.
  4. Brandt, D. Randall. "On the one number you need to grow, one size doesn't fit all." Marketing Management 16.1 (2007): 20-25.
  5. Marketingpower.com. 12 July 2007. Journal of Marketing Blog. 2 February 2008.
  6. Aapor.org. 2006-2007. American Association for Public Opinion Research. 2 February 2008.
  7. Mazor, Kathleen M., Brian E. Clauser, Terry Field, Robert A. Yood and Jerry H. Gurwitz. "A Demonstration of the Impact of Response Bias on the Results of Patient Satisfaction Surveys." Health Services Research 37.5 (2002): 1403-1417.