Categories: Product Marketing
| On 3 weeks ago

NPS: The Lie Behind One-metric-fits-all

By Aggelos Mouzakitis

How NPS became a famous metric? The story behind.

“Companies can keep customer surveys simple”, Fred Reichheld wrote in his groundbreaking article in Harvard Business Review (2003). He introduced Net Promoter Score (also known as NPS) to the business world, as the one number that companies need to grow. 

As he believed, “A customer’s willingness to recommend to a friend (…) is determined by all the functional areas that contribute to a customer’s experience.” 

Ever since, NPS has been solidly implanted into many businesses, as one of the most important customer survey metrics. Yet, measuring customers’ satisfaction and decision-making process, turns out to be way more complex. 

Eventually, it came out that NPS is not only misleading when it comes to understanding customers, but it’s generally a non-accurate metric. 

Let’s take a deep dive into what NPS really measures and why it is not reliable, nor profound. 

 

The NPS formula and the delusions behind it

There is a simple process behind NPS calculation. 

The input comes from a one-question survey. The customer simply has to answer the following: “How likely are you to recommend [product / company] to a friend or colleague?”. 

The answer is placed among an 11-point scale, with 0 marked as “Not likely at all” and 10 as “Extremely Likely”. 

As this may look like a question that leads to statistical results, yet, NPS is not calculated by the mean of all respondents’ scores. Instead, there are 3 response segments

  • Users that scored from 0-6 are considered Detractors.
  • Users that scored 7s and 8s are considered Passive Respondents.
  • Users that scored 9s and 10s are considered Promoters.

The formula to calculate Net Promoter Score is the one below: 

Net Promoter Score = % of Promoter respondents – % of Detractor respondents

For instance, if we had the following 10 users scores: 1, 3, 4, 2, 3, 6, 7, 8, 9, 10 the NPS would be calculated as below:

NPS = 20%(Promoters) – 60%(Detractors) 

NPS = -40%

As you may have noticed, Passive Respondents’ scores do not participate in the calculation formula. The thinking behind NPS is that someone who gives a neutral score will probably not say good things about the company, only a user with a promoter score, would. 

So, the goal for companies would be to turn passive respondents and detractors into promoters. 

 

What is so wrong with the NPS question?

NPS is not an accurate number. It tells us what, but it doesn’t tell us why. It provides us with some sort of information that is not accurate and doesn’t provide any stable ground to rely on for future decisions.

Research questions that extract truly valuable information are about past user behaviour, not future behaviour.

A rephrased version of the NPS question referring to users’ past behaviour would bring far more accurate data. Let’s take a look at the questions below:

  • In the last month, did you recommend (company/product) to a friend or colleague?
  • As a new subscriber, were you recommended to us by a friend or a family member?

As a matter of fact, Netflix used these rephrased NPS questions and observed a regular increase in new subscribers when users answered “yes”. The difference between this phrasing is that it asks about user behaviour that has already happened, not some fortune-telling future one. 

NPS itself does not measure loyalty, nor growth. For instance, think about Ryanair.  

For the last 6 years, Ryanair has been officially described as the worst aircraft company. Yet, it’s one of the highest-selling ones. 

Just imagine what would Rynair’s NPS score would look like.

How would you respond to an NPS question after a flight with unpredicted delays, no leg-room, no food and bad service? You may not recommend Ryanair to a friend, but this doesn’t mean that you will not use their services again. In fact, it is very likely that even if you hate it, you will probably travel again with Ryanair, because of their low prices. So, “Best choice” can be far more different than “Delightful choice”. 

As it appears, NPS score, customer experience and loyalty, rarely match

How to fabricate a positive NPS score

If your performance evaluation is tied to an increased NPS score, we have good news, as NPS is quite easy to game. Let’s see how:

  • Ask the NPS question after a successful completion of a task (e.g. a purchase). Unhappy users will probably abandon the procedure and will never get there in the first place. This way, you only ask users who succeeded.

Can you imagine how higher the promoters’ score would be, compared to the detractors’ one?

  • Take response rates out of the game. Frustrated users will probably be unwilling to invest more of their precious time to give private feedback to a company/ product that didn’t help them do their job. So, even if the response rate is low, your NPS might look super positive. Yet, deep down you know the hard truth.

The above NPS techniques will safely lead to higher scores. Still, the spooky part is that it may look as if you have made important improvements, even if you made things worse (!)

 

They lie behind “one-metric-fits-all”.

User experience cannot be described through a single number. We will never have a pure user opinion about a company/ product just by asking a single question. 

How people think and evaluate their experience is way more complex than that. There is a functional, an emotional and a social energy that contribute to people’s decision-making process. 

Plus, people from different cultural backgrounds answer an NPS question differently. Imagine the answers of a German, a French and an English customer who had the very same user experience. Would they click on the same number button on the NPS scale? 

People have different criteria depending on their cultural, social, emotional and functional background. So, unfortunately, there is no one number that describes a company’s customer experience. Not even NPS. 

 

So NPS is totally useless?

NPS doesn’t represent any reality we live in. Yet, the responses to the following “Why” questions do. 

The customers simply tell you what happened and why they were frustrated. This is the real valuable data. 

We usually insert these Why-type questions into in-person qualitative user interviews. The responses often bring to light the problems as well as the working parts of the product.

Except for the “Why” question, the only, may I say, good thing that can come out of metrics such as NPS, CSat e.t.c. is observing the benchmark. A positive trend of NPS growth is always a good sign, while the observation of a negative trend can pull some concerns. 

So, if you already have some NPS insights, just observe the numbers trend and focus on the “Why” question. (If you don’t have one, consider adding it!)

 

Ok, NPS is not trustworthy. What could we do instead?

Surely business decisions would be made so much easier if there was one single growth metric to rely on. Maybe in a galaxy far far away there is such a metric. But on planet earth, the only key to unlocking honest customer experience feedback is in in-depth customer research. 

It may include much more than just one question and a single math formula, but it brings deeper and far more accurate results, which a company can safely use to drive growth.

Most of the time, automated feedback touchpoints combine wrong questions (such as NPS) on wrong flow moments. Tweaking those touchpoints with customer research intelligence will create a constant stream of trustworthy and useful user insights. 

The main idea is to discreetly insert some of the qualitative research interview questions into different stages of user flow. 

Of course, user experience will still not be measured and evaluated by a single metric. Yet, the goal is to build an automated information & feedback flow that will provide the company with user insights that really matter!

 

We can’t reduce user experience to a single number.

This is the biggest flaw of NPS. It appears to provide an easy solution to a problem that will never have such a simple way out. 

Customer experience will always be a combination of users’ interaction with the product and the brand, along with cultural, social, emotional and functional criteria. Every sequence of interactions is unique for every single customer, and, may I say, that is the beauty of it! 

Managers might be lured by the impossible outcome that NPS promises to deliver. However, we can bring more value to a company by avoiding such traps and using truly worthy measures, based on real customer research. 

Would you recommend this article to a friend or colleague? 

Aggelos Mouzakitis

I have over 8 years experience in Digital and Growth marketing. Currently, running Growth Sandwich, a London based Product Growth Lab. Before turning in Growth Marketing and Product Growth, I had the chance to pass by head marketing and head digital positions in Athens and London, work with numerous tech startups but also build and run companies. In my spare time, I am consulting ambitious startups about their Growth and Business strategy.