REPORT: The Pros & Cons of C-Sat Surveys

Surveys are often plagued with biases and inaccuracies. The really big problem is that the information they omit or obscure can be what's most central to increasing customer loyalty.

From Inside the Trenches:
A Practitioner's Perspective on Customer Satisfaction Surveys

While customer satisfaction surveys make up a huge industry, which is not going away anytime soon, we find that companies frequently use poorly designed surveys that run counter to their best interests.

The list below details the many problems to watch out for, but the key underlying issue is this: Unless your survey is carefully constructed and executed with the right analytics, it will only uncover what customers say, and utterly fail to uncover how your customers feel. As Gallup Organization researchers have shown, this is significant because feelings—not thoughts—correlate with buying behaviors. [i]

Strategies To Consider

Strategy #1:

When conducting a satisfaction survey, be sure to augment this method with other satisfaction measurement methods. You'll benefit not because "more is better" but because measuring from a variety of perspectives with a variety of methods helps to confirm or deny the validity of your survey.

Strategy #2

Ask all parties involved with the design of your survey to keep this question top of mind: Is our survey measuring how well we do on our survey? If so, that's not very interesting. OR, is it measuring the feelings and needs of our real customers when they are in real situations with our company? If so, excellent! This is exactly what you want to be measuring.

Strategy #3

Be objective with your surveys. Look at your survey upside/downside/cross-side for problems. Survey questions should be carefully crafted to remove bias and to reflect customers' true concerns. On a basic level, surveys should always be conducted by outside, objective firms to ensure data accuracy. If this is not an option, at the very least work with a different department in your company to identify your survey's problems and gaps before it's released.

Strategy #4

Design your surveys and findings presentations with the end goal of facilitating next steps. Seek to combine qualitative insights with quantitative facts and incorporate media clips, quotes and anything else that will help your team decide how to move forward with its customer satisfaction strategy.

Academic Roundtable

Customer Satisfaction Surveys: Overview and History

We're so used to satisfaction surveys at this point that it almost seems like they've been around since the beginning of time. In fact, it wasn't until the early 20th century that Charles Darwin's half-cousin, the English statistician Francis Galton, invented the modern questionnaire, which has since come to define modern survey techniques. [i]

But the market research industry as a whole was born in the wake of the radical economic and technological changes of the post-World War II era. Suddenly customers had the option to choose among many high-quality products and began to differentiate products and companies on other attributes, like service.[ii] As the field of market research developed in response, it came to be dominated by companies conducting various kinds of surveys. Now both surveys and the companies that perform them are pervasive throughout the marketplace.

The Pros of Customer Satisfaction Surveys

It's pretty easy to identify the pros of surveys. There are a few simple reasons they're so widely used across all industries around the world:

  • They're quick.
  • They're cheap.
  • They can easily provide digestible quantitative data.
  • If done well, they present a useful trend line to management teams.
  • They give customers the opportunity to vent, which can in itself boost their overall opinions of a company.[iii]

The Cons of Customer Satisfaction Surveys

Despite their near-monopoly on the customer satisfaction measurement industry, forward-thinking companies may need to question their surveys or even dump them for the following reasons:

Sampling Bias

Sampling bias simply means that the sub populations being tested do not represent the fabric of the population as a whole. These are the two main types of sampling bias:

  • Sampling Error: If the sample is too small this will result in false information that does not reflect the true population. As CRM Buyer columnist Louis Columbus points out, this often leads to a secondary problem, when companies attempt to compensate by over-sampling.
  • Non-Representative Samples: This happens when the sampling method performed omits one or more customer tiers.

Response Bias

No matter how much care is taken in designing a representative sample of the customer base, the actual survey sample may be vastly different than the actual population due to response bias. Here are four major types of response bias:

  • Geographic Bias: Surveys may unwittingly leave out large sections of the population by polling in skewed geographic areas.
  • Temporal Bias: Phone surveys conducted during the day may be more likely to reach older segments of the population.
  • Changing Technology: Land-line surveys fail to reach a younger demographic which is increasingly reliant on cell phones. [v], [vi]
  • Problems of Motivation: Highly satisfied customers are much more likely to respond to survey requests than merely neutral or even dissatisfied customers. [vii]

Wording and Execution Biases

Let's say you've somehow managed to seal up all the problems with your survey sample design and you're confident your population is represented in the responses you've received. You're not in the clear yet; here's where the biggest problem of all may come in: Your questions themselves may be biasing your results. Customers have a variety of reasons—both intentional and unintentional—for answering in a way that does not reflect their true feelings. Below are four types of wording bias and execution bias to consider:

  • Lack of Options: The close-ended style of questioning used in the majority of satisfaction surveys can make subjects feel obligated to give an answer even if none of the options really reflects their true feeling. [viii]
  • Question Wording: Rewording questions in even subtle ways or asking questions with poor pronunciation (or with a dialect) can have significant impact on the number of "favorable" or "unfavorable" responses.[ix]
  • Bogus Questions: The questions are often simply irrelevant to the customer, who may respond more or less randomly to complete the survey. One of the most popular survey questions—"How satisfied are you"—has even been called the most "bogus question in the history of surveys".[x]
  • Happy Questions: Companies often use surveys to ask leading questions ("just wanted to make sure it all went well") that paint them in a favorable light. This may be done inadvertently, but companies also have motivations for hiding problem areas from management and executive teams.[iv]

Rigged Processes

The fact is this, employees often skew their own survey results. This happens for a variety of reasons: fear of demotion, criticism from management, links between survey results and employee bonuses, or even just lack of an outside perspective. Whatever the reason, a gamed system fails to provide informative survey results. Here are three possible ways of rigging satisfaction results:

  • Self-Administered QC: Quite often, the responsibility for measuring satisfaction levels relies on groups that naturally have a personal investment in receiving favorable satisfaction ratings. After all, how often have you received a comment card after you made some sort of positive remark about the service you received? And after you complained? Employees or management departments often sample only the most satisfied customers to boost their results.[iv]
  • Out-Right Cheating: Companies have other ways of inflating their scores as well. For example, if employees know that an an order is going to come in late they may push back the due date in the computer system.
  • Pressure to Produce Good Results: High motivation for customer loyalty ironically leads to high motivation to produce positive survey results, which can influence the development of biased research design.[iv]

No Meaningful Information Revealed

OK, so let's put aside all of these concerns and assume that your survey was perfectly conducted in every way. Sorry, but you're still not off the hook. No matter how great your survey design is, the responses may fail to reveal anything useful or valuable about your company's approach.

  • Trashed Surveys: Surveys often don't reach the people who could really use them—instead they wind up stashed in a back room or, worse yet, the dumpster.[iv]
  • Wrong Conclusions: Recently, the correlation between high levels of customer "satisfaction" and high levels of "loyalty" has been called into question. Surveys measure what people say, not what they do, and the two aren't always linked.[iii], [xi]
  • Meaningless Data: Many surveys ask meaningless questions: the questions may be important from a management standpoint but don't resonate with customers.[xii] Naturally this leads to meaningless data.

Interaction Metrics Free Solution

How useful and accurate is your customer satisfaction survey? For free, our experts will analyze your survey for biases and flaws. If they find problems, they'll make recommendations and give you a handful of hints for how to gather the best customer feedback.

References

  1. Fleming, John K., Curt Coffman, and James K. Harter. "Manage Your Human Sigma" Harvard Business Review. 83.7 (2005).
  2. "Francis Galton." Bio-Medecine.org. 5 February 2008.
  3. A Brief History of the Survey Research Industry in America Offering Frequently Asked Questions About Research. Port Jefferson, NY: Council of American Survey Research Organizations, 2001. Accessed online 5 February 2008.
  4. Klein, Mark and Arthur Einstein. "The Myth of Customer Satisfaction." strategy+business. Spring 2003. 05 February 2008.
  5. Columbus, Louis. "Measuring Customer Satisfaction Like You Mean It." CRMBuyer.com. 24 June 2005. 05 February 2008.
  6. "Fundamentals of Polling-Total Survey Error." ROPER Center Public Opinion Archives. 2008. 05 February 2008.
  7. Keeter, Scott. "How Serious Is Polling's Cell-Only Problem?" Pew Research Center Publications. 20 June 2007. 12 February 2008.
  8. Mazor, Kathleen M., Brian E. Clauser, Terry Field, Robert A. Yood, and Jerry H. Gurwitz. "A Demonstration of the Impact of Response Bias on the Results of Patient Satisfaction Surveys." Health Services Research. 37.5 (2002): 1403-1417.
  9. "Question Wording." American Association for Public Opinion Research. September 2007. 05 February 2008.
  10. Suchman, Edward A. and Louis Guttman. "A Solution to the Problem of Question ‘Bias'." The Public Opinion Quarterly. 11.3 (1947): 445-455.
  11. Hohman, Robin. "Ease of Use Means Big CRM Potential for Online Surveys." E-Commerce Times. 22 August 2006. 05 February 2008.
  12. Cusick, Bill. "Three Reasons to Be Wary of Customer Satisfaction Surveys." MarketingProfs.com. 7.1 (2008).
  13. Naumann, Earl and Kathleen Giel. "Checklist: customer satisfaction surveys." Entrepreneur.com. 24 June 1997. 05 February 2008.