Why Spy?

For a CEO who already has too much to manage, it may seem like a great idea: mystery shopping. Spot-checks in his company's stores by people posing as shoppers, with the aim of turning up useful information about which stores are surpassing company expectations for customer service and which are falling short. Clandestine? Without a doubt. Enlightening? Hardly.

If all customers were interchangeable or weighed different elements of the shopping experience equally or shopped at the same time of day, mystery shopping might have far-reaching value. But real customers don't fit those criteria. As a result, mystery shopping fails to reflect how a vast, varied customer base actually feels about a company's stores. And because it fails to capture customer experience, it has precious little to teach executives about improving customer service and is flawed as a predictor of customer behavior.

Unfortunately, mystery shopping is pitched as a tool for all those things. By 1999 the mystery-shopping industry in the U.S. was estimated at $750 million in annual revenues and growing, according to the Marketing Research Association. Fortunately, there is a better way. But first, full disclosure: Gallup is in the business of teaching executives how to better understand customers. Unlike the proponents of mystery shopping, we operate from the premise that to know how customers feel, you have to talk to them. More important, we're sure that statistical evidence and plain common sense support Gallup's approach.

Consider, for example, these four ways in which mystery shopping fails to live up to its hype:

• Mystery shopping produces a sample that is too small to be statistically valid. Many companies use one shopper a month, or at most three to five. But it's not uncommon for a popular retailer to have thousands of transactions a day. To suppose that five or so visits a month approximates the typical customer experience is folly. To further assume that those visits can meaningfully inform crucial decisions on marketing or management only compounds the original error.
  
• Mystery shoppers cannot speak for your customers. Even if mystery shoppers are demographically similar to your customers, they do not share the aims or expectations of a real customer. Accordingly, they're not likely to have a real customer's take on such service variables as courtesy, attentiveness, knowledge, or promptness.
  
• Mystery shoppers are paid to notice things that real customers don't, resulting in a distorted picture of a customer's experience. Imagine walking into a fast-food restaurant. If it is clean, and the service friendly and efficient -- and the food as you expected -- you probably will continue thinking about whatever was on your mind. But if your hamburger is cold or your fries are soggy, the excellence of the other factors won't even register. As a customer, you would give the restaurant a definitive thumbs-down. A mystery shopper, however, would give high marks in all categories but the food. Customers are not that meticulous or forgiving. They notice when a store delights or appalls them -- but not much in between.
  
• Mystery shopping cannot be used reliably to predict customer behavior or business outcomes. A retail giant that is one of Gallup's clients asked us to interview 500 customers, using the same questions a mystery shopper would use to evaluate a store. We analyzed the responses in relation to the customers' emotional attachments to a brand and their plans to stay loyal and found no statistically valid relationship. None. Anybody who implies that mystery shopping will unlock your customer's psyche is likely overpromising, to put it mildly.

In limited and tightly controlled circumstances, mystery shoppers can serve a purpose, however. At a major bank, for example, mystery shoppers test the wait time in lines and assess compliance with rules for hanging advertising posters. But even for such mundane tasks, managers must make sure that mystery shoppers are objective and that the elements of the shopping experience they evaluate are the same ones that real customers are likely to notice.

Mystery shopping has also been used to identify unfair and illegal business practices, such as gender or racial discrimination by brokers, lenders, cab drivers, and co-op boards. But the successful use of mystery shopping to expose civil rights violations is no reason to believe that it is appropriate as a gauge of customer service.

An alternative to mystery shopping is to elicit feedback from a statistically significant sample of real customers. For the amount of money spent on most mystery shopping programs, a more robust, reliable customer service system could be developed. A large organization could use Websites or toll-free numbers to conduct customer surveys that would produce vastly more and higher-quality data. Such computer and phone surveys are especially useful if a company has stores in many locations.

Or executives can opt for in-person customer interviews conducted, perhaps, as customers leave a location. This method is an especially useful data-gathering technique if you want to find out what customers think about a particular venue, such as a flagship location. The cost might be higher than mystery shopping; but such interviews yield real-time insights into customers' likes, dislikes, and intentions to visit the location again.

Whether from the Web, by phone or in person, information garnered from real customers can be organized into a report card that will inform managers about what needs improvement. Armed with reliable data about what works and what does not, regional and store managers can be confident that their most important actions, such as educating employees, implementing operational change and setting strategy, will be true to the needs of their customers.

Of course, companies must define their objectives and decide how best to meet them. Mystery shopping may show you how well your stores comply with quantifiable, objective standards, such as wait times. But to gauge the shifting truth of how customers feel and what they want, there's no substitute for asking them.