This approach of asking people their opinions brings us to a serious mistake made by those new to survey writing.

Perceptions are not the same as facts.

We’ve heard the stories where several people see a wreck or a robbery and each have a different eye-witness description.  Our perceptions are subjective.  Zuckerman and Jost (Social Psychology Quarterly, 2001, Vol. 64, No. 3, 207-223) remind us that Garrison Keillor was onto something in describing Lake Wobegon as a place where “the women are strong, the men are good looking and the children are above average.”  They note a Stanford study where 87 percent of MBA students rated their academic performance in the top two quartiles (top 50%).  Almost everyone thought they were above average as students. In general, we tend to see ourselves as better than average.

Zogby noted one of its polls (http://www.zogby.com/news/ReadNews.cfm?ID=1781) where 72% of Americans said most of the country does not follow the news from Washington, yet 85% said they follow the news closely.

So, knowing that people are not the most accurate judges of themselves or others, what do we do with this knowledge?

First, stay as close to the source as possible.  Rather than asking how the economy is faring, ask each person how they are doing, perhaps compared to a year ago.  Rather than asking if they think the country is going to elect person A or person B, ask how they plan to vote.

Next, be as specific as possible.  Ask their opinion in the context of their last purchase or their checkbook balance, giving them an anchor point in which to examine their situation or feelings. Vague questions result in vague answers.

Third, understand how to use your findings.  Realizing these are perceptions, try to find out what caused them.  Ask people why they are very satisfied or not at all satisfied.  Ask what is better or worse than a year ago.  “Why” and “what” questions can generate amazing insight.

We live in a world of our perceptions.  Customers may even perceive a problem that doesn’t exist.  But those perceptions are important to them and it should be important to our clients.

Reminder #1: People have fuzzy memories.

Our brains are constantly collating and regrouping our memories. This has real implications when it comes to customer satisfaction. If someone asks, what do you think of a company or a person or an event, our answer often involves a combination of the many times you have had such experiences. You’ll give an answer that may not reflect any one encounter.

That’s why, in a customer satisfaction survey, you need to be careful how you ask your questions. If you are trying to monitor satisfaction and look for any shifts, don’t ask the more generic version of the question. Instead, I recommend you ask those responding to think back to their last visit or purchase. That way, if something bad or good happened, you are more likely to hear about it. Otherwise, someone who likes a company might ignore the recent bad experiences when they answer until the bad experiences become overwhelming.

Put another way, with the generic question, there will be more of a delay in tracking changes. By asking about the last visit or last experience, you’ll know something is up if 5% of respondents had a bad experience one month and, in the next month, it rises to 30%.

It is difficult for individuals to give a good answer if the question is too vague.

It’s a wonder of science how quickly the human eye can glaze when someone hears the words “margin of error.”  People want to know the best or the biggest, but they often have no interest in knowing how certain we are about those responses.

Survey results and studies are often the basis for business decisions with incredible amounts of money on the line.  So, don’t we have a duty to try to show our clients whether they can trust our findings before they act on them?

At Zachry, we make sure our clients can see our best estimates of the error rate and confidence involved in our research.  After all, what does the chart below mean?  It may mean the majority favor person A.  Or, it may mean the race may be too close to call.  It may even be possible that B is the favorite.  The interpretation depends on how well we do our work and the level of error inherit within the survey.

Yet, today, some nationally known organizations have practically abandoned efforts to track error.  One group whose name you would recognize references all the kinds of error that could be present, saying “margin of error” may be misleading.  Then they say, “Because the sample is based on those who agreed to participate in [their] panel, no estimates of theoretical sample error can be calculated.

On the one hand, give them credit for explaining their limitations.  But, then, take away that credit because their news release proclaims, “X percent of Americans think Y.”  The top of their release reads as if their findings are absolute, even though they can’t attach any estimate of error or confidence to anything they say.

We know limiting and measuring error is hard, but do we really want to use techniques where we have to throw in the towel?  Is this group saying they have no idea whether their measures might by off by 5 or 50 percent?  At some point, if a technique means one can’t control or measure their error, why bother with the research?

At Zachry, we feel a responsibility to make sure we give our clients findings that are worth acting upon.  Consequently, we use methodologies and techniques that allow us to estimate the accuracy of the findings.   Our approach may not always be the least expensive, fastest or easiest way to go, but why report results if you have no idea whether they are accurate?

Occasionally, we all need to take a deep breath.

So, this blog is written for those businesspeople who work hard to correct and handle image problems, both real and perceived.  It’s written for those who work hard to increase customer satisfaction, trying to squeeze out an approval rating of 9.6 instead of 9.4.  It’s for those who can name every major issue customers have had with a brand or company, and quote the official company stance on each.

To those people, I say back away slowly.  Relax for a moment.

You are so caught up in your work that you don’t realize the single biggest complaint most people have with your company is probably…

Nothing!

Recently, we studied an organization in an industry that has taken a beating in the world of public opinion.  We asked their customers, “Can you think of anything you don’t like about this company?”

The answer we heard most often was “no.”

We asked, “Is there anything you wish this company would change to serve you better?”

Again, the answer we heard most often was “no.”

We asked, “What one thing would you like this company to do that it is not doing now?”

Most people answered, “Nothing.”

We surveyed another industry, asking “What do you dislike the most about this company?”

The most frequent answer was “nothing.”  Everything was fine.

So why the disconnect?  Don’t your customers care?

It gets back to the old research idea of sampling.  It is those customers who are most upset who tend to write, call or complain. They are the loudest.  But, that group does not necessarily represent the customer population as a whole.  Most of your customers don’t think about you as often as we believe.  When they need you, you come to mind.  Otherwise, they are busy living their lives.

So, relax a little.  Go ahead and work on the problems.  Address the concerns.  But realize there are many people who like you and your company just fine.

Yet, every day, many intelligent people with company websites use an electronic version of that box on the counter.  It has become so easy to generate a survey and link it to one’s website that we aren’t even surprised to see a pop-up or a request to share our opinions.

Recently, a financial institution hired our company to study customers living near their branches.  This client had used web-based surveys in the past and had made decisions based on the findings.   We gathered data on the phone, and did not let people self-select themselves into the study.   Having access to their web studies, we compared our phone responses with responses taken from the website.

The key is that self-selected website surveys reached only a subset of the financial institution’s customers and was NOT a reflection of all their customers.  Those who participated through the internet had key differences compared to those of the carefully generated random sample contacted by phone.  While website surveys may generate some interesting results, marketing decisions should be based on sound methodologies that produce an accurate picture of all customers!

Some might say their situation is different since their company doesn’t have brick-and-mortar locations.  But it is still like that box on the counter.  Those who grab the survey likely aren’t the same as those who would ignore the box.

In our business we sometimes hear, “It’s not perfect, but it’s better than no feedback at all.”  Think again.  Making decisions based on bad information is rarely a good idea.

The problem is that focus groups have very limited utility.   Let me say that again…VERY LIMITED!    I have found that it is often dangerous for clients to observe focus group sessions (or even read summary reports of those sessions) because they often create expectations that may not at all be appropriate.

Are focus groups valuable?   They can be.   But most groups suffer from a serious flaw that almost always skews the results.

Focus Group research can be exceptionally valuable in planning, strategy development and message evaluation.  Even though this research technique is often misused, this “qualitative” exercise can produce information that could never be gathered through any other research method.  When I say the method is often misused, I am referring to people/companies conducting focus groups and then making the assumption that the data gathered are generalizable to a larger population.    Focus groups can’t reflect a larger group with measurable accuracy – only a quantitative study based on randomness can do that.  But that does not mean they are useless.

The greatest value of focus groups is that it takes advantage of the human side of communication.  It recognizes the impact that human interaction in a social structure has in shaping an individual’s perceptions and motivating that person to act (or not act).  Another significant advantage of the technique is that researchers can put printed materials in their hands or expose them to other aural or visual stimuli.  When one adds this capability to the situation that fosters peer interaction and information sharing (approximating the opinion leader process), the technique can produce insightful results.

The key is selecting the “right” people.   It’s entirely possible that when you measure, let’s say, 4 focus groups, each having 15 people (60 people), that the majority of those people are not at all representative of your target group.   There are some things you can do to increase the representativeness of the people you are studying.

The most important thing you can do is to NEVER  let people volunteer for the groups.   Most people would not volunteer, so that means that those who do volunteer are not like the majority of the people you are wanting to study.  The key is to draw a random sample and then “ask” them to participate.

Here’s an example.  A few years ago I did a series of focus groups for an automobile manufacturer.  They wanted in-depth comments from buyers about their buying/dealership experiences.  They were trying to figure out why some dealerships had good sales and others in the same metro area with similar demographics did not.   And why some dealerships did a lot of company financing and why others did not, etc…..    So they sent me a list of every person who had recently purchased one of their cars in some selected metro areas.   I had each buyer’s name,  address,  phone number, the dealership of purchase, types of financing and the type of vehicle they bought.    I matched those local sales with the sales profile of the vehicles they sold.

Then once the profiles were drawn, I made sure I invited people who bought all types of vehicles  (sedans, SUVs, pick-ups, sports cars, etc) and also included the proper proportion of fleet and corporate buyers.  We send letters and made personalized follow-up phone calls.   In the end,  I had a good group of buyers of each type of vehicle,  from each dealership (in proportion to their sales),  two types of financing customers, plus cash customers.   And to keep the effects of “convenience” from being too great, I held the groups in two hotels in two different parts each city.   In other words, it was a lot of work making sure the right people were there.

Had I just sent a letter to the list they provided and let people volunteer into the groups, I would have had a ton of people who were unemployed or lonely,  too many who lived close to the research sites,  too many people who had purchased inexpensive cars (not busy working),  no corporate/fleet buyers, and a bunch of folks who either were mad at the manufacturer and wanted to speak their mind or who were car nuts and would attend anything that was remotely related to anything with a steering wheel.

Once we got the buyers in the sessions, they gave us some great information.  More importantly, we observed human interactions to various stimuli, what language people used to describe good and bad experiences, and how they persuaded others to come to their point of view.

It sure would have been good had we been able to say that what we observed was applicable to all their buyers, but of course, it wasn’t.   Helpful?  Yes.   Accurate?  Maybe.   Representative?   I have no idea.    But we did learn some useful things.

The first survey research project I ever conducted was a phone survey in 1978! (Yes, I’m old!). I worried about a lot of things on that project — Did I draw a good sample? Did I pick the right data-gathering method? Was my questionnaire solid? Too many open-ended questions? Would the response rate be high enough? And on and on. What I DID NOT think twice about, let alone worry about, was if I would be reaching the survey participants at home, at work, or if they would be driving on a freeway, shopping in a mall or in the gym! There was no chance of reaching anyone via cell phone. There weren’t any!

That was then. And now is now. We will never go back. Every year hundreds of thousands of households give up their land (fixed) line and go mobile.

Are there differences between surveying people on a fixed line vs. a mobile phone? There sure are. And the differences might surprise you.

A recent issue of the International Journal of Market Research reported the results of an interesting comparative treatment field experiment where researchers completed 1,000 interviews on fixed lines with 1,000 interviews on mobile phones. The questionnaire was the same for both groups, allowing direct comparisons. The data they were collecting were not the focus on the analysis. The researchers were interested in the metrics of the interviews themselves – response rate, number of dials required for an interview, percentage of respondents with item omissions (don’t know/no opinion), length of interview, etc.

Some results were predictable. Some were unexpected.

• The mobile phone survey had a lower response rate then the fixed phone survey.
• The mobile phone survey required almost 3 times as many dials (call attempts) as the fixed phone survey.
• Surprisingly, the mobile phone survey respondents took longer to complete the interview than the people who talked on a fixed line. (Were they distracted?)
• And the mobile phone survey respondents were less likely to give “don’t know” or “no opinion” answers than those who took the survey on a fixed line.

There are lots of other findings in this interesting article. Their methodology was solid and the analysis was precise and complete. It’s a great read if you want to see what the future holds for those of us that conduct surveys.

Vicente, Reis and Santos. Using Mobile Phones for Survey Research: A Comparison with Fixed Phones. International Journal of Market Research, Vol. 51, Issue 5, 2009.

Jeff recently was featured in the D.O.M. July/August issue for an article on research. Titled “Simple Marketing – Research that’s sure to increase your market share,” you can read it online at Director of Maintenance’s website www.dommagazine.com.