Pardon Me, Your Bias Is Showing: How to Avoid Bias in Survey Design

We are all susceptible to bias. Our preferences, upbringings, and experiences all leave us predisposed to thinking and feeling a particular way. The good news is that there are proven ways to either avoid bias in your surveys or, at the very least, minimize its impact on your survey results.

---

Our biases are ingrained in us. Mostly, we aren’t aware and we don’t pay much attention to them. That’s a dangerous combination because biased data usually doesn’t appear compromised. After all, a survey respondent is answering honestly and earnestly. However, bias could be lurking underneath the surface, invalidating your data in sneaky ways that will lead you to incorrect conclusions.

Avoiding bias in your survey is easier said than done, but it can be accomplished. The best course of action is knowing the various types of bias you can inadvertently build into your survey and how to address them. In some cases, it means being aware it exists and considering that context when analyzing the data. In other cases, you can avoid it altogether, but to do that, you need to know what you’re looking for.

So what are all these pesky forms of bias?

Self-Selection Bias

What is it? The people who choose to participate in your research study could be fundamentally different from those who don’t.

You may be susceptible to self-selection bias before anyone has answered a single question. If you choose to compensate people with Home Depot gift cards, you will be incentivizing a different kind of person than if you choose to compensate with Candy Crush gold bars.

Even something as standard as hosting your survey online presents the opportunity for self-selection bias. Your respondent pool will skew toward more tech-savvy respondents. This doesn’t mean you shouldn’t host your survey online; it just means you should consider how your results might be impacted by the type of person most likely to respond, given how the survey was promoted, how they are compensated, and the method by which the survey was administered.

Confirmation Bias

What is it? Asking questions in a way that confirms your own hypothesis.

Use neutral wording — this should be your mantra. Neutral wording and a neutral viewpoint are essential to unbiased question design.

Anchoring

What is it? Anchoring provides specific pieces of information, usually fact-based, that set up the question. This might not seem like such a bad thing, but it can undermine the quality of your survey results. The problem is that people will answer questions based on given information, which can lead to biased results.

In this example, you will force the respondent to draw from their own experience to answer the question.

Leading Questions

What is it? Wording questions in a way that directs respondents to a “correct” answer.

The first question implies a yes/no answer and presupposes that growth will likely occur. The second question allows the respondent to answer based on their knowledge of market dynamics.

Motivational Bias

What is it? Respondents may be biased in certain ways depending on their level of investment in the topic being asked about in the survey.

In this case, it doesn’t matter how you ask the question — what matters is this respondent’s personal involvement with the implementation. If they were on the implementation team, they are likely too close to the project to give an unbiased view. It doesn’t mean the response isn’t valid, but instead that the context should be considered when analyzing the response.

Availability Bias

What is it? People tend to focus on their most recent experiences or the latest events.

Instead of providing the recency for them, allow them to answer two separate and distinct questions. Again, the key is to get the respondent to answer based on their own knowledge and experience.

Social Desirability Bias

What is it: Even when anonymized, people will answer questions in accordance with social and cultural norms.

There are many ways to deal with social desirability bias, but the easiest is to remove it from the personal point of view.

Question Order Bias

What is it? The order of questions may lead respondents to answer differently. An earlier question can influence how they might answer a later question.  

If you were to ask the second question first, respondents would be primed to think negatively about the product. They may actually like the product, but they’ve just thought about all the things they don’t like about it. When asked about their overall view after that, they’ll probably rate the product lower than they truly perceive it.

The Takeaway

This isn’t a comprehensive list of all the types of biases that can creep into surveys. However, these are the most common and easiest to guard against. Armed with the information on how to address each of these, you can ensure your data will be vastly more representative of survey respondents’ actual views, knowledge, and experience.

Bias sits at the intersection of survey design and human psychology. It’s one of the most interesting aspects of market research. You shouldn’t take my word on that, though. I’m pretty biased, after all.

---

Will Mellor leads a team of accomplished project managers who serve financial service firms across North America. His team manages end-to-end survey delivery from first draft to final deliverable. Will is an expert on GLG’s internal membership and consumer populations, as well as survey design and research. Before coming to GLG, he was the Vice President of an economic consulting group, where he was responsible for designing economic impact models for clients in both the public and the private sectors. Will has bachelor’s degrees in international business and finance and a master’s degree in applied economics.