Essential Survey Data Quality Checks for Reliable Survey Insights

Why is Data Quality important in Survey Research?

A survey is only as good as its respondents’ answers. Respondents can become disengaged, lose interest, and give poor answers simply to get through a survey. That’s why it’s a smart practice to build survey data quality checks into your survey.

Quality checks are measures written or programmed into a survey that trigger a flag for further review of a respondent’s inputs. A quality check can be as simple as a flag that indicates someone has completed the survey too quickly, or more complex, like a programmed variable that can notify you if a respondent has provided inconsistent responses. These quality checks can be useful to ensure that the respondent is remaining engaged, taking the appropriate amount of time to complete each question, and providing consistent responses.

A best practice is to follow the “baseball rule” when analyzing respondents who have tripped quality check flags. If a respondent fails any three quality checks, they should automatically be removed from the data set. A respondent who fails only one or two of the checks should be subjected to additional review.

The number of survey data quality checks to use will depend on the length of your survey. A survey with a median length of interview (LOI) of about 10 minutes should have only around three quality checks. If your survey is closer to a 20-minute LOI, then we recommend five to seven survey data quality checks to confirm your respondents are still engaged as they get deeper into the survey.

While quality checks are useful in protecting the quality of your survey data, they should not be the only line of defense. A tight screening section will likely lead to fewer quality-check flags. A sample provider who has a relationship with their panelists can also follow up with the respondent to get clarity on flagged responses.

What Are the Main Types of Survey Data Quality Checks?

Quality checks are simple to incorporate into your survey and assist in safeguarding your data quality to give you higher confidence in your final data set. Once you integrate these best practices into your questionnaire, you can sit back and let the survey do all the heavy lifting.

• Knowledge Checks: These help ensure each respondent is familiar with and educated on a topic. Content knowledge checks can include asking a respondent to pick correct definitions, define various acronyms, or provide other information. These should be included in a screening  section.
• Attention Checks: This one is self-explanatory; it is designed to make sure the respondent is paying attention, and in the case of large-scale consumer surveys, catch bots. The question will have one obvious correct answer.
• Red Herrings: These are like content knowledge checks but are used to confirm a respondent’s familiarity with a subject or industry. You can test for this by inserting a red herring option in your question (e.g., adding a fake vendor to a list of real companies). Be careful not to make your fake options sound close to an actual one.

 

How to ensure data quality once bad-quality data is already there?

While preventative measures can do the heavy lifting, you will still likely need to do some fine-tuning. These best practices will highlight potential issues and turn what could otherwise be a several-hour manual process into an efficient, brief exercise in data cleaning.

Conflicting-Answer Flag: You may want to check important data for accuracy in your survey. Checking for conflicting answers can mean asking the same question twice or asking similar questions looking for conflicting answers. However, sometimes respondents don’t remember the exact answers they’ve already provided, so it may be helpful to reference their previous answer in a later question.

Straight-Lining Flag: This flag is triggered when a respondent selects a majority of options in a multiselect question or continually selects an individual answer on a grid.  Straight-lining can be a result of fatigue; therefore, it is heavily encouraged to design the survey to minimize the opportunity for this to occur. It’s important to note that this requires a judgment call. It may be entirely reasonable for someone to select down a row and be providing perfectly reasonable responses. However, in the example below, it would be highly unlikely that a respondent would feel that the highest level of satisfaction applies to all factors.

 

Open-End Validation: This check looks for duplicate answers, frequent misspellings, irrelevant answers, and gibberish entered in open-ended fields. While these checks don’t evaluate the quality of the responses, they can alert you to respondents who are not mindfully answering open-ended questions. Are they providing coherent and cogent responses? If not, you can remove them from the data set. If it’s less clear — perhaps the individual just misunderstood the question — then you can review the rest of that respondent’s data and make a more informed decision about how to handle it.

Speeder Flag: A respondent who completes a survey too quickly can mean that the person is not reading and thoughtfully answering the questions. Speeder checks can help by flagging any users who finish the survey within a chosen time frame. This is also a judgment call; although, the industry standard for speeder checks would flag anything less than one-third the median length of a survey.

Building flags and checks like these into a survey helps ensure that the time and money you spend conducting a survey produces useful, actionable results. Combined with well-written screener questions and an expert panel that matches your needs, a quantitative survey will produce powerful results.

---

GLG Surveys builds quality checks into every survey and our Quality Review Team assesses respondents and responses to identify potentially unqualified answers that could skew your data. What would you like to learn from your panel? Get in touch to start surveying for insight now.

Check out the other articles in our Survey Series:

• • When Statistical Significance Just Isn’t…Significant
  • Top 8 Tenets of Survey Design
  • What Type of Survey Do You Need?
  • Are You Running the Right Survey for the Wrong Reason?
  • Why the Screener Section of Your Survey Is Compromising Your Results
  • Surveying Basics: The Right Way to Reach Respondents
  • To Rate or to Rank? That Is the (Survey Design) Question