DIY Survey Research: Tips for the Savvy, Tricks for Dodging Disaster

As a business, you want your decisions to be based on facts, not your gut—and it’s never been easier to find out what your customers want, what they think of the competition, or how a new product might be perceived in the market. SurveyMonkey, Surveygizmo, and Google Surveys are just a few of the services out there, and these tools are getting more sophisticated every year.

As a professional market researcher, I think DIY market research is making an important contribution to the growth of market research! More and more people are discovering the power of market research and have access to the tools they need to explore business opportunities and make critical decisions. But I also see how companies get into major trouble when they try to use DIY research platforms without a good sense of their abilities and limitations.

If you’re going to do your own market research, following a few guidelines are really important for getting good answers to your questions. Plus, knowing how DIY research can go wrong can help you decide when you’re better off calling in a professional.

Why DIY? Good Reasons for Going It Alone

There are some great benefits for using DIY market research platforms. For example:

• Cost. This is the single biggest reason why companies do their own research—do it yourself, save some cash!
• Easy to use. Most of these tools offer fairly easy-to-use interfaces. Plus, their customer support tends to be pretty good in most cases. A lot of platforms are also providing templates for certain types of surveys, so you don’t have to start from scratch.
• Speed. Get it going quick and get results back fast!
• More research, more often. Because the tools are at your disposal, you can conduct research whenever you like, helping you make decisions faster.

DIY market research platforms have gone a long way, and have helped people to make smart business decisions. However, great tools don’t always guarantee great results for those that aren’t trained in market research fundamentals.

DIY Watch Outs: How Research Can Crash Fast

While there are some excellent benefits of these tools, there are some important pitfalls that I think people should be aware of before jumping into DIY research. Generally, people make mistakes within a few major areas:

Study setup: The most basic problem with DIY research is operator error—in other words, someone doesn’t know what they’re doing. If a study is set up poorly, you could get misleading results that can lead to bad business decisions. It becomes a case of garbage in, garbage out. Even worse, most don’t even know that the results are misleading!  For example, here are a few key questions to think about—if you don’t have a confident answer, then you might have a big problem on your hands.

• Do you know the right way to test and compare new product ideas, package designs, messages, etc.?
• Do you know how to get valid pricing feedback on a new feature or product?
• Would you know when a choice model would be more efficient than a multi-cell monadic test for your initiative?
• Do you know how many people you need to survey?
• Do you know what demographics to target?
• Do you have the right number of respondents? Too narrow and you could be generating a false positive for that new product concept. Too broad and you could prevent yourself from being able to read important subgroups.

Survey writing: Surveys seem so simple, right?  Ask questions, get answers. However, the way you ask a question, the scales you use, and the order of the questions can greatly impact your results. Every question you ask, every message, product, or package you show can create bias for subsequent questions.

Data Quality

• Do you know how to recruit a representative sample for your audience? If you just throw the survey out there for anyone to take, your data won’t be representative—this could give you misleading results. You need to control for response rate differences for different demographics, or set quotas. Most of the platforms I’ve seen don’t have much flexibility in either of these.
• Do you know what data needs to be cut? In our full service projects at Peel Research Partners, it’s not uncommon for us to remove 10% of respondents or more from our initial data sets, typically cutting out survey cheaters or those that are not engaged.
• Does your platform have a good system for getting rid of bad data and replacing it? Most DIY platforms that I’ve used don’t, and they don’t seem to want to bring up this issue either. Maybe it’s because they don’t want to be on the hook for replacing them for free, which is an industry standard. From the tools I’ve used, they let you filter out one person at a time, but don’t have a systematic way to clean out many people at once. You can download the data and do it outside of the platform, but then you won’t have access to any of the platform’s reporting tools, and any coworkers accessing that study will see the results without the data cleaned.

Limited Programming Capabilities
The DIY tools are always improving but still greatly lag behind professional survey software platforms, limiting the types of surveys that can be created. Here are a few common examples:

• Having respondents evaluate a subset of brands (such as 3 out of the 6 total in the study) among only those brands the respondent is aware of
• Testing a set of new designs sequentially where you’d like to create a random looping block of questions and feed in the design images
• Sample balancing – such as to balance the type of respondents across different cells of a study so they match

Limited tabbing and stat testing: DIY tools are now allowing you to cross tab the data and some even will create stat testing. However, I’ve found the tabbing capabilities are not exactly user friendly. For example, if you want to create NETs such as to show a “top 2 box” score or do a grid banner, most tools will let you create a net, but you’d have to manually do it for every question. Major time waster! A few other pitfalls I’ve seen with tabbing and stat testing:

• Grid banners, the ability to create a table that shows a different stimulus in each column such as different products, brands, or messages, are completely impossible to do in most platforms. Instead, you have to pull the data out and load it into a separate application to analyze the data, a manual and limiting process for most people. You could pay someone else to tab the data, but then you’re losing out on the low cost benefits of DIY research.
• If your DIY platform does not have stat testing, or if you have to download your data to clean it, trying to stat test by hand across multiple subgroups of data can be a long and painful process. If this is your situation, this tool that I created can save you tons of time.

Closed system, higher costs: If you do not have your own list and need to send your survey to a random sample of consumers, most DIY platforms force you to use their consumer panel. This is probably a big money making tool for them, as panel renting can be highly profitable, and it’s probably why all the major consumer panels all have their own DIY tools now. On the surface, this is not a big deal, but the problem is that DIY platforms with the most features seem to have shady panels, and you’re stuck with them. In these platforms, I’ve found more survey speeders and cheaters than normal, and the feedback from the open ends, which is often a sign of respondent engagement, is weaker.

On top of that, panel costs are often twice as high as what it normally would be from a high-quality consumer panel! Once you’ve signed the yearly contract and programmed your survey, you’re stuck—and they often jack up the panel costs. Most people using them have no idea what the panel costs should be, so they don’t know they’re paying too much.

You’re on your own: You have to do it all yourself – no team of experts to manage the process. You have to write the survey, program it, test it, and analyze the data. If you have problems, I’ve found customer service from these companies to be pretty good, and that’s promising. However, while they understand their platforms, they are not market researchers, so don’t expect them to give you any consultative research advice.

Granted, my list of “cons” is a bit longer than my list of “pros” regarding DIY tools…but it’s so important to know that these platforms are not foolproof. Especially when so much depends on your research, you need to know what DIY research can (and can’t) do.

Treading Carefully: Tips for Doing DIY Research Better

• Know when to use it. If the survey you need to run is relatively simple, then it could be a good candidate for a DIY platform. If your initiative is more complex and requires sophisticated survey programming, you need to reach a complicated target audience, or need advanced data tabulations, you probably need to use a professional research vendor.
• Use a quality panel. To ensure I’m getting a good quality panel and at reasonable prices, I personally like to use the DIY platforms of reputable panel companies. The only issue here is that they have not caught up to the veteran DIY platforms in terms of survey logic and features. You may have some surveys that require the veteran platforms. If you’re not sure, you can ask them directly, but be aware that they all tend to exaggerate what they can do. You may need to talk with a market research consultant.
• Clean your data. In terms of how to do it on your own, I’ll have to leave that as a topic for another blog post. But you can talk to your DIY platform company and ask if they have a process in place where you can replace bad respondents.
• Document your results. Even if you’re not writing a report with the DIY study, I highly recommend documenting every DIY study that you conduct and creating a shared repository for them. Even a short summary would suffice. A lot of companies that I’ve seen use DIY research don’t do this, and the research is forgotten. Insights get lost, and they often end up asking the same questions over and over.
• Get help if you have questions. There are market research consultants out there that have experience with DIY platforms. They can help in any or all the steps of the process from study design, writing the survey, programming, and analyzing the results.

Whether you’re a seasoned DIY researcher or ready to take the plunge for the first time, hopefully these tips help you improve the quality of your research and avoid any potential disasters.