How many surveys do we need for the data to be statistically significant? This is one of the most common questions for B2B product marketing teams when scoping out a quantitative survey project.
The short answer is: probably fewer than you assume in most use cases.
While large sample sizes are ideal, the practical reality is that most B2B survey projects make do with modest sample sizes. The reasons include one or more of these scenarios for any given survey.
Response rates in online customer surveys continue to decline. Product marketing teams often have to work with a limited data collection budget. And, feasibility often puts a cap on the number of surveys collected.
When the question about statistical confidence arises, it is usually not about the specifics of p-values and confidence intervals. The team is really asking: Can we be confident that this modest data set reflects my audience or market? In statistical terms, is the data reliable?
In most B2B surveys, smaller base sizes provide enough reliability to inform business decisions. Here’s why.
Lawrence Chapman
Relative homogeneous audiences
In consumer studies, a large sample size is important because of the variation in the audience. For example, US national political polls include 20-year-old college students from Portland, Oregon, and 75-year-old retirees from Florida. These two groups are quite different in their outlook, priorities, etc.
Most B2B markets have significantly less variation across individuals or entities—they are more homogeneous. For example, the business needs and priorities of the VP of Operations at two different manufacturing firms are more similar than those of two randomly selected consumers.
In addition, the questions asked in B2B surveys tend to focus on discrete topics that are a bit more rational in nature. For example:
- To what extent is tracking inventory a challenge?
- How often does your production line stop because of inventory shortages?
- What causes the shortage?
That said, homogeneity diminishes or disappears if you serve diverse market segments. For example, a bank’s cybersecurity needs differ from those of a manufacturing firm. If this is what your customer base looks like, the concept of reliable small data sets applies to each segment individually.
You can have a small data set for banks and a small one for manufacturing, but not a small data set that includes both segments.
The bell curve
On a practical level, in homogenous B2B data sets, the normal bell curve distribution tends to form at around 30-40 completed surveys. After that, the specific numbers may change, but not the overall trends.
For example, at 35 completed surveys, 37% of respondents selected “Yes,” and when you reach 100 completed surveys, the percentage of respondents that selected ” Yes ” shifts to 41%.
Most marketing professionals have taken a statistics course at some point in their lives and know that it’s possible that the numbers could be very different if you have a much higher base size. It’s also possible that you win the lottery.
However, that would be the exception. We don’t see the numbers change much in the real world as the sample size increases.
As a research firm, we’ve conducted hundreds of B2B surveys. We start tracking data early in the process; what we see at 25 surveys is similar to what we see at 50 and 100 surveys.
We also find that when we conduct a study that includes an upfront set of qualitative followed by a quantitative survey, the trends are broadly consistent across both phases of the research.
The trends we describe above only apply if you have a representative sample. No number of surveys will make a biased sample statistically reliable. For example, a survey of 200 of your best customers has good statistical reliability but is not representative of prospects.
Charley Gale
No single level of statistical significance
For shorthand researchers, especially in opinion polls and content marketing, make note of the statistical confidence of the data at the broadest level.
For example, “Our report is based on 400 surveys and has a +/-5% confidence interval.” That interval of +/-5% is the widest range of potential variability in the data.
Every individual question in your survey has a unique statistical reliability. As you move to the ends of the continuum of answers, your statistical reliability increases.
For example, assume your data set includes 100 surveys. If, on Question 1, 50% of respondents said “Yes,” and 50% said “No,” the confidence interval for that data point is +/-10%. If on Question 2, 90% said, “Yes,” and 10% said, “No,” the confidence interval is +/-5.4%, almost twice as strong as Question 1 with the same sample set.
A simple analogy helps conceptually illustrate what is going on. Imagine a new restaurant has opened in your town, and 20 of your friends have tried it. When you ask all 20 friends about it, 18 say it is good, and 2 say it is bad.
Based on that data, you likely feel confident that the restaurant is good and that the two friends who did not like it were outliers.
However, if 10 friends liked the restaurant and 10 disliked it (a 50/50 split), you would have more questions about whether you would like the restaurant or not. This is why political polls in close races often get it wrong. This analogy also illustrates the idea of a homogeneous audience – your 20 friends are likely to be more similar to you and each other than 20 strangers.
Artificial precision trap
It is important to remember that statistical precision is not some sort of Holy Grail. Even the best-designed survey with a large sample size still has fuzziness in the data that goes beyond the statistical calculations. Statistical confidence means that if you took the measurement again 95 times out of 100, it would be within a range of +/- X%.
Statistics and confidence intervals assume a set of controlled conditions where everything is exactly the same from measurement to measurement. This works fine when measuring quality in a manufacturing plant.
However, the conditions under which you conduct a survey vary from study to study. Your target audience may be slightly different; you may use a different list source; one study may use telephone interviews and another uses an online survey. And remember that statistical tests are based on mathematical equations alone, they don’t account for context.
A survey of 400 highly biased respondents still has a statistical confidence interval of +/-5% based on the math. But that data may be less representative and reflective of your market than a well-designed and well-recruited set of 15 qualitative in-depth interviews.
Joe Radwich
Larger sample sizes sometimes matter
We’ve focused on how and why you can use modest sample sizes to inform your decision. However, there are times when a larger sample size is more important.
The following are some examples:
- If the difference between 37% and 41% would make a difference in your decision.
- If you plan to use the results to address an internal disagreement between key stakeholders.
- Brand awareness or campaign effectiveness tracking.
- If you have a heavily engineering-oriented senior management team.
Making the decision
But as we said at the start, for many research objectives, a modest sample size is enough to support decisions. For example:
- When B2B firms are in the early stages of evaluating a market opportunity, they need to understand its scope of magnitude. Would one-tenth of the market be interested, or half the market?
- When developing a product roadmap, the question is about what features to develop first and which can be pushed further out.
- Product marketing professionals and product managers need to understand customers’ pain points and challenges when developing GTM strategies and sales and marketing materials.
Quantitative research cannot give you the absolute answer – it can only narrow the cone of uncertainty around your decisions. You need to combine the data from your survey with your other data, knowledge, and experience when deciding what the results mean for your business decisions.
The question on the table is: Have I taken the steps to ensure that the data is representative of my market? If so, you can feel confident using modest sample sizes to inform your decisions.
Get Market Research Certified
Boost your product marketing skills with our comprehensive market research certification.
You’ll delve deep into the what, why, when, and how of market research, learning how to apply practical examples in your setting.
What’s more, you’ll learn the importance of ethics in data collection – a topic often overlooked. We know there aren’t enough resources that cover market research from a product marketing perspective.
With this course, you can call off the search.
