A regular cadence of assumption testing helps product teams quickly determine which ideas will work and which ones won’t. It’s one of the highest value activities we can do.
And sadly, most product teams don’t do any assumption testing at all. And those that do, don’t do nearly enough. I want to change that.
In this article, I’ll cover assumption testing from beginning to end, including:
- Why should product teams test their assumptions?
- What is an assumption?
- What are the types of assumptions that product teams tend to make?
- Does it matter what category a particular assumption falls into?
- How do you identify the assumptions that your ideas depend upon?
- Does it matter how we phrase the assumptions that we generate?
- What is an assumption test?
- Are assumption tests the same as experiments?
- Do you need to test all of your assumptions?
- How do you identify the riskiest assumptions?
- When should you run assumption tests?
- How do you convince your organization to let you run assumption tests?
- What tools should you use to test your assumptions?
- How do you evaluate the results of your assumption tests?
- What do you do if an assumption test fails?
- What do you do if an assumption test passes?
- How should you make decisions based on your results?
- How should you keep track of your assumption tests?
- What if you don’t have time to run any assumption tests?
- How does assumption testing change based on the maturity of the product?
- How can you learn more about assumption testing?
Why should product teams test their assumptions?
When we consider more than one option, we make better decisions.
We all intuitively know this. It’s why we look at more than one house or apartment when choosing our next place to live. It’s why we talk to more than one company when looking for a job. In our everyday lives, this is intuitive.
At work, it’s harder. When we hear about an unmet customer need, pain point, or desire, we often jump to our first solution. If we do any discovery at all, we tend to ask, “Could this idea work?”
We intuitively know not to only look at one apartment or house or interview at only one company, but in the workplace we’re often tempted to jump to our first solution. This image was generated by DALL-E.
Imagine if you shopped for a new apartment this way. As you walk into a dimly lit fourth-floor walk-up, you gloss over the cracks in the walls and the exposed wiring in the bathroom, and start to rearrange the furniture mentally in your head—your bed will fit in the small master bedroom if you turn it diagonally, you can ditch your desk and use your small card table for both a dining room table and a desk since that’s all that will fit in the tiny kitchen/living room. You imagine how fit you’ll be from carrying your aging dog up the four flights of stairs multiple times a day.
This is absurd. Shouldn’t we look at more apartments? How do we know this is the best we can do? We haven’t compared and contrasted our options.
You would never shop for an apartment this way.
But this is exactly what we do when we try to make our first idea match the customer need we are trying to address, within the technical constraints we encounter, while still managing to deliver the right business results.
Why do we do this? The answer is simple. We don’t think we have time to consider more options. We worry it will slow us down. The engineers need something to work on next week.
But this is not true. We can consider multiple options and still move fast. The key to considering multiple options while still moving quickly is to stop testing whole ideas, and to start testing the underlying assumptions that your ideas depend upon.
The key to considering multiple options while still moving quickly is to stop testing whole ideas, and to start testing the underlying assumptions that your ideas depend upon. – Tweet This
Assumption testing is what allows us to quickly evaluate which ideas might work and throw out the ideas that won’t.
You can read more about the value of compare and contrast decisions in these articles:
- Stop Asking Whether or Not Questions
- How Compare and Contrast Decisions Lead to Better Product Outcomes
What is an assumption?
An assumption is a belief that may or may not be true. For product teams, we are talking about the assumptions that need to be true for your idea to succeed. As a general rule, the more specific your assumption, the easier it will be to test.
To get a feel for what assumptions look like, let’s walk through an example. To set the context, I’ll start with an outcome and a target opportunity, using a mini opportunity solution tree. If you want to learn more about these concepts, start here.
The opportunity solution tree can help you identify assumptions that are connected to each solution you’re considering.
Suppose I work for a local paper and my product team is responsible for bringing in new readers (our outcome). When interviewing readers, we uncovered a common desire: “I know someone who should read this article.”
We’ve chosen this as our target opportunity and have generated three potential solutions:
- Add social media share buttons that allow people to quickly share the title and URL of the article.
- Add the option to email the full text of the article to someone.
- Add the option to text someone the title and URL of the article.
Why three solutions? Revisit the answer to “Why should product teams test their assumptions?”
Each of these solutions depend on a number of assumptions that need to be true in order for each to succeed.
We can start with a broad assumption: Our readers want to share articles with other people.
All three of our ideas depend upon this assumption. This assumption assumes that our target opportunity, “I know someone who should read this article,” expresses a real desire.
But we can also enumerate many more specific assumptions:
- Our readers want to share this article with people on Facebook.
- Our readers will notice the option to share an article.
- We can format the full article text appropriately to be shared via email.
- The person receiving the shared article will be interested in the article.
- If the person receiving the article doesn’t have a subscription, they’ll buy a subscription to get access to the article.
- If the person receiving the article doesn’t have a subscription, they won’t be annoyed with the sharer since they can’t access the article.
- Our readers won’t share articles with people who might be offended by the content.
- Our readers will share articles via SMS with enough non-subscribers (who end up subscribing) to offset the cost of SMS.
- Our emails will get through spam filters and into readers’ inboxes.
- Our readers will know the email address of the person they want to share the article with.
We could enumerate dozens more. But this should give you an idea of what assumptions look like.
What are the types of assumptions that product teams tend to make?
Assumptions fall into five categories: desirability, feasibility, usability, viability, and ethical.
Product teams tend to make assumptions in five different categories:
- Desirability: Why do we think our customers want this solution and why do we think they’ll be willing to do what we need them to do to get value from it?
- Viability: Why do we think this solution will be good for our business?
- Feasibility: Why do we think we can build this solution?
- Usability: Why do we think the customer will be able to use this solution?
- Ethical: Is there any potential harm in building this solution?
You can learn more about the five assumption types (and see examples of each) in this article:
Does it matter what category a particular assumption falls into?
The short answer is no. If you read the more detailed article about assumption types, you’ll see that what I might call a feasibility assumption, someone else might call a viability assumption. That’s okay.
The value of the categories is they help us generate assumptions that represent risk in our ideas. So it’s not fruitful to spend time debating whether or not a specific assumption is a desirability assumption or a usability assumption. The point is to generate assumptions across the categories, increasing the likelihood that you uncover the riskiest assumptions.
How do you identify the assumptions that your ideas depend upon?
Story mapping—walking through each step of a customer or user’s journey as shown here—is an effective way of generating different types of assumptions.
There are many ways to identify the assumptions that your ideas depend upon. Here are some of my favorites:
- Use story mapping to help generate the assumptions that appear in each step of your story map. This is a great way to generate desirability and usability assumptions. But it can also surface feasibility, viability, and ethical assumptions as well.
- Walk the lines of your opportunity solution tree. In other words, why do you think your solution will address the target opportunity in a way that will drive your outcome? Enumerate your thinking. Each inference is an assumption that you can test.
- Define your ideal customer profile and generate the assumptions that it depends upon.
- Do a data audit and examine the ethical assumptions related to your data policies.
- Conduct a pre-mortem to catch any final blindspots.
We cover all of these methods in-depth in our Identifying Hidden Assumptions course.
Does it matter how we phrase the assumptions that we generate?
Yes. How we phrase our assumptions impacts how easy they are to test. Keep these two rules of thumb in mind when phrasing your assumptions:
- Phrase your assumptions such that they need to be true for your idea to succeed. For example, if your app requires a login, then you might generate the following assumption: Users will remember their username and password. Don’t phrase it as: Users will forget their username and password. It’s easier to test if customers will do something than it is to test if they won’t do something.
- Be specific. The more specific your assumption is, the smaller and faster the test will be. Tie each assumption to a specific test in your story map.
We cover how to phrase your assumptions in our Identifying Hidden Assumptions course.
What is an assumption test?
While there are hundreds of assumption tests we could run, most of them tend to fall into four categories.
An assumption test is a structured activity that we do to evaluate the risk in an assumption.
For desirability and usability assumptions, we are typically designing activities that allow us to evaluate customer behavior.
For feasibility assumptions, we are typically conducting engineering activities that allow us to understand how difficult something might be to build.
For viability and ethical assumptions, the activity can vary based on the specifics of the assumption.
While we have hundreds of activities we could do to evaluate our assumptions, most assumption tests tend to fall into the following four categories:
- Prototype tests: activities designed to simulate a moment so that we can evaluate customer behavior.
- One-question surveys: a quick way to evaluate past customer behavior and/or observe current customer behavior.
- Data mining: the use of existing data to evaluate the inherent risk in an assumption.
- Research spike: an engineering activity that allows us to evaluate feasibility risk (e.g. often an engineering prototype).
We cover all four of these assumption test types in our Assumption Testing course.
Are assumption tests the same as experiments?
The short answer is no. When I wrote Continuous Discovery Habits, I decided not to use the term “experiments,” and instead, I replaced it with the term “assumption tests.” Here’s why.
First, we rarely have time to run real experiments in discovery. We can run large-scale randomized controlled experiments, also known as A/B tests, after we build something, but that’s only if we have enough traffic.
And this is not the best discovery activity. We don’t want to do all the work to build something before we learn we built the wrong thing.
Second, most teams fall into the trap of testing a whole idea. Whole idea testing takes a lot of work and a lot of time—two things we want to avoid in discovery.
Most teams fall into the trap of testing a whole idea. Whole idea testing takes a lot of work and a lot of time—two things we want to avoid in discovery. – Tweet This
Assumption testing makes it clear that we’re testing a single assumption and not the whole idea. These tests are faster and take less work.
Assumption tests are used in discovery when trying to decide between ideas. Experiments are used to measure the impact of what we built.
Do you need to test all of your assumptions?
No, as long as you are engaging with your customers on a regular basis, most of your assumptions will not carry much risk. They will be mostly safe.
We only need to test the riskiest assumptions that could torpedo our ideas or cause harm to our customers or company.
How do you identify the riskiest assumptions?
Assumption mapping is an activity David Bland introduces in his book Testing Business Ideas.
In his book Testing Business Ideas, David Bland shares an exercise called assumption mapping. It’s a way to evaluate assumptions based on two factors:
- How important the assumption is to the success of the idea.
- How much evidence you already have for the assumption.
Our riskiest assumptions are the assumptions that are critical to the success of our idea where we have little evidence that suggests that they are safe.
Our goal with assumption testing is to collect more evidence.
We cover assumption mapping in our Identifying Hidden Assumptions course.
When should you run assumption tests?
Assumption testing helps us compare and contrast multiple solutions against each other. So I like to run assumption tests after I’ve chosen a target opportunity and selected three potential ideas to explore.
I also like to use assumption testing after I’ve chosen a final solution candidate if I still have open questions about how the solution should work or if there’s more risk to mitigate.
Assumption testing is a discovery activity that should be used when trying to decide what to build. A team that does continuous discovery is continuously running assumption tests.
Assumption testing is a discovery activity that should be used when trying to decide what to build. A team that does continuous discovery is continuously running assumption tests. – Tweet This
How do you convince your organization to let you run assumption tests?
Most teams can start assumption testing without waiting for permission. For example, if you have access to data sources like user behavioral analytics, support tickets, sales notes, customer interview transcripts, or any other data sources that hint at customer behavior, you might be able to start doing some data mining tests right away.
If you work in an organization where you don’t have any access to customers, running prototype tests might be hard. I’d befriend someone on the account management team, the support team, or the sales team and see if you can run some quick prototype tests with the folks they are already talking to.
Your engineers can usually create engineering prototypes or run research spikes as part of your regular delivery cycle. This could be as simple as creating a user story and adding it to your next sprint. If your sprints are already jam-packed, start with something teeny tiny to start building this muscle.
Don’t overthink it. Assumption testing looks like a lot of the work that we already do. To make steps in the right direction, all you have to do is add a little bit of structure to your existing activities.
What tools should you use to test your assumptions?
There are a number of tools that make assumption testing easier and faster. Here are a few that I like to have in my toolbox.
There are a number of tools that make assumption testing easier and faster. I like to have the following types of tools in my toolbox:
- An unmoderated testing platform: These tools allow us to upload a prototype, go home for the day, and come back to a set of videos of what customers did with the prototype. UserTesting innovated in this space. Maze is another popular platform.
- In-product survey tool: Any tools that allow you to embed a short survey inside your product or service works. Qualaroo innovated in this space and Ethnio was a fast follower. But there are now dozens of tools that do this. I use Typeform for my one-question surveys.
- User behavioral analytics tools: Anything that lets you see what your customers did while using your product or service works. There are dozens of tools in this space. Amplitude, Mixpanel, and Heap are a few of the popular ones.
- Prototyping tools: It’s getting easier and easier to create quick mockups. Balsamiq really innovated on making prototype creation easy. Sketch and Figma and a wide variety of no-code tools are also good options here.
- Data synthesis tools: Product teams are inundated with data inputs from sales notes, call-center transcripts, customer support tickets, NPS surveys, marketing insights, and so much more. Any tool that helps you make sense of all of these sources is a great one to have in your toolbox. This is the area where I’m hoping to see big gains quickly due to the rise of generative AI.
We now have hundreds of tools that can help us run assumption tests easier and faster. Pick the tools that work for your team. If you need some inspiration, check out our Tools of the Trade category.
We now have hundreds of tools that can help us run assumption tests easier and faster. Pick the tools that work for your team. – Tweet This
How do you evaluate the results of your assumption tests?
Product teams often ask me, “How do I know if my results are good enough or not?” The reality is: You don’t.
There are two things you can do to make evaluating your assumption test results easier:
- Use your assumption tests to compare and contrast different solutions against each other. Instead of asking, “Is this result good enough?”, you can now ask, “Which solution looks best?”
- As a team, work to define success upfront, before you run your assumption test. Think through what types of results you would need to see for your solution to succeed and then draw a line in the sand.
We spend an entire week of our Assumption Testing course on how to define success upfront and why it’s important to do so.
What do you do if an assumption test fails?
Get used to it. This is going to happen a lot. And it’s perfectly normal. Many (if not most) of our tests will fall short of our expectations.
The advantage of running assumption tests instead of whole idea tests is that when an assumption test fails, we now know exactly what to do.
When we test a whole idea and it fails, all we know is that it didn’t work, but we don’t always know why. When an assumption test fails, we know exactly what needs to change.
If the assumption is critical to the success of our idea, we might need to scrap the idea. But before you do that, be sure to read the answer to “How should you make decisions based on the results of your tests?”
If you can design around the assumption, do that. In other words, you learned something new in your assumption test, how can you use that information to improve your idea? Oftentimes, we can still make the idea work, we just need to iterate on it.
Most ideas will need iteration. It’s rare that a great idea just emerges from our head. Instead, we need to test and iterate to get to a good idea.
Most ideas will need iteration. It’s rare that a great idea just emerges from our head. Instead, we need to test and iterate to get to a good idea. – Tweet This
What do you do if an assumption test passes?
It depends. As we run assumption tests, we are collecting more evidence about the assumptions that our ideas depend upon.
If an assumption test passes and we think we’ve collected enough evidence for that particular assumption, we might move on to another assumption.
If we think we’ve collected enough evidence to choose a solution, we might do that.
It’s rare that we’ll make decisions on a single assumption test. We need to take into consideration what we are learning from the rest of our assumption tests.
How should you make decisions based on the results of your assumption tests?
I don’t recommend that you make decisions based on the results of a single assumption test. Remember, our goal with assumption testing is to test assumptions across different solutions, so that we can compare and contrast them against each other.
So we want to run multiple assumption tests and then make decisions based on what we are learning across our tests.
Sometimes all of our tests will fail. This either tells us there’s an issue with our target opportunity or we need a better set of solutions.
While it’s theoretically possible that all of our tests can pass, I’ll share that I’ve never seen this in practice. Usually some of our tests pass and some of our tests fail.
It’s easy to hyper-focus on choosing one of your solutions. But I recommend you take a step back and ask a different question, “What did we just learn about our customers from this round of assumption testing?”
When our assumption tests fail, it means that we learn something unexpected. We want to make sure that we take the time to absorb this new learning.
When our assumption tests fail, it means that we learn something unexpected. We want to make sure that we take the time to absorb this new learning. – Tweet This
You might be ready to choose one of your three solutions. Or you might have learned something that can help you generate even better solutions. Keep both in mind.
It usually takes successive rounds of assumption testing to develop a really good idea. Don’t rush the process.
How should you keep track of your assumption tests?
For every assumption test you run, I recommend keeping track of things like the assumption, who you tested it with, and a few other key points.
I like to keep things simple. I use a Trello card to track the tasks I have to do to get the assumption test live and I use Miro to keep track of all of the tests I’m currently running and their results. If I’m working on something evergreen, then I might create a dedicated Miro board just for tests on that challenge, so that I have everything in one place.
Every once in a while, I review all of the related tests on a topic to see if I can generate some broader insights.
I’m not suggesting this is the right solution for you. Some companies create research repositories and keep track of all of their assumption tests for all teams for all time. I can see how this could be helpful.
When tracking your assumption tests, I recommend making note of the following attributes:
- The assumption: Make sure it’s specific and that you understand the context in which it was being tested.
- The audience: Who did you test with?
- The details of the assumption test: the prototype, the one-question survey, the engineering task, the data you looked at, etc.
- The success criteria you set.
- The results.
- How you acted on the results.
For an alternative point of view, check out David Bland’s experiment card in Testing Business Ideas.
What if you don’t have time to run any assumption tests?
We often think we don’t have time for assumption tests when actually we do.
This is a tough question because we often think we don’t have time when we actually do. We think we don’t have time because while we think assumption testing is important, it’s not urgent.
The challenge here is that if we knew how often our assumption tests failed (in other words, how often what we expect to happen doesn’t), assumption testing would start to feel urgent.
It’s a little bit of a catch-22.
So my recommendation is to start small. Do the smallest assumption test that you can complete in the next hour. I’m being serious.
Take whatever solution you are working on right now. Identify one thing that you think might have risk and figure out something you can do in the next hour to evaluate that risk. That might be: look at some user behavioral analytics, talk to a sales rep, review user search queries, or read through recent support tickets.
Don’t worry about whether it’s the best assumption test. Just look for something you can do right now.
Do it again tomorrow. And again the next day.
You’ll get better and faster at it. And suddenly, you’ll have plenty of time to run assumption tests.
Oftentimes, this question is more about our resistance to something new than it is about not having time. We all have 24 hours in a day. To overcome this obstacle, find the smallest, easiest thing you can do and get started right now.
How does assumption testing change based on the maturity of the product?
It doesn’t. Whether we are pre-product or working on a decades-old mature product, we still have new ideas that can be broken down into their underlying assumptions.
In fact, as products age, we can even take existing solutions and break them down into their underlying assumptions and see if any of those have changed.
Assumption testing is a way to test our beliefs. It’s a core element of critical thinking and as a result is broadly applicable.
One thing to keep in mind is if you are pre-product and you have no customers, you might have to get creative about who you test with. This article on how continuous discovery works in early-stage startups can help with that.
How can you learn more about assumption testing?
We currently have two Product Talk Academy courses—Identifying Hidden Assumptions and Assumption Testing—that are designed to give you hands-on practice with many of the ideas discussed here.
Our Identifying Hidden Assumptions and Assumption Testing courses are designed to get you hands-on practice with many of the ideas in this article.
In Identifying Hidden Assumptions we cover how to:
- Create story maps for each of your potential solutions.
- Use those story maps to generate desirability, usability, and feasibility assumptions.
- Generate viability assumptions by walking the lines of your opportunity solution tree and how to generate ethical assumptions by evaluating your ideal customer profile and by doing a data audit.
- Ruthlessly prioritize what assumptions to test first.
In our Assumption Testing course, we cover how to:
- Design assumption tests using the four test types: prototype tests, one-question surveys, data mining, and research spikes.
- Define success upfront and why it’s critical to do so.
- Improve the reliability and validity of your assumption tests so that you can trust the evidence you collect.
- Make decisions based on sets of assumption tests so that you always know what to do next.
You should join us! We offer both courses several times a year.
The post Assumption Testing: Everything You Need to Know to Get Started appeared first on Product Talk.
Assumption Testing: Everything You Need to Know to Get Started was first posted on October 18, 2023 at 6:00 am.
©2022 “Product Talk“. Use of this feed is for personal non-commercial use only. If you are not reading this article in your feed reader, then the site is guilty of copyright infringement. Please let us know at support@producttalk.org.
