AI Evals: A Hands-On Guide for Product Teams

AI evals have been the “it” skill for product teams for over a year. I’ve even called evals a new discovery habit.

But I still meet product teams who only have a vague idea of what evals are. And it’s not their fault. Most of the writing on this topic is intended for engineers or just isn’t specific enough.

My goal today is to explain what evals are and why product teams can and should create them. I’m going to make this practical, hands-on, and easy to follow.

AI evals (short for evaluations) are methods for measuring whether an AI product or workflow is performing well. Evals give teams confidence that their AI applications are doing what they expect them to do. They help teams maintain quality and catch issues before they reach users.

Similar to other discovery habits like interviewing and assumption testing, evals can act as a feedback loop to ensure we are on the right track.

Why Product Teams Need to Learn to Build Evals

If you are using AI to help you write PRDs, synthesize what you learned from customer feedback or customer interviews, analyze behavioral analytics, make sense of meeting notes, or really anything else related to doing your day-to-day job, you can benefit from evals.

If you want to know if your AI-generated PRD includes everything you asked for, evals can help you measure this. If you want to know if any of the customer quotes the AI used are fabricated, evals can tell you this. If you want to make sure the data analysis started with the right questions and didn’t deviate as it ran complex calculations, evals can monitor this.

When I used ChatGPT to help me write synopses of all of the Lovable interviews that I conducted for this blog post, I wondered how I could trust that the AI got each participant’s story right. I built confidence in the summaries by building in some evals—a fact-checker and a hallucination guard. I’ll walk you through how both worked in a bit.

If you are building customer-facing AI products or services and have asked yourself, “How do I know if this is consistently good across all of our customers and all target use cases?” you can benefit from evals.

I ran into this with my very first AI product. I ran several interview transcripts through my nascent Interview Coach and the results looked pretty good. But I started to wonder, “How do I know if this is good enough to roll out to all of my students?”

The only way to know if our AI products and workflows are any good is with evals.

Evals Help Us Define and Measure What Good Looks Like

With traditional software, we write a requirements document, the engineers implement it, and we use automated testing to make sure it passes our acceptance criteria. When it does, we know it works the way we intended.

But when an LLM is involved, it can be harder to know if the software works. There are two reasons for this. First, given the same input, the LLM might give a different answer. And second, we tend to give LLMs semantic tasks that might have more than one right answer—or even better or worse answers.

The former makes it hard to use our traditional testing methods. With traditional software, what the software does is pre-determined. We can look at the code and know exactly what we’ll get every time. With LLMs, it’s not that simple. LLMs are probabilistic. That means their output can vary, even given the same input.

Code Will Always Give the Same Answer, LLMs Will Not

Here’s a simple way to think about it: If I write a function that takes two numbers and returns the sum, I can write unit tests for a range of cases. If the PRD requires that it works for integers (whole numbers), floats (numbers with decimals), negative numbers, and a mix of integers and floats, then I can write unit tests that span all of these cases.

• To test if the function handles integers well, I can write a test that calls the function with 2 and 3, and checks to make sure that it correctly returns 5.
• To test if the function handles floats, I can send it 3.2 and 4.1, and check to make sure it returns 7.3.
• To test negative numbers, I can send it -1 and -3, and check to see if it returns -4.
• To test a mix of integers and floats, I can send it 2 and 3.1, and check to make sure it returns 5.1.

I can write these tests once and run them every time I change the code. If all the tests pass, then I can be confident that my code works as expected.

But if instead I create an LLM service to do the same thing, this strategy doesn’t work. The first time I send it 2 and 3, it might return 5. But the second time it might return 4. Testing it once isn’t adequate. It might work once and then break the next time.

Evals allow us to measure the rate at which the LLM returns a correct response. This is different from a unit test. With a unit test, we expect it to always get it right. With an eval, we are measuring how often it gets it right.

The Unique Challenge of Semantic Output

Before we can measure how often the LLM got the right answer, we need to define what a right answer looks like.

When we are talking about simple addition, defining the right answer is easy. 2 + 3 = 5 is correct. 2 + 3 = 4 is not correct.

But we tend to turn to LLMs for semantic tasks—that’s their strength. And oftentimes, semantic tasks are harder to judge. It’s rare that we can categorically say this is right and this is wrong. Instead, we might say this is better and this is worse. There’s a lot of gray.

If you ask an LLM to write a joke, how would you determine if the response was right or wrong? Your primary criteria might be, “Did it make you laugh?” But you also might ask, “Is it clever?” Or, “Is it appropriate for my kids?”

Before you could evaluate the response, you have to first define what a right answer looks like. And more often than not, a right answer doesn’t always have clear boundaries. One joke might be funnier than another joke. Another might be more clever. And so on.

You’ll see with evals, one of our jobs will be to get clarity on what a right answer looks like, even when our intuition says it’s blurry.

When I had ChatGPT help me summarize real stories from my Lovable interviews, I defined correctness as follows:

• The story follows the participant’s narrative.
• All key moments are included.
• No quotes are made up or inaccurate.

For my Interview Coach, I also defined correctness on multiple dimensions:

• Did the agent sort each interview question into the right section for feedback?
• Did it avoid suggesting a leading question or a general question?
• Was each coaching tip appropriate given what the interviewer asked?
• Did it score each attribute correctly?
• Was the agent’s tone appropriate?

For each, I had to take the time to define what good looked like and then I had to figure out a way to measure that. We’ll walk through how to do exactly that in a minute.

Don’t Outsource What Good Looks Like

But first, I want to highlight that we typically can’t outsource this work. Some of the big eval tool providers offer built-in evals to help you get started. For example, they might evaluate your agent on conciseness or helpfulness. They’ve baked in their own definitions of correct for each of these.

But it’s pretty easy to see how one definition doesn’t work across all environments. If I’m a student trying to learn about astrophysics, I might need verbose, detailed responses designed to teach. But if I’m an astrophysics professor asking a similar question, I don’t need all the teaching detail.

Correctness is context dependent. Don’t let a vendor define correctness for your product. This is the product team’s job.

So how do we define a right or wrong answer?

Step 1: Look at What the LLM is Doing

When we work with an LLM in a chat interface, we are constantly evaluating the model. We ask for something, we get a response, and then we revise our ask based on how well the response met our need.

But when we are building a workflow or product, the goal is for the LLM to do the right thing when we aren’t watching. To get here, we have to look at a range of inputs and then evaluate how the LLM responds to those inputs. This process is often referred to as error analysis—where we manually evaluate what types of mistakes the LLM makes.

How much error analysis we do can vary based on how important it is for the LLM to get the output right every time. For personal workflows where a bad response only impacts us, we might do some lightweight error analysis. But for production products where a bad response affects a real customer, we should do more error analysis.

Error Analysis for Personal Workflows

When I was writing my Lovable blog post, I had 17 transcripts that I wanted to convert into vignettes. I could have written each vignette one at a time. Instead, I wrote a prompt instructing ChatGPT how to write each one. My prompt included my desired structure for each story, the tone and writing style, and the desired length.

When the prompt was ready, I gave ChatGPT the first transcript and asked it to write the story. But I didn’t just trust the output. I took the time to read the full transcript. I conducted the interview, but I wanted to remind myself what was said. I then identified what key moments I would have included in the story and checked them against what ChatGPT included. I checked each quote to make sure it came from the transcript verbatim. This took time.

But it also allowed me to see what types of mistakes ChatGPT made. I noticed two distinct errors:

1. It would get some of the specific details wrong—it would hallucinate the person’s job title or make up a detail about the product that wasn’t included in the transcript.
2. It would hallucinate quotes. It would take two different statements that the participant said and merge them into a new meaning that was not in the transcript.

This was enough for me to identify that I needed two evals: a fact-checker and a hallucination guard.

Was this process slow? Yes. But did it save me time in the long run? Yes. I could have manually reviewed all 17 stories myself. Instead, I reviewed one in-depth, identified the types of errors the LLM made, and then used those errors to identify which evals I needed.

For the next 16 transcripts, I then ran my evals to catch the LLM’s mistakes. The evals allowed me to trust the output. We’ll look at how I defined these evals in the next section.

Now, if I was turning this into a production product, I would probably want to look at more than one transcript. But for my personal workflow, this was plenty.

Error Analysis for Customer-Facing Products

For my Interview Coach, I originally tested it with my own customer interview transcripts. The Coach did a good job. But my interview transcripts were in exactly the format and style that the Coach expected.

To really test the Coach, I needed to test it on student interviews. I decided to release my Interview Coach in beta to my April 2025 cohort, even though I wasn’t sure it was ready. I needed traces to evaluate.

A trace is a detailed record of an AI interaction. It includes the user input, system prompts, tool calls, intermediate steps, and final LLM responses. In multi-turn conversations, it typically includes all of the back and forth between the user and the LLM.

For the Interview Coach, a trace is the system prompts, the interview transcript, and the Interview Coach’s feedback on that transcript.

To mitigate the risk of putting something in front of students before it was ready, I told my students that the Coach wasn’t perfect, but that I would be reviewing all of its responses. If it made a mistake, I would email them and let them know.

Reviewing each submission was how I did my error analysis. Every time the Interview Coach made a mistake, I would log it. I then categorized these errors and these became my candidates for evals.

Because this was a production product, I looked at hundreds of cases. I wanted to get a rich understanding of what mistakes occurred over a broad set of customer inputs. I still do these reviews today, even though my Interview Coach has been live for over a year.

As you do your error analysis, it can be tempting to try to fix everything all at once. But I recommend splitting this into phases. Annotate first, categorize errors, and then decide what’s worth addressing. Otherwise, you run the risk of spending time fixing less important errors before you’ve even uncovered the more glaring ones.

And if you are working on a customer-facing product, use your discovery habits to figure out which errors matter most to your customers. Those are the ones that need evals.

Now It’s Your Turn

Take a moment to put what you learned into practice. Pick a personal workflow that you use in your day-to-day job. How might you judge correctness? What errors do you see in the responses?

As you use the workflow, don’t just gloss over LLM mistakes. Instead, make note of them. Try to identify the most common categories. Some will be more important than others. Consider what types of mistakes you can live with and which ones you wish would go away.

For the ones that you wish would go away, these are great candidates for evals.

Step 2: Identify the Best Way to Count How Often the Error Occurs

Now that we’ve identified the errors that we care most about, it’s time to define some evals. Remember, an eval is a metric. It helps us count how often a specific error occurs in our LLM output.

There are four common types of evals.

1. Golden Dataset – A set of known inputs used to test the LLM
2. Code Assertion – A programmatic rule that helps us evaluate quality
3. LLM-as-a-Judge – A second model judges the output
4. Customer Feedback – The customer tells us if the response was good enough

Golden Dataset Evals Are a Common Starting Point (But Are Limited)

When teams are new to evals, they typically start with golden dataset evals. To create a dataset eval, you simply define a set of inputs that you expect to represent what you might see in production and you define the ideal outputs. The ideal outputs are your definition of correctness. When you hear product managers talking about evals as working in spreadsheets, they are typically referring to golden dataset evals.

They can be a great way to help you develop a rubric for what good looks like. Error analysis helps us see what “bad” looks like by identifying errors. If we fix those errors and add them to our golden dataset, we can start to curate what “good” looks like.

But they only work well for tasks where you can define one correct output. If there are many correct outputs, then it’s hard to define all variations of correctness. Similarly, if the inputs and outputs are quite large (like interview transcripts and opportunity solution trees), then it’s simply not feasible to create golden dataset evals.

As a general rule of thumb, golden dataset evals are good when the inputs and outputs are small and there is one correct answer. They are good for classification tasks (e.g. is this a business outcome or a product outcome?), factual answers (e.g. who was the first US President?), routing tasks (e.g. which skill best applies?), and so on. They tend not to work well for measuring complex output or when working with complex inputs.

But you can often break complex tasks into smaller tasks and then use golden dataset evals to measure performance on those smaller tasks.

For example, in my AI-generated opportunity solution tree service, the agent has to decide if similar, but differently worded opportunities (from different source opportunities) are actually the same general opportunity. I can create a golden dataset eval to test this ability by defining sets of interview opportunities as inputs and defining the ideal output as a judgment on whether they are similar enough to generalize or not.

There are two challenges to keep in mind when creating golden dataset evals: 1. It’s hard to know what your production inputs will look like before you launch, and 2. It’s incredibly tedious. It’s not a one-time activity. You have to continuously improve your golden dataset to make sure that it continues to match what you expect to see.

I use golden dataset evals for targeted steps where the input and output are simple and I am confident I can curate a golden dataset that represents what I’ll see in production.

Code Assertions Are Typically Cheap and Fast

A code assertion (also called a code-based assertion) is a type of AI eval where you use traditional deterministic code to evaluate the quality of an LLM response. Like unit tests, they are virtually free and the code works as intended every time.

The key to a good code assertion eval is you have to find a fixed structural approach to measuring the error in a way that doesn’t require judgment.

For example, my hallucination guard eval for my Lovable stories blog post was a code assertion. For every quote that appeared in a ChatGPT story, I searched for the exact string in the original transcript. If there was a match, the code returned true, if there wasn’t, it returned false.

With my Interview Coach, I came up with a list of red flag words that indicated the presence of a general question. This eval just did a string search for any of those words and returned true if it found one and false if it didn’t.

For my AI-generated opportunity tree service, I have a code assertion eval that counts the nodes on the tree and evaluates how well they are distributed. I then set designated thresholds to define different error categories.

For every new error where I need an eval, I always try to find a way to measure it with deterministic code. It’s fast and cheap. Only when an eval truly requires judgment do I turn to an LLM-as-a-Judge eval.

LLM-as-a-Judge Evals Measure Semantic Qualities (But Can Be Expensive)

An LLM-as-a-Judge eval is where you use a second LLM to evaluate the output of your first LLM. Instead of relying solely on human judgment or code-based checks, you send your LLM’s output to another LLM along with evaluation criteria, and ask it to score or judge the quality based on specific dimensions.

When I first heard about LLM-as-Judges, I immediately visualized turtles standing on turtles to infinity. If I can’t trust the first LLM, how can I trust the second LLM? It turns out there are ways to make this work.

To make an LLM-as-a-Judge work well, we have to:

• Give the judge a much simpler task than what we gave the original model.
• Define our criteria such that our judge can return a binary response: true/false, yes/no.
• Align our judges with our own judgment.
• Understand that our judge will always make mistakes and account for that error when reporting our eval error rate.

We won’t cover all of these steps in today’s article. But I’ll do a deeper dive on LLM-as-a-Judge evals soon.

For my Lovable stories blog post, I used an LLM-as-a-Judge to fact-check each claim in the ChatGPT-generated stories. The judge received one fact and a transcript and was told to respond with true if the fact was grounded in the transcript, false otherwise.

My Interview Coach has several LLM-as-a-Judge evals:

• My double-barreled-questions eval gets a question the interviewer asked and has to judge whether the question is asking for more than one thing at a time.
• My leading-question eval also gets a question, but it gets different judging criteria. It evaluates if the question assumes something about the participant or indicates a preferred response.
• My already-answered eval gets a question the Coach suggested and the transcript and has to evaluate if that question has already been answered in the transcript.

LLM-as-Judges can be incredibly helpful when we need to evaluate semantic quality. But they are expensive and it takes work to get them right.

Customer Feedback Evals Are The Ultimate Judge (But Can Be Tough to Make Actionable)

My favorite way to evaluate an AI product is to let the customer rate the response. For customer-facing products, this is the ultimate eval.

However, it can be difficult to get the customer to give us more than a thumbs up/thumbs down or a simple rating. So we often have to infer what went wrong. But ratings aren’t our only option.

We can also use other customer behaviors as a proxy for feedback. For example, if a customer uses AI to generate an image and then they re-generate it, that’s feedback that the first image didn’t work for them. If they made prompt changes before they regenerated the image, those changes might carry details about what went wrong.

I am experimenting with these types of evals in my own AI products. If you ask TeresaBot a question and then you follow up with another related question, that’s feedback that TeresaBot didn’t fully answer your question the first time.

With Vistaly, we are planning to use edits to our AI-generated interview snapshots and opportunity solution trees as feedback on what the LLM got wrong.

This category is endless and I’m excited to see more innovation in this category as more teams start designing evals.

How To Choose Which Type of Eval You Need

I’ve had a lot of success by starting with the simplest and dumbest measurement and then have been pleasantly surprised when it just worked.

I try to use code assertions whenever I can. That’s simply because they are fast and cheap. I use golden datasets to isolate and evaluate smaller tasks in a complex process. I use LLM-as-Judges only when identifying the error requires judgment. They are expensive (even when using small models). And I’m always looking to gather as much insight from real customers as I can.

You might even mix and match methods. For example, I recently used a code assertion and an LLM-as-a-Judge to measure the same error two different ways. I was trying to understand how often my opportunity solution tree agent was missing sub-groupings. I used a code assertion to count how many children were under each parent. And I used an LLM-as-a-Judge to evaluate if there were any semantic groups that were missing. The code assertion was cheap and fast; I could run it on every node. If a node had too many children, I could then run it against my LLM-as-a-Judge. This combination allowed me to keep my LLM-as-a-Judge costs low because it didn’t need to look at every node.

The error	Eval type	Reason
A quote in a story wasn’t actually said	Code assertion	A quote either appears in the transcript verbatim or it doesn’t. No judgment required.
A claim in a story wasn’t grounded in the transcript	LLM-as-a-Judge	The model can invent a claim using entirely different words, so there’s no string to search for.
The Coach suggested a general question	Code assertion	A short list of red flag words (“typically,” “usually”) turned out to catch these reliably.
The Coach suggested a leading question	LLM-as-a-Judge	Requires judging whether a question presumes an answer—there’s no fixed wording to look for.
Two differently worded opportunities are actually the same opportunity	Golden Dataset eval	Small inputs, small output, one right answer. Easy to label a set and reuse it.
A tree branch is missing a sub-grouping	Code assertion and LLM-as-a-Judge	Counting children is cheap enough to run on every node. Only the nodes that fail get sent to the judge.

Now It’s Your Turn

Start with your error and ask, “How can I best count how often this happens?” Let the nature of the error help you choose the right eval strategy.

Play with more than one way to count the same error. Get creative. Can you use a code assertion to filter before using an LLM-as-a-Judge? Is there a structure or shape that can be identified with code?

If judgment is required, what’s the simplest evaluation task you can give the judge? You might need to use more than one judge rather than giving a single judge a complex task.

Step 3: Use Your Evals to Improve Your Product or Workflow

Once you’ve identified ways to count each of your errors, you can now experiment to improve your product.

Design Inputs So You Can Collect a Baseline

Before you make any changes, be sure to get a baseline by running your evals against a set of pre-determined inputs.

If you are using golden dataset evals, the dataset already defines both the inputs and the outputs. If you are using code assertions or LLM-as-Judges, you’ll need to define just the inputs. Think of these inputs like a test. Every time you make a change to a prompt, to an orchestration pattern, a model change, or really anything else, you will run these inputs through your product or workflow and then you use your evals to score how it did.

Your baseline is the score for how your product or workflow works today. You’ll compare your experiment variants against your baseline. If the experiment variant gets a better score, you can be confident it’s an improvement.

If you are designing inputs for a personal workflow, you might only identify a small set of inputs that represent your typical usage. But if you are working on a production product, you’ll want to do your best to make sure they represent the types of inputs you might expect from real customers. This sounds simple, but it can be quite challenging to do in practice. I continue to evolve my input sets as I learn what real customers do.

Run Your Inputs Through Your Product or Workflow

To get a baseline, you’ll need a way to run each input through your product or workflow.

For personal workflows with just a few inputs, you might simply run these by hand. Be sure to log the LLM responses in a spreadsheet. You’ll need them for your evals.

Or you can ask your favorite model to write a Python script that takes each input, runs it through your workflow, and then saves the output to a text file.

For customer-facing products, I strongly recommend automating this step. You’ll be running your inputs through your product every time you make a change (e.g. model changes, prompt changes, orchestration changes). I’ll share more about how I built a simple test harness to automate this entire process in the next section.

Run Your Evals On the Outputs

Now that you’ve collected a set of outputs, you’ll want to run all of your evals on those outputs. This will tell you your baseline rate for each error category.

If you are primarily using golden dataset evals, then you are simply comparing the LLM’s actual output to the ideal output that you defined in the dataset and you are scoring how often the LLM got it right.

For code assertions and LLM-as-Judges, you’ll need a way to run the output through the code or through another LLM call. For LLM-as-Judges, you can sometimes run them as sub-agents in Codex, Claude Code, or Cowork.

For example, for my Lovable stories blog post, I asked Claude in Claude Code to use sub-agents with the LLM-as-a-Judge prompt to fact-check each claim. I also asked Claude to write a Python script that extracted quotes from the ChatGPT story and searched for them in the transcript.

I got scores for each transcript that looked like this:

• 15 out of 17 facts grounded in the transcript
• 3 out of 4 quotes verified verbatim in the transcript

The output told me exactly which facts and quotes didn’t appear in the transcript and I was able to fix them by hand.

For customer-facing products, I built a simple test harness to automate this whole process. It’s a Python script that:

• calls my LLM service for each input and logs the output
• takes the LLM response and transforms it into the input each eval expects
• runs each eval against the response
• scores the full run (e.g. how often did the LLM service get the right answer?)

Need help setting up a testing harness? Supporting Members and CDH Members can grab a copy of my testing harness here.

The output is a score for each error category. For my Interview Coach, the output looks like this:

• Leading Question: 15 out of 105 suggested questions
• General Question: 3 out of 105 suggested questions
• Already-Answered: 9 out of 105 suggested questions

Once you’ve collected your baseline, it’s time to run your first experiment.

Design and Run an Experiment Variant

Now that you have an accurate measure of how often different errors show up in your product’s or workflow’s output, it’s time to try to improve those error rates.

Start with one error. How might you reduce it? Here are some of the things we can do to get better performance out of the model:

• Change the prompt.
• Add context.
• Change the model.
• Break a complex LLM call into a smaller series of simpler LLM calls.
• Make a more advanced orchestration change like adding agentic tool calling or audit loops.

An experiment variant can include more than one change, but I like to group my changes based on a theory or hypothesis, rather than just trying a bunch of different things.

Important: Before you change anything, come up with a system for how you’ll document the changes. Not all experiments will work and you’ll want to roll back your changes when an experiment fails.

For personal workflow experiments, I document my experiments using my process-notes skill. You can learn about that skill here.

For my customer-facing products, I document each experiment variant in a yaml file (a plain text config file) that includes an experiment ID, a description of what changed, a commit sha (the unique identifier that git assigns to a commit) to capture what the code and my prompts looked like for the experiment, and what evals to run to evaluate the experiment.

Each of my evals is designed as a module that my test harness can just run. The module’s run function transforms the LLM’s output to the necessary eval inputs and then runs the code assertion or calls the LLM-as-a-Judge.

My test harness doesn’t have to understand how each module works. It just needs to know to call the module’s run function.

For example, my general-questions eval for my Interview Coach needs a list of questions that were suggested by the Coach. The module for that eval includes code that takes the raw LLM output and parses out all of the questions that the Coach suggested and then runs each question through my code assertion code.

Once you’ve made and documented the variant changes, it’s time to run the experiment:

• Run your inputs through the new variant.
• Run your evals on the outputs.
• Compare the results.

When comparing the results, be sure to look at changes across all of your evals. A prompt change, for example, might improve the error rate in one category, but make another one worse. Our goal is to make our product or workflow better across the board.

And if your first experiment doesn’t succeed, don’t be discouraged. It often takes multiple experiments to find improvements. In my next article, I’ll share a story about how it took 16 variants to fix a single customer complaint.

Now It’s Your Turn

Let’s put this section into practice:

• Start by defining your inputs. If you are working on a personal workflow, this can be as simple as your most common use cases. If this is for a customer-facing product, try to be comprehensive. Collect them in a spreadsheet (if running by hand) or a markdown file (if you are going to automate it).
• Decide how you will generate outputs. For personal workflows, I like to use Claude Code sub-agents. I simply give Claude my prompt and the markdown file where I’ve defined my inputs and ask it to call one sub-agent per input. I make sure my prompt instructs the sub-agent to write a results file. For my customer-facing products, I use my test harness.
• Decide how you will run your evals. Again, for personal workflows, I like to use Claude Code sub-agents combined with Python scripts for the bulk of the work. And for my customer-facing products, I’ve built all of this into my test harness.
• Collect your baseline. Run your inputs through your current workflow or product. Run your evals against the outputs.
• Design and run your first experiment. Run your inputs through the variant and run your evals against the new outputs.
• Compare the results. Don’t be discouraged if they fall short. You can always run another experiment.
• Repeat until you get the desired outcome. Sounds a lot like continuous discovery, doesn’t it?

Why Evals Are a New Discovery Habit

In Continuous Discovery Habits, I outlined several habits that help us make better decisions about what to build. I included habits like defining clear outcomes, customer interviewing, story mapping, and assumption testing. Each of these habits gives us a clear feedback loop on the decisions we are making as we decide what to build.

When we define clear outcomes and measure the impact of our releases against those outcomes, we get feedback on whether we are building the right thing. When we interview customers and learn about their unmet needs, pain points, and desires, we get feedback on whether we are solving the right customer problems. When we story map our solutions and test the assumptions our ideas depend upon, we get feedback on whether our solutions will satisfy those customer needs.

When we do error analysis of customer traces, design evals, and run experiments to reduce errors, we get feedback on whether our changes worked. If our error analysis is informed by what we learn from the rest of the discovery habits, we can build confidence that our AI products will work better for our customers.

I hope this guide helps you get started with this new discovery habit. As you work through it, if you encounter any questions or issues, please share them in the comments. If I get enough questions, I’ll host a dedicated ask-me-anything session for members.

Audio Version

The audio version is only available for paid subscribers.