Amplitude A/B Testing: How it Works [+ Better Alternative]

Amplitude A/B testing is a very handy solution it offers for improved product management. You can easily create and deploy feature variants to determine which works better at scale.

So, what’s A/B testing like on Amplitude, and how does it work? Let’s find the answers to these questions (and more).

TL;DR

• A/B testing is a process by which you test two versions of a feature or content to determine the better one.
• You must first define your goals and segment the audience for A/B testing on Amplitude.
• Then, design the variations for your split testing experiment through feature flags or payloads.
• Next, configure the delivery of your experiment on Amplitude and also determine the statistical significance. You can then launch and set the end date for your experiments.
• Finally, you can monitor the metrics of your experiment using the analytics feature of Amplitude.
• Userpilot is a better option for A/B testing, especially for those without technical skills, as it offers a visual and no-code builder, unlike Amplitude. Book a demo now to see if it’s the right fit for you.

Try Userpilot and Take Your Product Management to the Next Level

Get a Demo

• 14 Day Trial
• No Credit Card Required

What is A/B testing?

Also called split testing, this method involves creating two versions of a particular feature or content to determine which one works better. Typically, only one parameter is changed between the two versions, and the rest are kept constant to get valuable insights into how the audience responds to each. The idea behind A/B testing is to get the best possible version out to boost your conversion rate, feature adoption, and more.

An important note here—A/B testing is a continuous process and must be a part of your product experimentation framework. You must keep at it regularly to ensure that you find how well a new version performs over the previous one.

How does Amplitude A/B testing work?

Now that you know what A/B testing involves, let’s try to understand how it works on Amplitude, along with other relevant features like user segmentation and control group selection.

Define goals and target audience

Before you can begin creating your A/B test, you need to first select an audience for it. This step is critical as it’ll be meaningful only if you use it on the right set of cohorts.

Amplitude offers numerous options for creating user segments in just a few clicks. You can:

• Select your segment based on a user property like demographics.
• Choose the segment based on in-app user behavior.
• Segment users through random allocation.

It’s also possible to use the funnel analysis feature on Amplitude to understand your audience better and create cohorts based on the insights you gather from it.

Design your experiment variations

Once you’ve got your audience sorted, the next step involves designing the variants for your A/B testing experiment on Amplitude. But before you get started, it’s important to know that if you choose Amplitude as your A/B testing tool, you’ll need technical knowledge.

You’d have to use:

• Strings – They are used for simple text changes in elements like buttons and headlines. You can use them to create variants with little coding.
• Payloads – A JSON object that’s used for complex variations that could involve conditional logic or data manipulation. These objects can dynamically change your variant without needing you to write new code, which would be the case with strings.

That said, Amplitude will create the first variant by itself and call it “treatment.”

Alternatively, you could opt for feature flags on Amplitude. You can choose to enable or disable certain flags for each cohort to deliver different versions of a feature or page to them.

A feature flag is also quite handy as it helps you create gradual rollouts of new variations to your audience. That way, you can test them on a smaller group before launching them for all users.

But if you wish to use non-uniform allocation ratios during A/B testing, you won’t be able to do it through feature flags. Instead, you’d have to opt for strings and payloads.

Configure experiment delivery and statistical significance

Amplitude has certain default settings for presenting the statistics. These include:

• Confidence level.
• Bonferroni correction.
• CUPED (controlled experiment using pre-existing data).
• Custom exposure.
• Test type.

You can launch your A/B tests with the default settings right away. But Amplitude also gives you the option to modify these statistical preferences to suit your experiment better.

In terms of experiment delivery, Amplitude offers two broad options:

• Random allocation.
• Targeted allocation with user or device IDs for finer delivery.

You can even choose the control and social experiment percentages if you don’t want to evenly distribute the experiment.

Launch and monitor your experiment

Amplitude gives you complete control over your A/B testing experiments. You can select their duration right away or keep it dynamic based on the statistical significance.

When you set the end date before launching the experiment, Amplitude will also show you the number of users per variant of your A/B test.

In terms of statistical significance, Amplitude gives the following options for determining when the A/B test would end:

• Rollout the winning version.
• Rollback everything and return to where you started before A/B testing.
• Continue the experiments.

You can choose to roll out the winning version to all users or even a custom set of cohorts.

Analyze A/B test data using Amplitude Analytics

Once your test is complete, you need to dive into the data collected by Amplitude to understand its significance and make future product decisions. You can look at various metrics from your A/B test. These include:

• The absolute value in conversions.
• Absolute performance (in percentage points).
• Confidence interval (a narrow interval means you can be more confident about the impact of your variants).
• Relative performance (between control and treatment variants).
• Significance (the likelihood that the difference between control and treatment variants isn’t due to an error).

When should you use Amplitude A/B testing?

You should consider using this platform for running your A/B tests in the following cases:

• Mobile apps.
• Landing pages.
• Web apps.

If your platform is a different one, you’re perhaps better off selecting an alternative. Similarly, Amplitude A/B testing relies greatly on coding with JSON and other scripts involved. So, if you lack technical knowledge, you’ll likely not benefit from its full range of features. In this case, it makes sense to use another platform for A/B tests.

Pros and cons of Amplitude A/B testing

Let’s now look at how Amplitude’s A/B testing features stack up in terms of their advantages and disadvantages.

Pros of Amplitude A/B testing

There are several pros of using Amplitude for your A/B testing needs. These include:

• Easy setup – If you’re already an existing user of Amplitude, setting up A/B tests is quick, and you won’t need to invest a significant amount of effort into it.
• Real-time tracking – Once you start your A/B test, Amplitude will start reporting the related metrics in real time. It can help you make the right set of data-driven decisions without waiting for the test to end.
• Runs across platforms – You’re not restricted to a handful of platforms when you choose Amplitude. Whether it’s web or mobile apps, you can easily run A/B tests on a whole lot of platforms.

Cons of Amplitude A/B testing

While Amplitude does offer a bunch of useful features in A/B testing, there are certain areas where it lags.

• Coding knowledge – To make the most out of A/B testing on Amplitude, you must have technical skills. This makes it a significant barrier to usage.
• Amplitude ecosystem – If you aren’t already using the Amplitude ecosystem for analytics and other purposes, you won’t benefit from choosing it for A/B testing as your data will be divided across platforms.
• Not a standalone tool – Amplitude Experiment comes as a part of the full suite of Amplitude products. You can’t use it as a standalone tool. This can make it challenging for small businesses to adopt it as you need a larger budget to purchase the entire suite.

Pricing of Amplitude A/B testing

Amplitude offers three pricing plans, including a free one. While the Free plan is free forever, the other paid plan is the Plus plan, that starts at $49/month, billed annually. However, the A/B testing features aren’t available in these plans.

For Amplitude Experiment, you have to opt for the Growth plan that has custom pricing. You’d have to contact their sales team for a quote based on your needs.

Userpilot – A better alternative to Amplitude A/B testing

While Amplitude Analytics is a fantastic platform for A/B testing, it has certain drawbacks that make it challenging to use it to the fullest extent. That’s where Userpilot steps in. It offers several features, like segmenting based on user property, much like Amplitude.

But for many reasons, it’s the better option among the two, especially for those without technical skills. Here’s why:

• You can set up the platform with relative ease and launch your tests.
• The platform is purpose-built for in-app A/B testing.
• It has a visual editor, making it easy to create A/B tests.
• It’s a no-code tool, which means you don’t need any coding skills to get started.

Additionally, you can conduct three major types of A/B tests on Userpilot: controlled, head-to-head, and controlled multivariate. This gives you greater flexibility and even enables you to test multiple variables simultaneously.

Userpilot also provides detailed analytics into your A/B test by showing the winning variant and the percentage by which it won. It also has pricing transparency as the A/B testing feature is available in the growth plan, which costs $749/month.

Conclusion

A/B testing plays an important role in boosting your conversion rate throughout the customer journey and developing an app that users love to use. While Amplitude A/B testing is a great option for those with technical skills, it may not be suitable for those who prefer no-code options.

Userpilot makes for a better replacement and offers a range of A/B testing options. Book your demo now to see it in action.

Try Userpilot and Take Your Conversion Rates to the Next Level

Get a Demo

• 14 Day Trial
• No Credit Card Required