In its simplest form, A/B testing is a way to compare two versions of something to figure out which performs better. The principles and math behind A/B testing are around 100 years old.
Nowadays, A/B testing is most often associated with marketing and ecommerce. For example, testing different versions of websites or apps in an online, real-time environment.
Specifically, it involves creating two variations (A and B) that differ in one aspect. This could be a difference in design, layout, content or call-to-action. These variations are shown to similar groups of people, randomly selected from your target audience.
A/B testing is important for small businesses in several ways. It can help:
This helps boost conversions, such as more purchases. Making decisions based on data and real behaviour helps remove the guesswork or assumptions.
Collecting this information can help tailor your offerings to better meet customer needs or expectations.
Sometimes businesses lack the resources or budgets and are looking for ways to find cost efficiencies. A/B testing helps by trying different variations without making significant investments.
A/B testing begins with knowing what you want to test. For example, this could be a call to action, such as the subscribe button on your website with two different colours. Importantly, only one variable should be tested at a time to get clear results.
Also consider how you’ll measure performance, for example conversion rates. For a subscribe button test, this is likely the number of people who click on the button.
To run the test, two sets of users, assigned randomly when they visit your website, are shown the different versions. Success is determined by which colour receives the most clicks.
Many factors can influence which colour people click on. For example, personal preference. This is where randomisation is crucial. It helps minimise the chances these factors affect your results.
A/B testing is often considered the most basic form of a randomised controlled experiment. As with all experiments of this kind, consider the sample size you need to gain statistical significance. This will help ensure the result isn’t due to other influences.
Some variables can impact the success metric. For example, it could be that mobile users spend less time on websites, compared with desktop users.
In performing an experiment, the randomisation process in one set (A) may contain slightly more mobile users than the other (B). As a result, that set may see a lower click-through rate regardless of the button size they’re seeing.
To help level the playing field, divide users by mobile and desktop and then randomly assign them to each version.
A/B testing can be applied in a variety of different ways to help better inform your digital marketing strategy and content creation. Below are some examples of where you can use A/B testing:
If you want to run an A/B test, below is a step-by-step guide to help:
Software can help you analyse A/B testing results. However, it’s helpful to have a basic understanding of the output to help you make decisions.
When interpreting survey results, it's helpful to consider margin of error and confidence levels. For example, your A/B testing may report 21% of people prefer a red call to action button. That result may have a margin of error of +/- 2% at 95% confidence.
But what does that mean? Margin of error is the measure of variability in your results based on your sample data. In this case our true result may sit between 19% and 23% of people who prefer a red call to action button.
A larger sample size generally results in a smaller margin of error. This is because it reduces sampling variability. Similarly, a smaller sample size may result in a larger margin of error.
Confidence levels indicate the level of certainty the results fall within the estimated range. Common confidence levels are 95% or 99%. This means that if you repeated your A/B testing many times, you would expect the result to fall within the margin of error 95 or 99 times out of 100.
A/B testing is a quick and easy way to understand customer preferences and to help implement changes to your marketing initiatives. Below we list some tips to help enable a smooth process.
A/B testing is a simple, yet powerful tool to help you make the most of your digital presence. It helps turn assumptions into insights and uncertainties into opportunities. By comparing variations and analysing customer behaviour, you can make decisions based on data. This will help lead to improved conversion rates, better customer experience and greater success in meeting your business goals.
Whether you need a website or a complete digital marketing strategy, we can help your business to thrive online.
Discover how you can use tech to help evolve your digital marketing strategies and meet your customers’ changing expectations.
