# A/B testing (My CXL experience)

In the lesson, l learnt about the evolution of A/B testing.

Down to when VWO and optimizely came into the market in 2010, They created a drag and drop solution for A/B testing.

So every marketer could just log on,drag and drop something,press start and run the experiment.

Now we’re embedding optimization frameworks on the server level at companies now.

A/B testing is very importanat. We can use A/B-testing for several reasons.

Deployment

And the first one companies should use it for deployments.When you deploy something on your website,could be because of legal reasons,could be a new feature, could be an update, whatever.You want to deploy this as an experiments.

I could even start with shifting only 5% of the traffic and then move on to 50% all the way to 95%.Could be a split like a proper A/B-experiment.Because you want to learn if your deployment does not have a negative impacts on the KPI show measuring.If it has no negative impact.So a wing,is even perfect,but a flatline,that’s fine to.Just deploy it.

Research

And research can be split into more parts.The first thing you can use it for is what we call a **conversion signal map**.If you have a specific webpage,let’s say you have this product page with a picture,and some lines, and a button and so on.All the elements on there.You can run experiments by just leaving out elements.So here you’re not looking for winners.You’re looking if leaving out some elements has some impact, or no impact.If you can see positive signals or negative signals,or flatlines, doesn’t really matter.

If you leave it out and then there’s a really big negative impact,then it’s a really important element.So you can pick that one to optimize.And if it’s leaving out means you have an uplift you really optimized the website.

And deployment could even be a little bit different because of the fundamental system.So you can still run this as an experiment.So in the end, this is real deployments.This is like lean deployment.

It’s all optimization.It’s all A/B-testing.And this is where you can use A/B-testing for research,lean deployment,and real deployment,looking for no negative signals.

One of thr important questions to ask is “Do you have enough Data to conduct an A/B test?

This class objectives,introduce the ROAR model,explain statistical power and of course learn how to calculate if you have enough data to conduct A/B tests.

Jeff Bezos from Amazon.He says “Our success at Amazon is a function”of how many experiments we do per year,”per month, per week, per day.

ROAR Model

This is the ROAR model.It’s a model that was created four or five years ago as a rule of thumb model to explain when you can run how many experiments.It has four phases, the optimization phases,and it has this graph line.The graph line grows to the phases.And on the vertical axis you have conversions per month,and on the horizontal axis we we have time span.And if you’re starting a new company,you will start with zero conversions per month,and hopefully it will grow, it will grow and grow faster and grow faster and grow faster.

So risk, optimization, automation, re-think.This is why it’s called the ROAR model.And you see the borders over here,There’s a border between risk and optimization.And the rule of thumb border to 1,000 conversions per month.If you are below 1,000 conversions per month,you cannot run A/B tests.

And there’s a second border at 10,000 conversions per month.And bear with me this is the rule of thumb.It depends on a lot.We will go through power calculations in this class so you can calculate this stuff yourself.But rule of thumb 1,000, 10,000.Now if you’re going up from 1,000 to 10,000,you can do A/B testing.You can do more and more and more.the closer you are to 10,000 conversions per month.

if you have 10,000 conversions per month,you can run four or start four A/B tests per week,so have 200 A/B tests per year and then you’ll need a full team.

**Why 1,000, 10,000?**

Well, if you have 1,000 conversions,your challenger needs to beat the control,in reality with 15 percent.So this means that if your A, your control,has 100 conversions, which is quite low.100 conversions,your challenger needs to make 115 conversions.A 15 percent uplift to be able to be recognized as a winner in A/B tests, now that’s a lot.Sometimes it happens that your running such a big uplift,but it only happens once a year, twice a year not like every weekend even if you have 10,000 conversions per month you still need to make an impact of 5 percent to be able to find a winning A/B test outcome.So, that’s a lot, this is why you cannot run A/B tests if you’re below 1000 conversions per month.

https://abtestguide.com/bayesian/

https://abtestguide.com/abtestsize/

but that’s a lot let’s say I want to be able to detect 5 percent uplift, even with 100,000 visitors,a two percent conversion rate,this experiment needs to run for five weeksto be able to detect a 5% uplift.And if you only want to run it for four weeksyou need to get an improvement of six percent.

Which KPI should i pick?

So I have 1,000 visitors, 10% conversion rates,so I found those people with one in conversion,and 100 people with zero,no conversion.So this is a plot you can make, an assumption.This is a normal distribution.You convert or you don’t convert.So that means if you go for, let’s say, average order value,I want to increase average order value,the typical way to do this is,let’s say, your average order value is $100,let’s optimize for the number of people with at leasta $100 order value.So then you have peoplethat have a transaction of below $100and people with transaction above $100.And now you can shift on more transactionswith a high value or maybe profit marginis even a better way, something to target on.This is how you can translateaverage order value comparison to a 0/1 comparison.

So 0/1 comparison, binary comparison,

**Research to get insights for A/B testing**

I’m going to show you the 6V modelwhich you can use for conducting proper research and it’s all about the value of your company,it’s all about verses, your competition,it’s view of the data, it’s voice of the customer,it’s verified data and validated data.Really important lessons on research.

Understanding customer behaviour,

Without an analyzed data set, you shouldnt be able to run any A/B test experiment.

- Insights into the most important customer journey
- Understand the basic user behaviour
- Input for setting hypothesis

So you want to know what’s happening on this website,or mobile app, or whatever you’re trying to optimize.Then you want to understand the basic user behavior.What do they do, what’s their motivation?Are they able to perform certain tasks?Understand this basic user behavior.

It leads to inputs for setting hypothesis.So that’s the end of the customer behavior study,you’ll end with a couple hypotheses based on research.

Visual ping:

Track changes on the competitors website. Other tools are Wachete has good ratings, ChangeTower, Pagescreen.

Conversions.com extension shows if optimizely is running on a website.