What Does the Future Hold for A/B Testing? 4 Experts Weigh in
Jack Maden | December 10, 2015
Jack Maden | December 10, 2015
The third and final instalment in our series on A/B testing asks four optimization experts on where they see the industry going. Part one looked at overall testing strategy, while part two asked our experts the most effective way to come up with testing hypotheses. We conclude the series by staring into our crystal balls… (or, to complement the above photo, by performing a tarot card reading).
Here’s the question we asked our experts.
What future trends do you see developing in A/B testing with the emergence of new technologies? And do you think there will be a drift towards MVT?
And here’s what they said!
“Two words: machine learning. Intelligent algorithms will automate a lot of manual work, start coming up with personalization rules on their own, do predictive analytics and all that jazz. Humans will still be needed though - mainly for qualitative research.
“A/B vs multi-variate testing has nothing to do with new technologies, that only depends on your traffic as well as what you're testing.”
“With CRO emerging as a hot-topic amongst marketers, there are so many new tools being developed that enable you to look at data in different ways.
"Just like with testing, I think we will start to see trends in the way companies are using data to drive insights. With the mass amounts of data that can be collected we are still looking for an optimized way to summarize and convert that data into results. The industry is making strides to automate this process so I am excited to see what happens!
“That being said, it is important to separate the methodology from the technology. While technologies can be ever-changing, multi-variate testing is one of the methods that can be used to answer questions. With any technology, there are multiple ways to approach and answer a question so it is important to pick the one that best fits the scenario.
"One fact of testing is that MVT requires more traffic than A/B testing. New technology may be able to help slice the data in different ways, but it won't drive more qualified traffic into the experiment. Multivariate tests are not necessarily 'better' or 'preferred' over A/B tests and they each have strengths and weaknesses. An improvement in technology won't change that so each company will still have to analyze its situation and use the methodology that best suits its needs, regardless of the tools it uses.”
“We’re working for a high traffic site as The Next Web has millions of unique users on a monthly basis so we would like to have the process of testing as flawless as possible.
"We see most testing tools currently focusing on front-end testing, so I hope we’re going to see more innovation in the area of back-end testing to decrease flickering but also make automating certain processes easier.
“I don’t see it happening that there will be a massive drift towards MVT, it’s something that is only useful for sites with more traffic anyway. For other sites it will usually take too long to get results and it would be faster for them to improve their conversion rates by using A/B testing as the way to go.”
“I think there will be more quality data and results out of tests because more businesses are beginning to understand how testing really works.
"I also see a future in testing through tag management platforms like Google Tag Manager.”
More automation and accuracy is the future our experts would like to see materialize for A/B testing. Moreover, despite the fact that, as Michael says, new technologies will enable data to be broken up, structured and put together in different ways, our experts generally agree that there will not be a drift towards multi-variate testing.
Indeed, no matter how much automation and machine learning is introduced to testing tools, this doesn’t change the fact that multi-variate tests simply require a lot more traffic than A/B tests – and in some cases lend unnecessary complexity to the testing process.
What do you think the future holds for A/B testing? Let us know on Twitter or in the comments below!