In the world of design optimization methods, A/B testing attracts the most attention. Multivariate testing is a rarer and less understood alternative, often considered too time-consuming to bother with. It has its limitations, but they are offset by advantages that A/B tests alone can’t easily give you. Below are the advantages and disadvantages of both methods, and the cases in which to use each.
What are A/B tests?
When you run an A/B test, you create two different versions of the website (or of one page) and split the traffic equally between them. You can also run an A/B/C test with three versions, an A/B/C/D test with four, and so on, often called A/B/n testing. You can change any variable from one version to the next, and it is good practice to test clearly different pages. When the versions live at different addresses, it is a split URL (redirect) test.
Large product companies decide this way. Netflix describes an A/B test as a simple controlled experiment: members are randomly split, one group keeps the current experience and the other gets the change. Because the assignment is random, everything else stays equal between the groups, and in Netflix’s words, “A/B tests let us make causal statements.” A before-and-after comparison can’t, because a hit title may change behavior at the same time.
What are multivariate tests?
In a multivariate test (MVT) you don’t test another version of the website, as you would in an A/B test. You run a much more subtle test of elements within one page. The purpose is to show which elements play the greatest role in achieving the goal of this website. Multivariate tests are more complex and suit more advanced testers.
The number of combinations is the product of the versions of each element. Three headlines, two hero images and two button labels give 3 × 2 × 2 = 12 versions of the page, and each needs enough visitors of its own.
What you gain is knowledge of interactions. Nielsen Norman Group uses the example of a product page with a photo or a video and a button that says “Buy Now” or “Add to Cart”. Two A/B tests run one after the other could pick the video first, then a button label that only works well with the photo, so the best pairing would never be tested. A multivariate test shows all four combinations at once and reveals which element moves the result most.
Despite the name, this is not multivariate analysis in the statistical sense, where bivariate analysis looks at two variables and multivariate analysis at three or more, such as a regression that explains sales by price, season and ad spend. A multivariate test borrows its logic from the design of experiments: several factors, each with a few levels, tested together.
Full factorial and partial factorial tests
A full factorial test shows every combination with equal traffic. It gives the cleanest answer, including all interactions, but the traffic it needs grows with every element you add.
A partial (fractional) factorial test shows only a planned subset of combinations, usually chosen with the Taguchi method, which assumes some interactions are small enough to ignore. Adobe Target’s documentation shows the saving: three elements with two versions each need 4 experiences instead of 8, and four elements with three versions each need 9 instead of 81. The interactions you assumed away stay invisible, so use partial designs when the elements are fairly independent.
Which type of test should I use?
A/B testing is a great method if you need quick results: since the changes between pages are large, the winner is easier to see. It’s also the right choice if you don’t have a lot of traffic to your website.
To run a successful multivariate test, you need enough traffic. Use it when you want to change the page subtly and understand how elements work together, to gradually improve an existing design, or to get results you can apply to a larger redesign of your site. Nielsen Norman Group calls it incremental versus radical change: test a radical redesign with an A/B test, then refine the winner with a multivariate test.
Traffic decides. In Mixpanel’s example, a page with 1,000 views a week gives each version 500 visitors in an A/B test, but only about 83 in a test with 12 combinations. Adobe recommends a multivariate test only when you have at least three elements to test.
Remember that general traffic on your site is not enough: the pages you test also need significant traffic. Choose pages that people visit regularly, so the test actually collects data.
When should I use A/B tests or multivariate tests?
The answer depends on the number of conversions the page generates per month. Our rules of thumb:
- Fewer than 200 conversions per month: first increase the number of visitors to your website through other channels (SEO, PPC, community, etc.). If you want to start testing, use A/B tests with micro conversions, such as clicks on “Add to cart”.
- Between 200 and 500 conversions per month: start A/B testing with two or three changes from the original.
- Between 500 and 1,000 conversions per month: A/B tests can carry five to eight changes from the original.
- More than 1,000 conversions per month: you can choose between extended A/B tests and limited MVT tests.
- If you have never tested on your website before, start with A/B tests even when your data shows you could do more.
How long should I run my tests?
Don’t stop a test before both the original and the winner have collected 100 conversions; in our practice we require at least 500 conversions per variation. Run every test for at least a week, in full weeks, so that weekdays and weekends are both covered. If a test runs for more than four weeks without a clear winner, treat it as inconclusive. Aim for a test confidence level of over 95%. We recommend a minimum of 90%, and in a few low-risk cases 85% is acceptable.
Set the sample size in advance with a sample size calculator, and don’t stop the moment the tool first shows significance. Evan Miller showed in 2010 why this “peeking” is dangerous: in his worst case, a test with no real difference, checked after every visitor and stopped at the first significant result, gave a false positive rate of 26.1% instead of the intended 5%. The fix is a fixed sample size, or sequential or Bayesian methods built for checks during the test.
For short-lived content, such as a headline or a holiday promotion, a multi-armed bandit can fit better. It shifts traffic toward the better version while the test runs, so you lose fewer conversions, but it doesn’t give you the clean significance of an A/B test.
How to perform any test correctly?
Collecting relevant data is the basis of conversion optimization. A/B and MVT tests are great tools for it if they are run correctly:
- Planning: knowing what to expect from testing has a huge impact on the results. Tests run without goals and without deciding which results to collect are probably a waste of time, money and, most of all, effort. Write a hypothesis for each test: what you change, which metric should move and why.
- Traffic: multivariate tests are greedy and need many visitors, so first make sure you have enough traffic to run one. For A/B testing, traffic is less of a problem.
- Length of time: when you stop a test affects the results. Tests should not run longer than a month, but always leave enough time to reach statistical significance. Don’t rush to stop a test when the conversion rate looks dramatically high, to avoid false alarms; in an MVT the temptation is even stronger than in an A/B test.
Testing in conversion rate optimization
A/B testing is the default and most-used procedure in CRO programs. Implemented properly, testing removes guessing from conversion optimization: every change you keep is the result of a conscious decision. Each site is unique, and what works on one page may or may not work for you.
Increasing conversion rates starts with understanding the target market, creating personas representing that market, and then applying the elements of the conversion structure in the context of these personas.
Putting this into practice is not a walk in the park. Don’t limit yourself to the conversion rate: look for a version that raises the average order value or keeps visitors more involved, and add a guardrail metric, such as returns or unsubscribes, so that a gain in one place doesn’t hide a loss in another.
Team members and stakeholders usually disagree about the best website or landing page design, the best user flows or the best way to sell. A test settles it. Test each modification against the old version and compare the impact on conversions. Split testing software allows you to make these comparisons, for example to see which of two home page designs converts better.
Testing tools in 2026
Google Optimize, the free tool many small sites started with, stopped working on September 30, 2023. Google now points users to third-party testing tools that integrate with Google Analytics, naming AB Tasty, Optimizely and VWO, and has opened its APIs to other tools. Most testing tools support A/B, split URL and multivariate tests, and many add bandits and personalization. AI makes it cheap to write ten headlines instead of two, but every extra version still needs its own share of traffic, so the arithmetic above doesn’t change.
The testing process, step by step
- Find the right page for optimization. On a large website this takes time. The best page to optimize is the one that loses the most visitors. Then check how many visitors on this page will actually take part in the tests, and never run tests for more than four weeks.
- Define your goal. It doesn’t have to be the main (macro) conversion: a test of a smaller step can still have a big impact on the macro conversion rate.
- Be selective about the elements you test, and pick only those with the greatest impact.
Any questions? Let's talk!
Write to us,