A/B testing

Running an A/B test

A/B testing lets you run two versions of a popup at the same time and find out which one performs better. Instead of guessing whether a shorter headline works, or whether 10% off beats free shipping, you put both in front of real visitors and let the numbers decide.

Pop Convert splits your traffic between the two campaigns. Half the visitors see Campaign A, half see Campaign B. Everything else stays the same, so any difference in results comes from the difference between the two popups.

You'll find it under the A/B Testing tab at the top of the Displays page.

Before you start

You need two campaigns already built. A/B testing pairs two existing campaigns; it doesn't create them for you. So build both first on the Displays page, then come back here.

Two things to get right before you set the test up:

• Change one thing at a time. If Campaign B has a different headline, a different offer, a different image, and a different button color, and it wins, you've learned nothing useful. You won't know which change did the work. Pick one variable, change only that, and you get an answer you can apply to every popup you build afterwards.
• Make the two campaigns comparable. Same trigger, same targeting rules, same pages, same audience. If Campaign A shows on exit intent and Campaign B shows after 5 seconds, you're testing timing, not the content you thought you were testing.

Creating a test

Click Create a test in the top right. A short form opens.

Test Title

Name the test after what you're testing, not after the campaigns. You'll have several of these eventually, and "Test 1" tells you nothing in three weeks.
Good titles:

• Discount vs free shipping offer
• Short headline vs long headline
• Exit popup with image vs without

Campaign A

Pick your first campaign from the dropdown. This is usually your current popup, the one you're trying to beat. Think of it as the control.

Campaign B

Pick your second campaign. This is the variation you're testing.
The Campaign B dropdown stays greyed out until you've chosen Campaign A. Once A is selected, B unlocks.

Click Create and the test is set up.

What to test

If you're not sure where to start, these are the variables that tend to move the numbers most, roughly in order:

1. The offer. 10% off vs free shipping vs early access vs a gift. Usually the biggest swing of anything you can test.
2. The headline. Benefit-led vs direct vs curious. Second biggest.
3. Trigger timing. Exit intent vs a time delay vs scroll depth. Worth testing separately from content.
4. Popup format. A centered modal vs a smart bar vs a corner popup.
5. Form length. Email only vs email plus name.
6. Button copy. "Subscribe" vs "Get my discount" vs "Send it over."
7. Visuals. With a product image vs text only.

Start at the top of that list. Testing button color before you've tested the offer is optimizing the wrong thing.

Reading the results

This is where most tests go wrong. Two rules:

• Give it enough traffic. A test with 40 views per campaign will show a "winner" that's pure noise. As a rough floor, wait for a few hundred views on each side and a meaningful number of conversions before you take the result seriously. Low-traffic stores may need to leave a test running for several weeks.
• Give it enough time. Run for at least one full week, and ideally two. Traffic behaves differently on weekdays and weekends, and a test that only covers Tuesday to Thursday will mislead you. If you're running a promotion or a sale mid-test, that skews things too.

A close result is still a result. If the two campaigns finish within a point or two of each other after plenty of traffic, that means the thing you changed doesn't matter much to your customers. That's useful. Move on and test something bigger.

Once you have a winner: pause the test, keep the winning campaign running on its own, and use what you learned as the starting point for your next test.

Common mistakes

1. Stopping early because one side jumped ahead. Early leads flip all the time. The first 50 visitors tell you almost nothing.
2. Testing during an unusual period. Black Friday, a site outage, a viral post, or a big paid campaign will all distort results. Test during normal traffic.
3. Changing a campaign mid-test. If you edit Campaign B halfway through, the data before and after the edit aren't comparable. Let the test finish, then make changes.
4. Running several tests on the same page at once. They interfere with each other. One test at a time per placement.
5. Not writing down what you learned. Keep a short note of each test and its outcome. After six months that list is worth more than any single test result.

Common questions

• How is traffic split? 

Evenly. Roughly half of eligible visitors see Campaign A and half see Campaign B.

• Do I need to enable both campaigns?

Yes. A test only runs if both campaigns are active. If one is paused or in draft, half your traffic has nothing to show, and the test either won't start or will report results for one side only. Check both campaigns on the Displays page before you create the test, and leave both running until the test is finished.

• Will the same visitor see both versions? 

No. A visitor stays with the version they were first assigned.

• Can I test more than two campaigns at once? 

No. Tests pair exactly two campaigns. To compare three ideas, run the first two, then test the winner against the third.

• Can I use the same campaign in two different tests? 

This isn't recommended even where it's possible. Overlapping tests make results hard to interpret.

• What happens when I delete a test? 

The test is removed. The two campaigns themselves aren't deleted, they stay on your Displays page.

• How long should I run a test? 

One to two weeks minimum, longer if your traffic is low. Time matters as much as volume.

Updated on: 16/09/2026