Both stores have free native A/B testing: Apple's Product Page Optimization runs up to 3 treatments against your default for up to 90 days, and Google Play's Store Listing Experiments tests screenshots, icons, and feature graphics the same way. Test one variable at a time, give each variant enough traffic to reach 95% confidence (roughly thousands of views), and run it 14 to 21 days before reading the result. The lead screenshot is almost always the highest-impact thing to test first.
What A/B testing tools do the app stores give you?
Apple — Product Page Optimization (PPO). Inside App Store Connect, you can create up to 3 alternate treatments of your product page and run them against your current default. Apple splits your real store traffic across them and reports which converts best, with up to a 90-day run. You can test screenshots, the app icon, and the preview video.
Google Play — Store Listing Experiments. In Play Console, you run experiments that split traffic between your current listing and variants. You can test screenshots, the feature graphic, the icon, the short and full descriptions, and more. Google shows you a confidence indicator and a recommended winner.
Both are using *your own live store traffic*, which is the big advantage over any external mockup-survey tool. A panel of strangers clicking a survey isn't the same as a real person deciding whether to install. The store experiment measures the actual decision, with actual store intent behind it.
The catch is traffic. These tools need enough visitors to reach statistical significance, and a brand-new app with twenty page views a day will never conclude a test — or worse, will show you a "winner" that's just noise. I'll come back to that, because it's the single most common way people fool themselves with ASO testing.
If you want the official mechanics, Apple documents PPO in the App Store Connect help and Google documents experiments in the Play Console help area. Read the real docs before you trust a blog's screenshot of an old dashboard — both UIs change.
What should you test first?
1. The lead screenshot. This is the one. Analysis across enormous volumes of store impressions keeps finding that screenshot creative drives bigger conversion swings than the icon, title, and description combined — and the *first* screenshot does most of that work because it's what shows in search results. If you only ever run one test, test your lead frame: a different feature in the lead position, or a different headline on the same frame.
2. Benefit headline vs feature headline. "Track every workout" versus "Built-in workout timer." Outcome language usually beats feature language, but *usually* isn't *always* for your specific audience — which is exactly why you test instead of guessing.
3. With device frame vs without. Framed screenshots generally read as more professional and convert better, but it's worth confirming for your category. Some minimalist app listings do fine with frameless, full-bleed screens. Run it and see.
4. Screenshot order. Same eight screenshots, different sequence. Moving your strongest feature from slot three to slot one can shift conversion without changing a single pixel of the art.
5. Background color and palette. Lower impact than the above, but a dark vs light test sometimes surprises you, especially against the store's own background.
The discipline that matters: test one variable at a time. If you change the lead screenshot *and* the headline *and* the background in one variant and it wins, you've learned nothing about *why*. You can't bank the lesson. Change one thing, learn one thing. It's slower, and it's the only way the results compound.
Because you're producing variant after variant, the cost of making each one has to be near zero or you'll quit testing. That's the whole reason I frame and re-export in a browser tool — swapping a lead frame or a headline and re-exporting a fresh variant in the PlayMockUp studio takes a couple of minutes, so generating the B version of a test isn't a project, it's a coffee break.
How long should a screenshot test run — and how much traffic?
Run for at least 14 to 21 days. A shorter window catches a weekday/weekend skew or a single weird traffic spike and reads it as signal. Two to three weeks smooths that out. Don't peek on day three, see your variant "winning," and ship it — early leads flip constantly.
Get enough views per variant. The common rule of thumb is roughly 5,000 views per variant before you read anything, and you only call a winner at 95% confidence (the dashboards show this as p < 0.05 or a confidence percentage). Below that, what you're seeing is noise wearing a costume.
Here's the uncomfortable math for a new app: if you get 100 product-page views a day, a two-variant test needs 10,000 total views, which is 100 days — longer than Apple even lets the test run. So for low-traffic apps, formal A/B testing basically doesn't work yet. That's not a failure; it's just the wrong tool for your stage.
What do you do instead at low traffic? You make bigger, more confident changes based on principles rather than tiny tested tweaks. Don't A/B test a background color shade when you have no traffic — just apply the known-good move (clear headline, strong lead, consistent frames) and ship it. Save formal testing for when you've got the volume to trust it. I ran my first app at maybe 80 views a day and wasted a month on a "test" that never reached significance. Lesson learned: at small scale, follow the playbook; at large scale, test the playbook.
One more honest caveat about Apple specifically: PPO conversion improvements sometimes don't fully carry over to your live default, because the test traffic and your overall traffic mix can differ. Treat the winner as a strong signal, apply it, and keep watching your real conversion rate after — not just the experiment's number.
Why do most screenshot A/B tests fail to teach anything?
Testing too many things at once. Covered above, but it's the number one killer. A variant that changes five things and wins teaches you nothing actionable. One variable per test.
Calling it too early. Day-two leads are almost always noise. If you can't wait two weeks without acting on the chart, don't start the test.
Not enough traffic. A test that never hits 95% confidence isn't a result — it's a coin flip you stared at for a month. Know your daily views and do the math before you start.
Testing trivia. Whether your headline is 18px or 20px is not worth a 21-day test. Test things that could plausibly change someone's install decision: which feature leads, benefit vs feature framing, frame vs no frame. Big swings, not pixel nudges.
No hypothesis. "Let's try some stuff" isn't a test. "I think leading with the timer feature will beat leading with the dashboard because new users care about the core action first" is a hypothesis — and whether it wins or loses, you learn something. Write the hypothesis down before you build the variant.
Forgetting to ship the winner everywhere. You tested on the App Store, found a better lead frame, updated iOS, and left your Play listing on the old version for three months. Now your two stores disagree. When a test wins, roll the lesson to both platforms.
The meta-point: A/B testing is a multiplier on a good process, not a substitute for one. If your screenshots are fundamentally weak — no headlines, no frames, no clear lead — testing background colors is rearranging deck chairs. Get the fundamentals right first (the rules in how to make app screenshots that get downloads are where I'd start), then use testing to sharpen what's already solid. And when you're producing your control and variant, pulling both from the same device frame library keeps everything but your one tested variable identical, which is the only way the result means anything.
Frequently asked questions
Is app store A/B testing free?
Yes. Apple's Product Page Optimization and Google Play's Store Listing Experiments are both built into the developer consoles at no cost. They split your real store traffic across variants and report a winner, which is more trustworthy than a paid survey panel because it measures actual install decisions.
How many screenshot variants can I test at once?
Apple's Product Page Optimization lets you run up to 3 alternate treatments against your default at the same time. Google Play's experiments also support multiple variants. Even though you can run several, it's best to change only one variable per variant so you can tell what actually caused the difference.
How long should I run an app screenshot A/B test?
Run it at least 14 to 21 days so you average out weekday and weekend swings, and don't act on early leads because they flip often. Only call a winner once a variant hits 95% confidence, which usually needs around 5,000 views per variant. Below that traffic, the result is noise.
Can I A/B test screenshots with low traffic?
Not effectively. Formal tests need thousands of views per variant to reach significance, so a new app with a handful of daily views will never conclude one. At low traffic, apply the known-good principles confidently instead and use a fast tool like the [PlayMockUp studio](/create) to ship strong screenshots, then test later once your volume grows.
What should I test first in my app store screenshots?
Your lead screenshot. Across huge volumes of store data, screenshot creative drives more conversion change than the icon, title, and description combined, and the first frame does most of that work. Test a different feature in the lead slot or a different headline on it before you touch anything else.
Why did my A/B test not show a clear winner?
Usually it's too little traffic to reach 95% confidence, too short a run, or too many variables changed at once. Make sure each variant differs by only one thing, give it two to three weeks and thousands of views, and write a hypothesis before you start so you learn something whether it wins or loses.
Build the mockup in your browser.
Drop a screenshot into a real device frame and export at the exact store size — free, no signup.