Written in by More Playables production team
A Creative Testing Framework for Playable Ads
Connect a named production change to a metric definition, decision, confounder record, and follow-up action.
Testing framework; all sample numbers are hypothetical
Updated
A test starts with the decision
Write what you will do if the evidence points each way. If no result could change the next action, the team is producing a variation rather than testing a decision.
The test record
Field | Example |
|---|---|
Question | Does a shorter instruction increase starts without reducing completion? |
Control | Master M01, instruction A |
Variant | V01, instruction B; all listed production elements held constant |
Primary observation | Playable start rate, defined by the network event |
Guardrail | Completion among starters and downstream install quality |
Allocation | Campaign owner records audience, placement, budget, dates, and delivery method |
Decision | Keep B, keep A, iterate, or declare inconclusive |
Single and combined changes
Changing one element can make interpretation easier. It is not always the right business choice. A fatigue response may require a deliberately combined hook-and-art refresh. Label that treatment honestly and avoid attributing the result to one component.
Hypothetical sample
Imagine the control receives 100,000 impressions, 4,000 starts, 1,600 completions, and 480 attributed installs. A variant receives the same rounded impression count, 4,600 starts, 1,610 completions, and 475 attributed installs. Those numbers are fictional. They suggest the shorter instruction may change starts without a corresponding install increase, but they do not establish statistical certainty, explain allocation quality, or show downstream retention.
Confounders to record
- Audience, geography, placement, operating system, device mix, schedule, bid and budget strategy.
- Unequal delivery or optimization decisions made by the network.
- App-store page, game version, attribution model, privacy thresholds, and reporting delay.
- Simultaneous creative, offer, event, seasonality, or competitor changes.
No universal sample-size rule
Required evidence depends on baseline rates, expected effect, variation, allocation, decision cost, and analysis method. Do not use one fixed number of installs as a guarantee. The campaign analyst should define the method before reading the result, and the decision record should allow ‘inconclusive.’
Close the loop
- Join the campaign creative ID to the exact variant ID.
- Freeze metric definitions and windows before comparison.
- Record missing or delayed data.
- Make the keep, iterate, replace, or investigate decision.
- Turn the learning into the next specific hypothesis rather than a vague best practice.
Copy the editable matrix and replace every hypothetical value with the campaign owner's sourced data.
Sources
Put this into practice
- Creative test matrix
Blank fields and a hypothetical sample.
- Creative test metrics
Calculate and interpret IPM, CPI, retention, and ROAS.
Make your next test playable
Turn a creative question into a scoped playable, variant set, and delivery plan.
Start a playable brief