Cold DM A/B Test Checklist

How to use this checklist

Design the test on paper before sending a single message. Decide the one variable, the sample size, and the success metric up front, because deciding after the replies arrive is how you talk yourself into a result the data does not support. A test designed in advance is a measurement; one designed after the fact is a story.

Test design table

Write the design before sending. The table is your commitment; once both variants are in flight, the table stops you from quietly moving the goalposts when the result is inconvenient. A written design is what separates a test from a toss of the coin.

Common test mistakes

The first mistake is testing two things at once, so you cannot say which one won. The second is a sample too small to mean anything, where a single reply swings the rate. The third is declaring a winner on a single week of noise. All three produce answers you should not trust, and trusting them is how programs get stuck.

• Two changes at once, so cause is unknown.
• Under 50 sends per variant, pure noise.
• Winner called before both variants finish.
• Metric switched after the fact to fit the result.

Sample size and reading

Run at least 50 to 100 sends per variant before cutting or promoting, and prefer a clear gap over a thin one. A one-point difference on small samples is noise dressed as insight. Patience here is what makes the next decision trustworthy, because a wrong 'winner' propagates into every message you send afterward.

From result to action

When a variant wins clearly, promote it and design the next test on a different variable. The loop is what compounds: each clean test improves one element, and over a quarter the messages get systematically better instead of randomly different. A program that tests cleanly learns; one that guesses stays flat.

1. 1Promote the clear winner to the live message.
2. 2Document the result with the sample size.
3. 3Design the next test on a new variable.
4. 4Avoid re-testing the same thing without a reason.

What to test, in priority order

Not all variables are worth the same test slot. Start where the leverage is highest and the change is cheapest, which is almost always the opener, then work down. Testing the closing line before the opener is like polishing a door nobody opens; fix what gates the most replies first.

Worked example: reading a result honestly

Suppose Variant A gets 9 replies from 100 sends and Variant B gets 12 from 100. That is 9 percent versus 12 percent, a three-point gap on a decent sample, so B is a plausible winner worth promoting and confirming. But if A got 4 from 40 and B got 6 from 40, that same-looking edge rests on two extra replies and is well inside the noise — no decision yet.

1. 1Check both variants reached at least 50 to 100 sends.
2. 2Require a gap of roughly 2 points or more before calling it.
3. 3If the gap is thin, extend the sample rather than guessing.
4. 4Record the sample size beside the result so future you can trust it.

Documenting tests so learning compounds

A test you do not record is a test you will repeat. Keep a simple log of every experiment with the variable, the sample, the result, and the decision, so the team builds a library of what works instead of relitigating the same questions. Over a quarter this log becomes the most valuable asset in the program: a map of your own audience's preferences.

Building a testing roadmap

Random tests produce random learning. Sequence your experiments so each builds on the last, starting at the top of the funnel where leverage is highest and moving down only once the earlier elements are settled. A roadmap turns scattered A/B tests into a compounding program that gets systematically better each cycle.

1. 1Lock the opener first, since it gates every downstream rate.
2. 2Then test the personalization hook against the winning opener.
3. 3Move to offer framing once the top of the funnel is stable.
4. 4Revisit an earlier element only if audience or channel changes.

Judging significance without the math

You do not need statistics training to avoid the worst A/B mistakes; you need a few rules of thumb that keep you from reading noise as signal. The core idea is simple: small samples swing wildly, so require both a decent sample and a clear gap before you believe a result. When in doubt, gather more data rather than deciding early.

Isolating the variable cleanly

The single most common way A/B tests fail is that the two variants differ in more than one way, so the result cannot be attributed. Clean isolation means holding everything else identical: same audience, same volume, same timing, same follow-up, so the only difference is the one thing you are testing. If the list or the send window differs between variants, you are testing that too, whether you meant to or not.

• Split the same list randomly, not by segment, across the two variants.
• Send both variants in the same window, not A this week and B next.
• Keep the follow-up sequence identical for both variants.
• Change one element only; if you must change two, run two tests.

Suggested image brief

Quick checklist

• One variable chosen, two changes never tested at once.
• Sample size set to at least 50 to 100 per variant.
• Success metric named before sending (reply rate).
• Decision rule written so the goalposts stay fixed.
• Both variants run to completion before reading.
• Winner promoted only on a clear, sized gap.
• Result documented to guide the next test.

Frequently asked questions