Cold DM Variant Test Worksheet

How to use this test worksheet

Pick one variable to test, write the hypothesis, set the sample size, then split traffic evenly. Do not change the variable mid-test or you invalidate the result. Read the winner only after the planned sample is reached, not the moment it feels ahead.

Hypothesis and variable

A good test changes exactly one thing: the hook, the offer, or the ask. Testing three changes at once tells you something moved but not what, which is useless for the next message.

Sample size planning

Small samples lie. Use a sample size that can detect the lift you care about; for a few percent difference you need hundreds of sends per arm, not dozens. The ab testing guide covers the math; this sheet captures your plan.

• Aim for at least 200 to 300 sends per variant.
• Increase sample if the expected lift is small.
• Split 50/50 so arms are comparable.

Running the test steps

Execution discipline is the test. Randomize assignment and keep all other conditions equal, or the result reflects your process, not your message.

Results table

Fill the results once the sample completes. The gap and the direction tell you what to scale; the magnitude tells you whether it was worth the effort.

What to do with the winner

Promote the winner to your default and start the next test on a new variable. Testing is a loop, not a one-time event; the teams that win are the ones who keep cycling instead of declaring victory after one result.

Worked test example

Here is a completed A/B test of two hooks on 600 total sends, 300 per arm. The gap is small but the sample is large enough to trust it, which is the whole point of planning sample size before you read any result.

Reading the result and next test

Promote B to default, then pick the next single variable to test. Stacking tests this way compounds; four wins of 2-4 points each can lift reply rate from 9% toward 20% over time without any single gamble.

Designing the hypothesis

A test is only as good as its hypothesis. Write the expected direction and size before you send, because without a stated bet you will rationalize any result after the fact and learn nothing you can trust.

Edge cases and caveats

Small samples and ties are where teams fool themselves. Know the calls before the data arrives so you do not invent a winner from noise after the fact.

Do and don't quick list

• Do write the hypothesis before sending.
• Do wait for the full sample.
• Don't test more than one variable at once.
• Don't call a 1-point gap a win.

Copy-this results table

Use this shell and fill it only after the full sample is reached. Reading it early is the mistake that turns noise into a false winner you then scale with confidence you should not have.

What a decisive test looks like

A decisive test has one variable, a pre-stated hypothesis, and a sample large enough to trust. Without all three, you have an opinion with a spreadsheet attached, not evidence you can scale.

Troubleshooting the test

When a test teaches you nothing, it usually broke one of the three rules: one variable, a pre-stated hypothesis, and a big enough sample. Fix the broken rule before you trust any result the dashboard shows you.

Your first 15 minutes

Design the next test on a card before you send. A test designed in the moment is a test designed to confirm what you already believed, which is not testing at all.

1. 1Name the one variable to change.
2. 2Write the hypothesis and expected lift.
3. 3Set sample size at 300 per arm.
4. 4Plan the promotion of the winner.

Before you launch: final check

Before the test sends, confirm one variable, a written hypothesis, and a sample size of at least 200 per arm. Without all three, the result is a story you tell yourself, not evidence you can safely scale.

• Exactly one variable changed.
• Hypothesis written with expected lift.
• Sample 200-plus per arm.
• Traffic split 50/50 and randomized.

Statistical significance sanity

A 4-point lift looks great, but is it real? At 300 per arm, a difference of a few points can still be within normal chance. Use a simple rule: the larger the expected lift, the smaller the sample you need, and vice versa.

Worked second test example

After promoting the proof hook, the next test changed the value beat while holding the hook. On 500 per arm, reply rate moved from 13.0% to 15.5%, another real lift that compounds on the first. Stacking small wins is how programs climb from 9% to 20% over a few cycles.

• Promote B as the new control.
• Log the learning to the swipe file.
• Choose the next single variable.

Test program red flags

These are the ways teams waste test cycles. Each one produces a result they cannot trust, which is more dangerous than no test at all because it feels like evidence.

Test documentation template

Write up every test the same way so results compound into a knowledge base instead of scattered memory. A one-line doc per test is enough to avoid repeating a loser or forgetting a winner six months later.

Suggested image brief

Quick checklist

• One variable isolated for the test.
• Hypothesis written with expected lift.
• Sample size set at 200-plus per arm.
• Traffic split 50/50 and randomized.
• All other conditions held constant.
• Results table ready for end-of-test.
• Winner promotion and next test planned.

Frequently asked questions