Cold DM A/B Testing Guide
A/B testing is the best way to learn what actually works in your cold DMs. This guide covers what to test, how to set up a test, how large a sample you need, and how to interpret the results without falling into common statistical traps.
The A/B Testing Framework for DMs
A proper A/B test for cold DMs follows the same principles as any scientific test: change one variable, control everything else, and collect enough data to make a confident decision.
Step 1: Pick one variable to change
Only change one element between variant A and variant B. If you change two things (opener and CTA) and one performs better, you won't know which change caused the improvement. Change one thing at a time, test it, then change the next thing.
Step 2: Keep everything else identical
Use the same platform, same audience segment (job title, industry, company size), same time period, and same sender profile. If you test variant A on Monday and variant B on Friday, the day of the week could skew your results.
Step 3: Determine sample size
Aim for 50–100 sends per variant minimum. If you expect a low reply rate (5% or less), increase sample size proportionally to get a reasonable number of replies for comparison.
Step 4: Track the right metric
For opener tests, track unique reply rate. For CTA tests, track positive reply rate (the percentage of replies that show interest). For message length tests, track both — shorter messages might get more replies but fewer positive ones.
Step 5: Analyze and decide
If one variant leads by 20% or more after the minimum sample size, call the winner. If they're within 10% of each other, send more until you have confidence or decide the difference doesn matter for your business.
What to Test: Variables Table
Start with variables that are most likely to affect your reply rate. The table below shows testing options and typical sample sizes.
| Variable | What to change | Min. sample per variant | Duration |
|---|---|---|---|
| Opener style | Lead with a personalized reference vs. a direct question | 50–100 | 3–7 days |
| Call to action | Ask for a reply with a question vs. request a short conversation vs. offer a free resource | 50–100 | 3–5 days |
| Personalization depth | Line 1 custom vs. first 2 sentences custom vs. full message custom | 50–100 | 4–7 days |
| Message length | 2–3 lines vs. 5–6 lines | 50–100 | 3–5 days |
| Framing (recipient focus) | Benefit them vs. help their customer vs. improve a metric they own | 50–100 | 5–10 days |
Use the DM Script Scorecard to identify which element of your script needs the most improvement before you create variants.
Sample Size and Statistical Significance
You don't need a degree in statistics to run a good A/B test. The key principle is that a bigger sample gives a more reliable result.
The “50/50” rule
Send at least 50 of each variant. If you can send 100 of each, even better. At 50 per variant, you can detect a reply rate difference of 15% or more reliably. For smaller differences, you need larger samples.
Reply rate vs. reply count
If variant A gets 8 replies out of 50 (16%) and variant B gets 4 out of 50 (8%), A is the winner. But if A gets 8 and B gets 6 (12% vs 8%), the difference is not yet significant. You would need to send more to be confident B is actually worse.
The “Big Enough” rule
For your purposes, you don't need to hit a 95% confidence level. If after 100 sends of each variant, variant A is consistently at 12% and B at 7%, that is a winner. You can be confident enough to move forward. Only if the difference is < 20% should you worry about statistical significance.
For more on using your actual reply rates in forecasting, see the Reply Rate Calculator.
Controlling Variables
To get clean A/B test results, control everything except the variable you are testing:
- Platform: Test variants on the same platform. Do not compare LinkedIn results to Instagram results.
- Audience segment: Send both variants to the same type of prospect (same job title, same industry). Different audiences will have different baseline reply rates.
- Time period: Run test variants concurrently, not one after the other. Day-of-week and seasonality affect reply rates.
- Time of day: If possible, send both variants at similar times to avoid time-of-day bias.
- Sender identity: Use the same sender profile for both variants.
- Follow-up sequence: Keep follow-ups identical for both variants. A difference in follow-up could mask the real effect of the script.
Testing Calendar Template
A typical 8-week testing timeline that covers two A/B test rounds:
| Period | Activity | Output |
|---|---|---|
| Week 1–2 | Audience selection and script variant creation | 2 variants (A and B) with exactly one variable changed |
| Week 3 | Send 50–100 of each variant; track unique replies and positive replies | Initial data for each variant |
| Week 4 | Analyze results, pick winner, or extend test | Decision: which variant to scale or whether to rerun |
| Week 5–6 | Test next variable on the winning variant | Two more variants with a new variable changed |
| Week 7–8 | Run second test and analyze | Optimized script with two improved elements |
After two test rounds, you should have a script that is clearly better than your starting version. Scale that script using the Cold DM Calculator and continue monitoring to see if the improved rates hold at scale.
Common A/B Testing Mistakes
- Changing two variables at once so you cannot identify the cause of improvement
- Calling a winner after 15 sends of each variant (too small to trust)
- Ignoring day-of-week effects (Monday morning vs. Friday afternoon)
- Comparing results across different audience segments
- Testing variables that don't matter (the color of your display picture, for example)
- Not tracking data consistently (reply rate, positive reply rate, and the actual content of replies)
Frequently asked questions
How long should I run a test?
Run a test until you have at least 50 sends per variant. For low-reply-rate markets (5% or less), this can mean 1,000 total sends (500 per variant) to get 25 replies for comparison. The minimum duration should be 3–5 days to account for day-of-week effects. Do not call a test before you have at least 50 sends per variant regardless of how many days have passed.
What if results are tied (within 5% of each other)?
A tie means either you haven't sent enough to see a difference, or the variable you changed doesn't matter much. Increase sample size to 150–200 per variant. If they are still tied, pick the variant that is easier for you to produce consistently or test a different variable. Not every variable is worth testing — some differences are negligible.
Can I test across different platforms?
No, because the platform is itself a variable. If you test variant A on LinkedIn and variant B on Instagram, you can't tell whether the result is due to the script or the platform. Run separate tests on each platform and compare results between platforms only after you have an optimized script on each one.
Run a real experiment on your reply rate.
Test two script variants and see the forecast difference before you scale.
Forecasts are estimates based on user-provided assumptions. Results are not guaranteed.
Related: DM Script Scorecard · Reply Rate Calculator · Cold DM Reply Rate Guide · Cold DM Calculator