A/B Testing

What is A/B Testing?

A/B testing lets you compare two versions of a Cancel Flow to see which one performs better. Churnkey splits your traffic between a control variant (your current flow) and a test variant (your modified flow), then measures which one saves more customers and generates more revenue.

The 5 Lifecycle States

Every A/B test moves through five states:

1. Not Started — Test created, waiting for you to start
2. Enrolling (7 days) — Traffic splits between Control and Test
3. Tracking (30 days) — Measuring if saved customers actually stay
4. Awaiting Decision — Data complete, waiting for you to pick a winner
5. Completed — Winner declared, test archived

Total timeline: ~37 days minimum (7 days enrollment + 30 days tracking)

State 1: Not Started

This state gives you time to review your test variant in the Cancel Flow builder and ensure everything is configured correctly before going live.

State 2: Enrolling (7 days)

During enrollment, every customer who enters your Cancel Flow is randomly assigned to one variant. This assignment stays consistent if they return later.

When enrollment ends, the cohort is locked. No new sessions enter the test after day 7.

State 3: Tracking (30 days)

Why this phase matters: When a customer accepts an offer (pause, discount), we mark them as "saved" but we don't know yet if they actually stayed. They might cancel again next week, skip their next invoice, or churn silently.

The 30-day tracking window captures the real outcome by measuring whether saved customers pay their next invoice and remain subscribed.

State 4: Awaiting Decision

Look at the statistical confidence level to know how trustworthy your results are:

State 5: Completed

When you confirm your decision, the winning variant becomes active and the losing variant is deactivated. All future customers in this segment see the winner.

Setting Up a Test

Prerequisite: A/B tests can only run on segmented Cancel Flows, not on your primary (default) Cancel Flow.

Primary Metrics Reference

Your primary metric determines which variant "wins." Choose based on what matters most to your business.

Revenue Per Exposure (Recommended)

• Formula: Total Revenue from Saved Customers ÷ Total Sessions
• Best for: Overall business impact—balances save rate against revenue quality
• Example: Test saves 100 customers at $10 each ($1,000). Control saves 50 at $25 each ($1,250). Control wins despite lower save rate.

Save Rate

• Formula: Customers Saved ÷ Total Sessions × 100
• Best for: Maximizing retention count; good for testing copy, layout, or flow length
• Watch out: Can lead to over-discounting if used alone

Reactivation Rate

• Formula: Saved Customers Who Paid Next Invoice ÷ Total Saved Customers × 100
• Best for: When saved customers frequently cancel before their next payment
• Why it matters: A customer who accepts an offer but never pays again = $0 value

Pause Acceptance Rate

• Formula: Customers Who Accepted Pause ÷ Total Sessions × 100
• Best for: Testing pause duration, messaging, or positioning specifically

Discount Acceptance Rate

• Formula: Customers Who Accepted Discount ÷ Total Sessions × 100
• Best for: Optimizing discount percentages, durations, or presentation
• Watch out: Higher acceptance with larger discounts might hurt revenue

LTV Extension

• Formula: Sum of Additional Months Stayed ÷ Total Saved Customers
• Best for: Long-term optimization when you have historical LTV data
• Note: Requires longer tracking periods to produce meaningful data

Reading Results

The Results Dashboard

The results dashboard shows side-by-side performance cards for Control and Test variants. Each card displays the same metrics, making it easy to compare performance directly.

The variant with better performance on your primary metric is highlighted. If the difference is statistically significant (95%+ confidence), you'll see a "Significant" badge.

Key metrics on each variant card:

Offer Breakdown

The offer breakdown shows which retention offers customers accepted in each variant:

Compare the distribution between Control and Test to understand how customers are being saved, not just whether they're saved.

Retention Timeline

The retention timeline tracks what percentage of saved customers remain active over time (day 7, 14, 30, 60, 90). This reveals whether your saves are "sticky" or if customers churn shortly after accepting an offer.

A steep drop-off early in the timeline suggests customers are accepting offers but not genuinely retained. A flat line indicates strong long-term retention.

Expected Impact

Once results are in, the dashboard calculates the projected business impact if you roll out the winning variant:

These projections help you quantify whether the improvement is worth implementing. A statistically significant result with minimal ARR impact might not justify the change.

Making a Decision

When your test reaches "Awaiting Decision," you need to analyze the results and pick a winner. This section helps you understand what the numbers mean and how to decide.

Understanding Statistical Significance

Statistical significance tells you whether the difference between variants is real or just random chance.

When you flip a coin 10 times and get 6 heads, that doesn't prove the coin is biased—it could easily happen by chance. But if you flip 1,000 times and get 600 heads, something is definitely going on. A/B testing works the same way.

Important: You can always make a decision regardless of confidence level. Low confidence doesn't prevent you from choosing—it just means there's higher risk that the "winner" isn't actually better. You're making a judgment call, not a data-driven decision.

Understanding Lift

Lift measures how much better (or worse) Test performed compared to Control, as a percentage.

Formula: Lift = (Test - Control) / Control × 100

How Sessions Affect Confidence

The smaller the improvement you're trying to detect, the more data you need. Think of it like hearing someone in a noisy room:

Minimum requirement: Below 30 sessions per variant, confidence is automatically 0%. The math simply doesn't work with fewer samples.

Practical guidance based on your volume:

Bottom line: If you're making small tweaks expecting 5% improvements, you need a lot of data. If you're testing dramatically different approaches, you'll know faster.

The Decision Path

When evaluating your results, check these metrics in order:

Step 1: Check Statistical Confidence

• Is it 95%+? → You have reliable data to make a decision
• Is it below 95%? → Results are not mathematically reliable (see "Decision with Risk" cases below)

Step 2: Look at Your Primary Metric

• Which variant performed better on the metric you chose (e.g., Revenue Per Exposure)?
• How big is the difference? A 2% lift vs. a 20% lift have very different implications.

Step 3: Review Secondary Metrics

• Does the "winner" also perform well on other metrics?
• Watch for trade-offs: higher save rate but lower reactivation rate could mean you're saving customers who churn again quickly.

Step 4: Consider the Lift

• Positive lift = Test outperformed Control
• Negative lift = Control outperformed Test
• Near-zero lift = Both performed similarly

Decision by Case

Here are common scenarios you'll encounter and how to handle each:

Case 1: Clear Winner (High Confidence + Strong Lift)

Confidence: 98% · Lift: +55%

What this means: Test dramatically outperforms Control, and you have enough data to trust this result.

Decision: Choose Test. This is the ideal outcome—clear winner with high confidence.

Case 2: Insufficient Data (Low Confidence)

Confidence: 15% · Lift: +14%

What this means: Test looks better, but with only 15 sessions total, this could easily be random chance. The 15% confidence means there's an 85% probability this difference is noise.

Decision: The result is not statistically reliable. Your options:

1. Choose Control (recommended) — It's your proven baseline. Don't change what works based on unreliable data.
2. Choose Test anyway — If you have strong qualitative reasons to believe the changes are better, you can accept the risk.
3. Run another test — Wait until you have more traffic and test again.

Case 3: Equal Performance (No Difference)

Confidence: 50% · Lift: 0%

What this means: Both variants perform identically. Your changes made no measurable impact.

Decision: Choose Control. When there's no difference, stick with your original flow—it's simpler and already proven. Consider testing a more significant change next time.

Case 4: Test Performs Worse (Negative Lift)

Confidence: 99% · Lift: -40%

What this means: Your changes hurt performance. Control is significantly better, and the high confidence means this is definitely real, not random.

Decision: Choose Control. Your hypothesis was wrong—the test variant made things worse. This is still a valuable learning: you now know what doesn't work.

Case 5: High Volume, Small Difference

Confidence: 97% · Lift: +12%

What this means: Test is better, and with 11,000 total sessions you can trust this result. However, the improvement is modest (3 percentage points on save rate).

Decision: Choose Test. Even small improvements compound over time. A 3% save rate increase across thousands of customers adds up to significant revenue.

Case 6: Missing Data (Technical Issue)

Confidence: N/A · Lift: N/A

What this means: Something went wrong. Test variant received no traffic—possibly a configuration error, broken flow, or technical issue.

Decision: Choose Control (you have no choice). Investigate why Test got no sessions before running another test. Check that the Test variant is published and active.

Document Your Rationale

Use the rationale field to record why you chose this winner. Good documentation helps your team understand past decisions.

Examples:

• "Test showed 55% revenue lift with 98% confidence. Clear winner."
• "Only 15 sessions enrolled. Choosing Control due to insufficient data."
• "Test performed 40% worse. Reverting to Control."
• "Small 3% lift but high confidence. Choosing Test for incremental gains."

What Happens After Confirmation

When you confirm your decision:

• The winning variant goes live immediately for all future customers in this segment
• The losing variant is deactivated but preserved for reference
• The test moves to your completed tests history

FAQs

Can I test my primary Cancel Flow? No. A/B tests require segmented Cancel Flows because the primary flow serves as the fallback for all customers who don't match segment criteria.

Can I run multiple tests at once? Yes, as long as they test different segments with non-overlapping customer populations.

What happens if I need to edit a variant during the test? Technically possible, but strongly discouraged. It invalidates results because customers before and after the change experienced different flows.

Can I pause a test? Yes. Pausing stops new enrollments. You can resume later, but enrollment restarts from zero.

How do I see which variant a specific customer saw? Check the Exposure Stream section in the test results page.

Why can't I see final results immediately? The 30-day tracking period ensures we measure actual retention, not just initial offer acceptance. Without it, you'd optimize for people who say "yes" but churn anyway.

What happens to the losing variant's enrolled users? Nothing changes for them. The test measures what happened; it doesn't retroactively change anyone's experience.