Growth Experiment SOP

A growth experiment is a small, time-boxed test of one change aimed at one number in your funnel, with a prediction written down before you start. This SOP runs each experiment in two to four weeks and keeps a one-hour weekly review. Many ideas fail, which is why you need a process: it tells you quickly which ones to drop and stops you from shipping changes nobody measured. Founders who run ten clean experiments learn more about their customers than founders who ship forty untested features.

What is a growth experiment and when should you use it?

A growth experiment changes one thing (a headline, a signup step, a pricing page, an email, a channel) and measures its effect on one metric against a control. The method comes from the growth teams Sean Ellis describes in Hacking Growth, co-written with Morgan Brown: a weekly rhythm of ideas, ranking, testing and learning.

It has clear limits. A growth experiment improves a funnel that already works for some people. If nobody stays, experiments on signup pages only speed up the leak; fix fit first with the Sean Ellis test and customer retention analysis. Pre-launch tests of demand belong to the landing page test and fake door testing SOPs.

Use this SOP:

• When your AARRR dashboard shows one weak stage and you have ideas to fix it.
• When the team argues about changes and nobody can prove what works.
• When you have steady traffic, at least a few hundred visitors or signups a week.
• When you change something big (pricing, first-week flow, a channel) and need proof it helped.

What do you need before you start?

• One target metric from the weakest funnel stage, with its current baseline.
• A way to split traffic: GrowthBook (open source) or PostHog feature flags both have free plans.
• An experiment log in a spreadsheet, set up before the first test.
• A sample size check, using a free tool such as Evan Miller's sample size calculator.
• A weekly 60-minute slot with whoever builds and ships the changes.

How do you run it, step by step?

1. Choose the target metric and baseline

Take the weakest stage from your funnel and write the metric with its last four weeks' average: "Signup to first project: 28%". Every idea in this cycle must aim at this number.

2. Collect ideas and write them as hypotheses

Gather ideas from user calls, session recordings and competitors. Write each in one line: "If we [change], then [metric] will rise from [baseline] to [target], because [reason from evidence]." The "because" forces you to name the evidence; ideas without one go to the bottom.

3. Rank ideas with ICE

Score each idea from 1 to 10 on Impact (how much it moves the metric), Confidence (how strong the evidence is) and Ease (how fast to build). Average the three. This ICE method comes from Sean Ellis and the GrowthHackers community. Run the top one or two.

4. Check the sample size before you build

Enter your baseline and the smallest lift worth detecting in a sample size calculator. With a 5% baseline conversion, detecting a lift to 6% needs roughly 8,000 visitors per version; detecting a lift to 7.5% needs about 1,500. Low traffic means you test bold changes, because small tweaks can't be measured.

5. Run the test for full weeks

Split traffic 50/50, launch, and leave it alone until the planned sample is reached, in whole weeks so weekday and weekend behaviour both count. Evan Miller's essay How Not To Run an A/B Test explains why stopping the moment the dashboard looks good produces false wins.

6. Read the result and log it

Record the result in the log: winner, loser or no difference, the lift with its confidence, and what you learned about customers. A "no difference" result still teaches you that the metric doesn't depend on that change.

7. Ship winners and pick the next test

Roll out winners to everyone and watch the metric for two more weeks to confirm. Then return to the idea list. Aim for 2 to 4 completed experiments a month.

Template you can copy

GROWTH EXPERIMENT CARD

ID:                  [EXP-12]
Funnel stage:        [Acquisition / Activation / Retention / Referral / Revenue]
Target metric:       [signup to first project]
Baseline:            [28%, last 4 weeks]
Hypothesis:          If we [change], then [metric] will rise from [x%]
                     to [y%], because [evidence].
ICE:                 Impact [ ] Confidence [ ] Ease [ ] = [average]
Sample needed:       [visitors per version, from calculator]
Run time:            [start date] to [end date], whole weeks
Owner:               [name]

RESULT
Control:             [x%] of [n]
Variant:             [y%] of [n]
Outcome:             [win / loss / no difference]
Confidence:          [%]
What we learned:     [one or two sentences about customers]
Decision:            [ship / drop / follow-up test]

EXPERIMENT LOG COLUMNS
ID | Stage | Hypothesis | ICE | Start | End | Sample | Result | Lift |
Learning | Decision

Worked example: a recipe subscription box

This example is illustrative. A meal-kit startup sees 12,000 visitors a month and a 4% visitor-to-trial conversion. Its AARRR review shows acquisition is fine but trial signup is weak. The team lists 14 ideas and ranks them with ICE.

The top idea: show the first week's actual menu on the landing page instead of stock photos, because five interview subjects said they didn't know what they'd get. ICE: 7, 6, 8. The sample size check for a lift from 4% to 5.5% shows about 3,000 visitors per version, so the test runs two weeks.

Result: control converts 4.1% (124 of 3,020), the menu version 5.3% (161 of 3,040). The team ships it, logs "visitors want to see the food before trusting the box", and the next experiment tests showing a full month of menus.

What should you measure, and when do you stop?

Stop an individual experiment when it reaches its planned sample, never earlier. Move to a new target metric when the current one hits its target or three tests in a row show no difference, which tells you the next lift lies in a different stage.

Which mistakes should you avoid?

• Testing on a product that doesn't retain. Growth experiments speed up whatever already happens, good or bad.
• Peeking and stopping early. Early "wins" vanish with more data.
• Testing button colours with 500 visitors a month. You can't detect small lifts with small traffic; test bigger ideas.
• Changing several things in one variant without saying so. You get a result with no lesson attached.
• Not writing the prediction first. Without it, every result looks like what you expected.

How can AI speed this up?

Give an AI assistant your funnel numbers, interview notes and the weak stage, and ask it for 20 hypotheses in the template format with an evidence line for each. Ask it to pre-score ICE, then rescore yourself, since AI rates Ease without knowing your codebase. AI can also write variant copy and analyse a finished test from raw counts. Check its statistics against the calculator, and never let it declare a winner before the planned sample is reached.

For idea lists to feed the hypothesis stage, try the free marketing ideas generator, and the F/MS landing page module covers the page elements most worth testing.

Where does this SOP fit in your validation plan?

Growth experiments come after you have proof of fit and a working funnel. Before this SOP, set up the AARRR metrics framework to find the weak stage. Related SOPs:

1. Test channels for the acquisition stage with customer acquisition channel validation.
2. Test headlines and copy with marketing message testing.
3. Find activation ideas in product usage analysis.

See all Product-Market Fit SOPs.

FAQ

What is the ICE score in growth experiments?

ICE ranks ideas by Impact, Confidence and Ease, each scored 1 to 10 and averaged. It was popularised by Sean Ellis and the GrowthHackers community.

How long should a growth experiment run?

Until it reaches the sample size you calculated, in whole weeks. For most early products that means two to four weeks.

How many visitors do I need for an A/B test?

It depends on your baseline and the lift you want to detect. At a 5% baseline, a lift to 6% needs about 8,000 visitors per version; a lift to 7.5% needs about 1,500.

What if my startup doesn't have enough traffic for A/B tests?

Test bigger changes, run before-and-after comparisons over full weeks with care, and add qualitative checks such as five user calls per change.