Online Survey Validation SOP

An online survey validation SOP tells you how many people share a problem, and it takes about one day to build and one or two weeks to collect answers. Interviews give you the story behind a problem. A survey gives you the count: out of 100 people who match your customer profile, how many had the problem last month, and how many already pay to get rid of it. Run it after a round of calls, and the numbers either confirm what you heard or show that your 15 interviewees were the exception.

What is the online survey validation SOP and when should you use it?

It is a short questionnaire, sent to a screened audience, that measures how frequent and how expensive one problem is for one customer group. It does not ask people to judge your idea. People answer "Would you use this?" with a polite yes, and that yes predicts nothing.

Use this SOP:

• After 10 to 20 customer interviews, when you have a problem hypothesis and the customers' own words for it.
• When you need a number for investors or co-founders, such as "62% of solo accountants we surveyed lost a client to late reporting last quarter".
• When you want to compare two customer segments before choosing one.

If you need to rank five competing problems against each other, use the problem ranking survey SOP instead. If you already have users and want to test fit, the Sean Ellis test and the NPS survey are the right tools.

What do you need before you start?

• A written customer profile with two or three filters you can turn into screening questions. The persona development SOP gives you one.
• A problem hypothesis in one sentence, ideally phrased with words you heard in interviews.
• A distribution channel that reaches at least 300 people in that profile: a newsletter, a community, a paid panel such as Prolific, or targeted ads.
• A spreadsheet with the pass thresholds written in before the first answer arrives.

How do you run it, step by step?

1. Write the decision the survey must support

Write one line: "If 40% or more of qualified respondents had this problem in the last 30 days, we test a solution; if fewer, we change the customer or the problem." A survey without a decision rule turns into a pile of charts that support any conclusion.

2. Screen first, ask second

Open with two or three screening questions that remove people outside your profile, such as role, company size or a recent trigger event. Send unqualified people to a polite thank-you page. Report results only for qualified respondents.

3. Ask about past behaviour with concrete time frames

"In the last 30 days, how many times did you…" beats "How frequently do you…". Ask what they did about it, what it cost and which tool or person they used. Keep the survey to 10 questions or fewer, with one open question at most, so completion stays high on a phone.

4. Test the survey on five people before launch

Watch five people from your profile fill it in on a call. Every question they reread, misread or skip gets rewritten. Check for leading words ("frustrating", "annoying"), double questions ("cost and time") and answer options that overlap.

5. Distribute until you reach your sample target

For a large population, 385 qualified answers give a margin of error of about 5 points at 95% confidence, and about 100 answers give roughly 10 points. A first-round validation survey works with 100. Post in two or three channels so one community does not skew the result, and offer a small incentive such as a summary report or a $5 to $10 voucher draw.

6. Analyse against the thresholds you set

Filter to qualified answers, calculate each metric and compare it with the decision rule from step 1. Split by segment: a 30% average can hide one group at 60% and another at 10%. Read every open answer and tag the repeated phrases.

7. Book follow-up calls with the strongest respondents

The last question asks for an email for a 20-minute follow-up. Call the people who reported the highest frequency and spend. They are your first waitlist and your best interview pool.

Template you can copy

ONLINE SURVEY: [problem area] for [profile] (3-4 minutes)

Screening
1. Which best describes your role? [options, one = qualified]
2. How many [clients / employees / orders] do you handle per month?
   [ranges; set the qualifying range]

Problem frequency
3. In the last 30 days, did you [problem in their words]?  Yes / No / Not sure
4. If yes: how many times?  1 / 2-3 / 4-10 / More than 10

Current behaviour
5. The last time it happened, what did you do?
   [Nothing / Did it myself / Asked a colleague / Used a tool / Paid someone]
6. Roughly how much time did that take?  <15 min / 15-60 min / 1-3 h / 3 h+
7. Do you pay for any tool or service to handle this today?  Yes (which?) / No

Priority
8. How does this rank among your work problems this month?
   Top 1 / Top 3 / Top 10 / Not on the list

Open
9. What is the hardest part of [problem] for you? (optional)

Follow-up
10. Can we call you for 20 minutes about this? Email: ______

SPREADSHEET COLUMNS
Respondent ID | Qualified Y/N | Segment | Had problem 30d | Frequency |
Workaround | Time spent | Pays today | Rank | Open answer tags | Email

Worked example: a reporting tool for freelance bookkeepers

This is an illustrative case with invented numbers. A founder believes freelance bookkeepers lose hours every month chasing receipts from clients. After 12 interviews she runs a survey in two bookkeeping communities and one paid panel.

She collects 214 answers, and 131 pass the screening. Of those 131, 84 (64%) chased missing receipts in the last 30 days, and 49 did it four or more times. Only 18 (14%) pay for a tool that helps, while 61 spend over an hour a month on manual follow-ups. Thirty-seven leave an email.

The frequency passes her 40% rule easily. Paid spend sits below her 25% line, which tells her people suffer but have not tried to buy a fix. Her next step is a pricing sensitivity test with the 37 volunteers before any build.

What should you measure, and when do you stop?

Stop collecting when you pass your sample target or when 10 days pass with fewer than 10 new qualified answers a day. A slow trickle means the channel is wrong, and more days in the same channel will not fix that.

Which mistakes should you avoid?

• Asking for opinions on your idea. "Would you buy an app that…" produces cheap yeses. Ask about last month's behaviour.
• Surveying anyone who clicks. Without screening, students and curious founders fill your data with noise.
• Writing 25 questions. Completion drops sharply on long forms. Cut every question that does not feed the decision rule.
• Reading the average only. Segment splits show where the problem is strongest.
• Using a survey before any interviews. You will ask about the wrong problem in the wrong words.

How can AI speed this up?

Paste your interview quotes into an AI assistant and ask it to draft 10 neutral, past-tense survey questions with non-overlapping answer ranges. Then ask it to flag leading or double-barrelled questions in its own draft. After collection, export open answers and ask it to group them into themes with counts. Check every theme against the raw answers yourself, because AI groupings blend different complaints into one tidy label. The free idea validator gives a quick read of your concept before you spend a week collecting answers.

Where does this SOP fit in your validation plan?

Surveys come after interviews and before solution tests in the Customer Discovery stage.

1. Start with customer interviews to find the problem and its language.
2. Run this survey to count how many people share it.
3. Size the opportunity with the market sizing SOP, using your frequency rate as the share of the market with the problem.
4. Move to a landing page test to see whether people act on a promise.

Browse the full Customer Discovery SOP list. The F/MS free startup school covers survey design inside its validation lessons, with exercises you complete on your own idea.

FAQ

How many survey responses do I need to validate a startup idea?

Aim for 100 qualified responses for a first read, which gives a margin of error of about 10 points. For a number you will put in a pitch deck, collect around 385.

What is a good survey response rate?

It depends on the channel. Track completion rate instead: if fewer than 70% of people who start the survey finish it, the survey is too long or a question confuses people.

Should I offer an incentive for a validation survey?

A small one helps: a results summary, early access or a draw for a $5 to $10 voucher. Large cash rewards attract people who click through without reading.

Can a survey replace customer interviews?

No. A survey counts things you already know to ask about. Interviews find the things you did not know to ask, so run them first.