Problem Ranking Survey SOP

A problem ranking survey SOP tells you which of the problems you heard in interviews matters most to the most people, and it takes about a week from draft to decision. Interviews give you depth from 15 to 20 people. A ranking survey gives you a ranked list from 100 or more, scored on two axes: how important the problem is and how badly today's fixes handle it. The problem that wins on both is where your product should start.

What is the problem ranking survey SOP and when should you use it?

A problem ranking survey is a short questionnaire that shows respondents a fixed list of problems and forces trade-offs between them. This SOP combines two published methods. The first is Anthony Ulwick's opportunity score from his ODI method (the approach his firm Strategyn teaches), described in his book What Customers Want: importance plus the gap between importance and satisfaction. The second is MaxDiff (best-worst scaling), developed by Jordan Louviere, where people pick the most and least important item from small sets. Ratings alone let people mark everything "very important". MaxDiff makes them choose.

Use this SOP:

• When interviews surfaced three or more problems and you can't tell which one is biggest.
• When your team disagrees on focus, and you want a number from customers instead of a vote in the office.
• Before you test solutions, so your concept tests aim at the top problem.

This SOP ranks problems. To rank features of a product you already chose, use the feature prioritization SOP. For a general-purpose questionnaire about attitudes and demographics, use the online survey validation SOP.

What do you need before you start?

• 6 to 10 problem statements written in customer language, each one sentence, with no solution inside. The problem statement validation SOP shows the format.
• One defined customer group with a screener question that filters out everyone else. The free user persona generator helps you write the profile.
• A channel to reach 150 to 300 people of that group: newsletters, communities, LinkedIn, a panel such as Prolific, or your waitlist.
• A spreadsheet with the opportunity score formula ready before the first response arrives.

How do you run it, step by step?

1. Write the problem list

Pull 6 to 10 problems from interview notes and phrase each as a situation the respondent lives through ("Finding cover when an instructor calls in sick"). Keep every item the same length and tone so none stands out. Add one problem you expect to score low as a sanity check.

2. Add a screener and two profile questions

Start with one or two questions that confirm the person is in your group, and end the survey politely for anyone who fails. Add two profile questions (company size, role, years in the job) so you can split results by segment later.

3. Ask importance and satisfaction for each problem

For each problem, ask "How important is it for you to solve this?" and "How satisfied are you with the way you handle it today?" on a 1 to 5 scale. Randomise the order of problems for each respondent so position bias cancels out.

4. Add a MaxDiff block

Show sets of 4 or 5 problems and ask which is most and which is least important. Design the sets so each problem appears at least three times per respondent. Tools such as OpinionX and Conjointly build the sets for you, and a hand-made version in Typeform works for up to 8 problems.

5. Collect 100+ complete responses per segment

Send the survey through your channels and offer a small thank-you, such as the results summary. Track completion rate daily. If more than 30% drop out, the survey is too long: cut the profile questions before you cut problems.

6. Compute the opportunity score

For each problem, importance = share of people who rated it 4 or 5, multiplied by 10. Do the same for satisfaction. Opportunity = importance + max(importance minus satisfaction, 0). Ulwick treats scores above 10 as underserved and above 15 as extreme opportunities. Compare the opportunity list with the MaxDiff ranking.

7. Decide and write it down

Pick the problem that sits in the top two on both lists. Write a one-page memo: the winning problem, its scores, the segment where it scores highest and the runner-up you will keep an eye on.

Template you can copy

SCREENER
S1. Which describes you best? [target role] / [other] / [other]  → end survey if not target
S2. In the last 3 months, have you [trigger event]? Yes / No  → end if No

PROFILE
P1. [company size / household type]
P2. [years in role / experience level]

PER PROBLEM (repeat for each, random order)
Q1. How important is it for you to solve this?        1  2  3  4  5
Q2. How satisfied are you with how you handle it today? 1  2  3  4  5

MAXDIFF (repeat 6–8 screens, 4–5 problems per screen)
"Of these, which is MOST important to you to solve? Which is LEAST?"

CLOSE
C1. Anything we missed? (open text)
C2. Can we contact you about the results? Email (optional)

SPREADSHEET COLUMNS
Problem | % importance 4–5 | Importance (x10) | % satisfaction 4–5 |
Satisfaction (x10) | Opportunity = I + MAX(I−S,0) | MaxDiff best−worst count | Rank

Worked example: owners of independent yoga studios

This is an illustrative case. A founder interviewed 18 studio owners and heard about eight problems. She screens for owners with one or two locations and collects 124 complete responses through two studio-owner newsletters.

Three results stand out. "Finding cover when an instructor is sick" gets importance 8.6 and satisfaction 3.4, so the opportunity score is 8.6 + 5.2 = 13.8. "Filling off-peak classes" scores 8.1 and 2.9, giving 13.3. "Collecting payments", the problem her friends told her to solve, scores importance 6.0 and satisfaction 7.5, giving 6.0, because existing booking tools already handle it.

The MaxDiff block puts instructor cover first and off-peak classes second. Split by segment, owners with two locations score instructor cover at 15.1. She picks that problem, narrows her customer to two-location studios and books her next round of interviews about how cover is arranged today.

What should you measure, and when do you stop?

Stop collecting once the top two problems hold their position across three daily snapshots. If no problem passes 10, your list is missing the real problem: go back to interviews or run a pain point diary to find it.

Which mistakes should you avoid?

• Adding problems nobody mentioned. Every item on the list must come from interviews, or you are testing your imagination.
• Ranking without satisfaction. A problem can be important and already well handled. Satisfaction is what shows the gap.
• Mixing customer groups. Freelancers and agencies rank problems differently, and the average hides both.
• Writing solutions into the items. "Automatic cover booking" tests a feature, and respondents will score the feature.
• Treating rank one as permission to build. A high score says the problem matters. Whether people pay is the next question, answered by a problem monetization assessment.

How can AI speed this up?

An AI assistant can rewrite your problem list so every item has the same length and reading level, check it for leading words, and generate a balanced MaxDiff design for up to 12 items. After collection, paste the CSV and ask it to compute importance, satisfaction and opportunity scores per segment, then check two rows by hand. A formula error in one cell can flip your ranking, so verify before you decide.

If you want a structured course on surveys, interviews and the rest of validation, the F/MS free startup school covers them in short modules.

Where does this SOP fit in your validation plan?

Run it after customer interviews and after you have tested your problem statements. Then check how big the winner is with problem search volume analysis and whether it is growing with problem trend analysis. All ten SOPs in this stage are on the Problem Validation page.

FAQ

How many problems should a ranking survey include?

Between 6 and 10. Fewer than 6 gives you little to compare, and more than 10 makes the survey long enough that people start clicking at random.

What is a good opportunity score?

In Ulwick's ODI method, scores above 10 mark an underserved need and above 15 an extreme one. Below 10, people are reasonably happy with what they have.

Is MaxDiff better than a simple ranking question?

For more than five items, yes. Dragging ten items into order is tiring and the middle positions become noise. MaxDiff asks easier questions and still produces a full ranking.

How many responses do I need?

At least 100 complete responses per segment you want to compare. With fewer, one enthusiastic community can tilt the whole result.